Hey there, I’m Austin, currently running https://manifund.org. Always happy to meet people; reach out at akrolsmir@gmail.com!
Austin
Karma: 4,168
Introducing the Mox Guest Program
For the $1m estimate, I think the figures were intended to include estimated opportunity cost foregone (eg when self-funding), and Marcus ballparked it at $100k/y * 10 years? But this is obviously a tricky calculation.
tbh, I would have assumed that the $300k through LTFF was not your primary source of funding—it’s awesome that you’ve produced your videos on relatively low budgets! (and maybe we should work on getting you more funding, haha)
Oh definitely! I agree that by default, paid ads reach lower-quality & less-engaged audiences, and the question would be how much to adjust that by.
(though paid ads might work better for a goal of reaching new people, of increasing total # of people who have heard of core AI safety ideas)
Thanks for the thoughtful replies, here and elsewhere!
Very much appreciate data corrections! I think medium-to-long term, our goal is to have this info in some kind of database where anyone can suggest data corrections or upload their own numbers, like Wikipedia or Github
Tentatively, I think paid advertising is reasonable to include. Maybe more creators should be buying ads! So long as you’re getting exposure in a cost-effective way and reaching equivalently-good people, I think “spend money/effort to create content” and “spend money/effort to distribute content” are both very reasonable interventions
I don’t have strong takes on quality weightings—Marcus is much more of a video junkie than me, and has spent the last couple weeks with these videos playing constantly, so I’ll let him weigh in. But broadly I do expect people to have very different takes on quality—I’m not expecting people to agree on quality, but rather want people to have the chance to put down their own estimates. (I’m curious to see your takes on all the other channels too!)
Sorry if we didn’t include your feedback in the first post—I think the nature of this project is that waiting for feedback from everyone is going to delay our output by too much, and we’re aiming to post often and wait for corrections in public, mostly because we’re extremely bandwidth constrained (running this on something like 0.25 FTE between Marcus and me)
Hey Liron! I think growth in viewership is a key reason to start and continue projects like Doom Debates. I think we’re still pretty early in the AI safety discourse, and the “market” should grow, along with all of these channels.
I also think that there are many other credible sources of impact other than raw viewership—for example, I think you interviewing Vitalik is great, because it legitimizes the field, puts his views on the record, and gives him space to reflect on what actions to take—even if not that many people end up seeing the video. (compare irl talks from speakers eg at Manifest—much fewer viewers per talk but the theory of impact is somewhat different)
I have some intuitions in that direction (ie for a given individual to a topic, the first minute of exposure is more valuable than the 100th), and that would be the case for supporting things like TikToks.
I’d love to get some estimates on what the drop-off in value looks like! It might be tricky to actually apply—we/creators have individual video view counts and lengths, but no data on uniqueness of viewer (both for a single video and across different videos on the same channel, which I’d think should count as cumulative)
The drop-off might be less than your example suggests—it’s actually very unclear to me which of those 2 I’d prefer.
AI safety videos can have impact by:
Introducing people to a topic
Convincing people to take an action (change careers, donate)
Providing info to people working in the field
And shortform does relatively better on 1 and worse on 2 and 3, imo.
Super excited to have this out; Marcus and I have been thinking about this for the last couple of weeks. We’re hoping this is a first step towards getting public cost-effectiveness estimates in AI safety; even if our estimates aren’t perfect, it’s good to have some made-up numbers.
Other thoughts:
Before doing this, we might have guessed that it’d be most cost-effective to make many cheap, low effort-videos. AI in Context belies this; they’ve spent the most per video (>$100k/video vs $10-$20k that others) but still achieve strong QAVM/$
I think of Dwarkesh as perhaps being “too successful to be pigeonholed into ‘AI safety’”, but I still think his case is important because I expect media reach to be very heavy-tailed. Also, even if the vast minority of his content is safety-oriented, he has the strongest audience quality (with eg heads of state and megacorps on the record as fans)
Marcus chose to set Qa as 1 for “human chosen at random worldwide”; I think this might artificially inflate QAVM, and better baselines might be “random video watcher” (excluding the very poor) or “random US citizen”. But I’m still unsure. Overall, we’re more confident in the relative rankings, and less confident about what “1 QAVM” is worth.
One further goal I have is to establish impact valuations for different orgs/creators based on their “impact revenue”. Perhaps before we published this, we should have polled some grantmakers or creators to see what they would have been excited to pay for 1 QAVM...
How cost-effective are AI safety YouTubers?
Broadly agree that applying EA principles towards other cause areas would be great, especially for areas that are already intuitively popular and have a lot of money behind them (eg climate change, education). One could imagine a consultancy or research org that specializes as “Givewell of X”.
Influencing where other billionaires give also seems great. My understanding is that this is Longview’s remit, but I don’t think they’ve succeeded at moving giant amounts yet, and there’s a lot of room for others to try similar things. It might be harder to advise billionaires who already have a foundation (eg Bill Gates), since their foundations see it as their role to decide where money should go; but doing work to catch the eye of newly minted billionaires might be a viable strategy, similar to how Givewell found a Dustin.
We pre-ordered a batch of 50 books for Mox, and are hosting a Q&A and book signing with Nate Soares on (tentatively) Oct 9th. I encourage other groups (eg fellowships like MATS, hubs like Constellation) to organize events like this if your audiences would be interested!
We’ve been considering an effort like this on Manifund’s side, and will likely publish some (very rudimentary) results soon!
Here are some of my guesses why this hasn’t happened already:
As others mentioned, longtermism/xrisk work has long feedback loops, and the impact of different kinds of work is very sensitive to background assumptions
AI safety is newer as a field—it’s more like early-stage venture funding (which is about speculating on unproven teams and ideas) or academic research, rather than public equities (where there’s lots of data for analysts to go over)
AI safety is also a tight-knit field, so impressions travel by word of mouth rather than through public analyses
It takes a special kind of person to be able to do Givewell-type analyses well; grantmaking skill is rare. It then takes some thick skin to publish work that’s critical of people in a tight-knit field
OpenPhil and Longview don’t have much incentive to publish their own analyses (as opposed to just showing them to their own donors); they’ll get funded either way, and on the flip side, publishing their work exposes them to downside risk
I’m excited for the rest of this sequence!
Some basic questions I’m curious to get data are: what are the sizes of the boards you’ve been surveying? And how do board sizes compare to team sizes? How often do boards meet? Outside of meetings, how often are there other comms (email/Slack/etc)?
Excellent list! I think these are broadly promising ideas and would love to see people try them out.
I just had an in-person discussion about one of these specifically (4. AI safety communications consultancy), and thought I’d state some reasons why I think that might be less good:
I’m not sure how much demand there would be for a consultancy; unsure whether having AI safety orgs communicate more broadly about their work is useful/good for a vast swath of AI safety orgs (especially TAIS, which I think is trying to produce research for other researchers to adopt).
Comms is an overloaded term and one might say mean “helping TAIS orgs get their research into other researcher’s hands would be good”—this might be reasonable, I don’t know enough about
I tend to think, ala Paul Graham, “big media launch/getting into NYT” type things are generally overrated, in terms of actual impact. (This is a cached heuristic from startup-land; one could argue that AI safety stuff is different)
Insofar as comms matter, I’m not sure a consultancy model is the right fit. I have a prior against working with consultancies in general, vs doing things in-house. Maybe one question is whether comms needs to be a core competency of your org—if so, you should in-house it, and if not, maybe you should just… ignore it?
I’m a much bigger believer in founder-led comms than outsourcing comms
Caveats: I do think of AI safety comms as being quite impactful and think eg the recent 80k video on AI 2027, and the upcoming MIRI book, as being very promising. I’m also excited for more writing, summarizations, analyses (like Zvi or Steve Newman’s, or what Asterisk’s fellowship is aiming for) So perhaps I’m excited for people/orgs that are just focused on doing comms, as opposed to advising others to do comms. And on a meta level I think people should just try doing things—if you were thinking about doing #4, I endorse going for it and ignoring naysayers like me :)
Quick responses:
I’m not familiar on the tradeoffs between farmed animals/mammals vs wild/invertebrates; I think arguments like yours are plausible but also brittle
My question is, “given that humans care so much about human welfare, why isn’t there already a human welfare market? The lack of such a market may be evidence that animal welfare markets will be unpopular”
I think you’re taking the ballparks a bit too literally—though I also invite you to put down your own best guess about how much eg US consumers would be willing to pay for welfare, I probably believe in higher numbers than you
I’ve come around to believing that narrowly focusing on cost effectiveness is the wrong approach for me, for moral parliament and learning reasons
Without knowing a ton about the economics, my understanding is that Project Vesta, as a startup working on carbon capture and sequestration, costs more per ton than other initiatives currently, but the hope is with continued revenue & investment they can go down the cost curve. I agree it’s hard to know for sure what to believe—the geoengineering route taken by Make Sunsets is somewhat more controversial than CC&S (and I think, encodes more assumptions about efficacy), and one might reasonably prefer a more direct if expensive route to reversing carbon emissions. I might make a rough analogy to the difference between GiveDirectly and AMF, with reasonable people preferring the first due to being more direct (even if less cost effective).
We’d definitely want to avoid establishing perverse incentives (such as the famous “paying for dead cobras ⇒ cobra farming”). At first glance, your example seems to gesture at a problem with establish credits for “alleviating suffering” versus “producing positive welfare”, which I agree we might want to avoid.
I don’t think that paying for welfare has to be implemented as “paying for someone not to do something bad”—some carbon credits are set up way (paying not to chop down trees), but others are not (Stripe Frontier’s carbon capture & sequestration). Perhaps you as a wealthy individual would be better served by offering eg $1 per proven happy hen-year? So long as you set the prices correctly, I do think it would be a good thing to farm more animals in a way where the animals are generally happy, similar to how it would be a good thing to institute policies creating more happy human lives.
In practice, I think we’d want to balance considerations like potential perverse incentives/blackmail, versus “how feasible is it to actually implement”, and also rely on sanity checks/monitoring/investigative reporting to understand how these markets are functioning.
Questions about animal welfare markets
Thanks for the writeup, I found this to be well considered and written, and I agree with many of the top level points (analysis of what EA is; criticism of lack of structure and centralization; individual EAs being pretty great).
Hey Trevor! One of the neat things about Manival is the idea that you can create custom criteria to try and find supporting information that you as a grantmaker want to weigh heavily, such as for adverse selection. So for example, one could create their own scoring system that includes a data fetcher node or a synthesizer node, which looks for signals like “OpenPhil funded this two years ago, but has declined to fund this now”.
Re: adverse selection in particular, I still believe what I wrote a couple years ago: adverse selection seems like a relevant consideration for longtermist/xrisk grantmaking, but not one of the most important problems to tackle (which, off the top of my head, I might identify as “not enough great projects”, “not enough activated money”, “long and unclear feedback loops”). Or: my intuition is that the amount of money wasted, or impact lost, due to adverse selection problems is pretty negligible, compared to upside potential in growing the field. I’m not super confident in this though and curious if you have different takes!
Thanks for writing this—I’m a pro-life EA, which feels like being a member of a rare species, and so I appreciated you posting this here. I thought the diagrams were an especially nice touch!