Hi, I’m Max :)
looking for work in AI governance (general strategy, expert surveys, research infrastructure, EU tech policy fellow)
background in cognitive science & biology (did research on metacognition)
most worried about AI going badly for technical & coordination reasons
vegan for the animals
doing my own forecasts: https://www.metaculus.com/accounts/profile/110500/
Fwiw, I also think the name is a bit complicated, and less memorable than Open Philanthropy. Here the reasoning from the Vox interview:
> Why “coefficient”? As CEO, Alexander Berger, puts it in my conversation with him, “coefficient is a multiplier”: the “co-” nods to collaboration with other givers; the “efficient” is a reminder of the north star of effectiveness.
Huge fan of your work, one of the few newsletters I read every week.
Random question, I wonder whether prediction markets are a potentially promising income stream for the team? E.g. Polymarket seems to have a bunch of overlap with the topics you’re covering.
Also, thanks for making your news-parsing code open source, was often curious how it looks like under the hood.
Hi Connacher! Thanks for the responses, makes sense.
On your question, one example I often miss from expert surveys is something like this open-ended question: “Do you have any other considerations that would help with understanding this topic?”
I generally agree that quantitative questions are intimately connected with identifying cruxes. Being quantitative about concrete events is a neat way of forcing the experts get more concrete and incentivize them to not get lost in a vague story, etc. But I suspect that often the individual insights from the experts might seem like cruxes to them, as they’re not used to think like that. So I think giving experts some prompts to just pour out their thoughts is often neglected. Furthermore, sometimes quantitative questions don’t fully capture all important angles of an issue and so it’s useful to give responders many chances to add additional comments.
Thanks for the work, this is great!
I especially appreciate the rationale summaries, and generally I’d encourage you to lean more into identifying underlying cruxes as opposed to quantitative estimates. (E.g. I’m skeptical on experts being sufficiently well calibrated to give particularly informative timeline forecasts).
I’m looking forward to the risk-related surveys. Would be interesting to hear their thoughts on the likelihood of concrete risks. One idea that comes to mind would be conditional forecasts on specific interventions to reduce risks.
Also, I wonder whether the presentation on the website could also feature some more “snack-sized” insights, next to the more long-form report-focussed presentation. E.g. the Chicago Booth expert surveys on economic experts focus on ~2-3 questions per article, with a handful of rationales of experts quoted in full. It keeps me coming back because it’s informative and takes up less than 5 minutes of my time.
https://kentclarkcenter.org/us-economic-experts-panel/
PS: Just in case something went wrong:
The wave 1 report questions imply there would be a section “Cognitive Limitations I” that is not included in the wave 1 report
Same for “Adoption Barriers I” that is not in the wave 2 report.
Thanks for the interesting interview!
Fwiw, this section made me feel like it should be thought through more deeply:
Luisa Rodriguez: [...] How confident are you that leaders in the countries that are set up to race and are already racing a little bit are going to see this as close to existential?
Daniel Kokotajlo: I think it will be existential if one side is racing and the other side isn’t. And even if they don’t see that yet, by the time they have superintelligences, then they will see it — because the superintelligences, being superintelligent, will be able to correctly identify this strategic consideration, and probably communicate that to the humans around them.
Luisa Rodriguez: Right. Once you have AGI, the AGI is like, “This is existential. We should do this big wartime effort to create a robot economy that’s going to give us this big advantage.”
Daniel Kokotajlo: That’s right. [...]
Some random thoughts:
Leaders already abstractly believe that AI is existentially important (e.g. Putin since 2017 apparently) and don’t really behave accordingly.
I think it’s very likely that there are already advisors in China and the U.S. that frequently argue for the existential importance of AI / ASI to the leader or people close to them. It’s unclear to me that e.g. Trump would trust an ASI much more than personal advisors.
I think it’s unclear that an ASI, especially from the current paradigm, will advise to race, and probably will strongly urge for cooperation and at most limited racing, e.g. defensive acceleration.
(Just quick random thoughts.)
The more that Trump is perceived as a liability for the party, the more likely they would go along with an impeachment after a scandal.
Reach out to Republicans in your state about your unhappiness about the recent behavior of the Trump administration.
Financially support investigative reporting on the Trump administration.
Go to protests?
Comment on Twitter? On Truth Social?
It’s possibly underrated to write concise and common sense pushback in the Republican Twitter sphere?
I relate hard with the career struggles, thanks for sharing! :’) Also very sweet (and again relatable) to drop everything for true love. :3
Thanks for writing this up, I think it’s a really useful benchmark for tracking AI capabilities.
One minor feedback point, I feel like instead of reporting on statistical significance in the summary, I’d report on effect sizes, or maybe even better just put the discrimination plots in the summary as they give a very concrete and striking sense of the difference in performance. Statistical significance is affected by how many datapoints you have, which makes lack of a difference especially hard to interpret in terms of how real-world significant the difference is.
Most of my donations were forgone payments for hiring rounds at organizations I consider among the most promising at reducing risks from AI (e.g. Horizon and MIRI).
Thanks for sharing, I really appreciate your committment, and that you announce it.
Fwiw, my immediate reaction is that this type of protest might be a little too soon and it will cause more ridicule and backlash because the general public’s and newsmedia’s impression is that there is currently no immediate danger. Would be interested in learning more about the timing considerations. Like, I’d imagine that doing this barricading in the aftermath of some concrete harm happening would make favorable reporting for newsmedia much more likely, and then you could steer the discourse towards future and greater harms.
Cool, thanks for sharing!
We can sponsor US visas for technical roles
Does this apply to any of the roles you list here?
I love the new profiles, and also all the new formatting options for comments, plus the filtering for private notes. Thanks so much! :)
Thanks so much for all your contributions Lizka! :) I really appreciated your presence on the forum, like a friendly, alive, and thoughtful soul that was attending to and helping grow this part of our ecosystem.
I can relate to the part about how unthankful it can be to be a mediator… it’s a pretty interesting dynamic where something really useful is being disincentivized, would be interested in hearing more of your, or others’ thoughts about it.
And I feel sorry that your work on moderation had negative effects on your interpersonal relationships. :| I don’t know how that exactly looked like for you, but could also imagine an exploration into dynamics here might be pretty interested and potentially helpful to understand more widely.
The part about people being confused about CEA’s role and activities, and the autonomy of the invidual teams, made me think that it might make sense to make the teams more prominent in comparison to CEA overall?
Like, giving teams more prominent names and brands (and cool logos!), emphasizing the teams more as the relevant entity that did something as opposed to CEA?
Thanks for doing this work, this seems like a particularly useful benchmark to track the world model of AI systems.
I found it pretty interesting to read the prompts you use, which are quite extensive and give a lot of useful structure to the reasoning. I was surprised to see in table 16 that the zero-shot prompts had almost the same performance level. The prompting kinda introduces a bunch of variance I imagine, and I wonder whether I should expect scaffolding (like https://futuresearch.ai/ are presumable focussing on) to cause significant improvements.
Thanks, that all makes sense and moderates my optimism a bit, and it feels like we roughly exhausted the depth of my thinking. Sigh… anyways, I’m really thankful and maybe also optimistic for the work that dedicated and strategically thinking people like you have been and will be doing for animals.
That’s interesting, based on thinking that animal protein in the end comes from plant protein, and that animals use up a lot of space, food, and extra infrastructure that is not directly involved in turning plant protein into meat, I’d’ve guessed that plant protein would be much cheaper than animal protein.
I quickly asked chatGPT for the cheapest animal vs. plant proteins in the US:
Chicken: Approximately 6.6 cents per gram of protein
Lentils: Approximately 3.7 cents per gram of protein
Less difference than I’d’ve guessed.
Interesting, hard for me to judge. Reading the Innovations needed section, it seems like most hurdles are in the 3 OOM range, only the growth factor price is 6 OOM off.
My naive reaction is: AI + increased wealth + generally improving science & engineering + increased caring + those are “just” engineering problems → I’m much more bullish than the authors of the report, who are 9% for:
>50M metric tons of cellular meat will be sold at any price within a continuous 12-month span before the end of 2051. 9%
(For context, the annual production of conventional meat (excluding seafood) in 2018 was 346M metric tons)
I’m maybe at 40%, given that plant-based meat might just do it by itself, or that something disruptive happens that affects R&D broadly, or that it’s just unintuitively difficult.
I’m more hopeful this will be more comparable to civil rights / racism / sexism, as there are concrete victims who are suffering (and there is already a broad agreement that animals in human guardianship shouldn’t suffer), compared to climate change, which is much more abstract and indirect.
Yeah, I somewhat agree that the steady increase will probably bottom out at maybe 20%. But my hopeful vision is that at 20%, there will be critical mass effects for political action and for the demand of alternative products to lead to a much more mature industry.
Plus I expect health and climate change angles on meat consumption will also more likely than not steadily increase.
Finally, I’m also probably more optimistic about your last point, thinking that price/taste competitive meat alternatives will be huge. I think the Beyond and Impossible “moments” were huge milestones, and a few more “moments” like that will reduce resistance against higher welfare standards & higher prices for conventional meat.
Really interesting, thanks for sharing. I was particularly surprised about your changes of mind here:
We can make meaningful progress on abolishing factory farming or improving farmed animal welfare by 2050 75% 10% -65% We can make meaningful progress on abolishing factory farming or improving farmed animal welfare by 2100 85% 15% -70%
E.g. some spontaneous potential cruxes that might be interesting to hear your disagreement with, in case they capture your reasons for pessimism:
Plant protein sources will become price- and taste-competitive with more than half of all consumed meat products by 2040.
Cultured meat will become price- and taste-competitive with more than half of all consumed meat by 2050.
There is a critical mass of people caring about animal welfare at which animal rights will become major political issues in most democracies.
There will be a steady increase in people caring about animal welfare in the coming decade.
E.g. I expect Germany to have a >20% vegetarian population by 2040 (currently apparently 12%).
Thanks for this! :) I unfortunately only had time for skimming but I found the summary of pathways super useful and I had a positive impression of the level of rigor that went into this. Also appreciate the section on downside risks and how to address them.
Thanks so much for sharing your writing, it resonated deeply with me and made me cry more than once.
It’s great that you already have a rationale prompt for each question. I would probably recommend having one prompt like this at the end, with “(Optional)” in front so experts can share all further thoughts they think might be useful.