AI safety fieldbuilding down under. Former management consultant and data scientist.
mjkerrisonšøļø
Karma: 99
FWIW I agree with the sentiment!
However, I think this post as it stands leans a lot on the Importance dimension of the ITN framework, gestures at Neglectedness, and doesnāt directly address Tractability. To be clear, thatās fine! Not every post needs to address everything; some of the linked posts address those somewhat; other resources out there do too.
But I do think that the question of āwhat can be done with 100s of extra $bnā is also a question of T and N, and I think part (majority?) of what Ransohoff is saying is that weāre woefully unprepared ~1 level down from ācause areaā at the infrastructure /ā intervention /ā ecosystem /ā etc. level (where T and N mostly cash out, IMO).
If people have or can point to write-ups about how many and what size of shovel-ready projects we have for the various animal problems listed: definitely post and signal boost those!!
Maybe a crux here might be āwhat sort of net effect one expects from more or less just spending any amount of $ to hire anyone you can to try to find those projectsā. Not saying thatās the authorās take per se; but I am somewhat skeptical of things shaped like āwe need[...] a lot more people working on [X]ā if what they mean is āwe need to get warm bodies on this at all costsā. People can have net negative impact! There are opportunity costs to going all-in like that!
Overall I hope a large chunk of this wave of philanthropy goes to animal welfare, but also that it can be spent well and deployed quickly :)
Very cool, and will strongly be considering switching my donations if/āwhere I can, potentially even at the cost of tax-effectiveness (in Australia)!
Weighing in as a comment I made has been cited below: thank you for your moderation efforts.
Iām an admittedly infrequent poster/ācommenter on the Forum, in no small part because I donāt have a lot of time on my hands. As such Iām often discouraged from engaging with certain topics or (potentially) productive disagreements because I simply cannot afford to engage with the kind of [charitable ? detailed : gish galloping] posts and comments exemplified in this thread.
Just wanted to provide something to counterbalance any adverse selection in responses.
I think it reflects poorly on you that a āhighly-upvotedā comment was a stronger update towards āthe Forum at large is wrongā more than āyou were wrongā or even āyou should change how you communicateā.
On the object level: I read your post, and like ~half of people who reacted to it, disagreed with it; specifically I think it has narrative-like qualities, and regardless of how much we quibble over the precise point at which text can be reasonably called ānarrativeā, I thought it was non-evidence-based argumentation, which you were arguing against.
I think in general it is bad forum etiquette to āsnitch tagā like this. I think youāre misrepresenting my comment and raising it in an irrelevant context.
Is there any work on calibration /ā assessing how well the 2024 report held up? Would be really useful to know how well the hiring managersā (etc.) predictions can be expected to hold up. Put another way, everything in this report rings true to me, but if itās true but e.g. the opportunities/ājobs for Connectors and Amplifiers donāt actually manifest, thenā¦ that seems bad?
Rather than go through this paragraph-by-paragraph, let me pick one particular thing.
Your overall thesis is that thereās little or no evidence behind many models of AI risk.
By Ad-hoc predictive models, youāve cited a sum total of 2 blog posts criticising AI 2027 - hardly the āacademic consensusā you ask for in that paragraph.
You also donāt actually point to any specific or characteristic issues from those 2 blog posts in that paragraph, instead appealing to heuristics, concept handles, and accusations. I would honestly describe that paragraph as a narrative argument.
Overall I disagree and am also downvoting this post as not a helpful contribution.
AI Safety UnĀconĀferĀence, Melbourne 2025
I think this dichotomy is interesting: IMO āaccountabilityā and āhealthinessā are different dimensions. Iām voting ānoā on āprotecting influential members from accountabilityā, but abstaining from āencouraging healthy accountabilityā, because while there is quite a bit of that (again IMO), thereās also unhealthy accountability at times, which (once again IMO) contributes to sometimes sparse details being published.
Oh manā¦ this really make it sound like Itās So Over
I also have this question. As someone looking to apply for funding to continue an org in the space, all of this uncertainty is tough to grapple with.
Hey NobodyInteresting,
I see youāre getting downvoted and disagreed with without any direct interaction, so Iāll bite.
diversityā¦ western developed mindsetā¦ clearly lacking
I think this is a bit combative in how itās delivered. I suspect a number of people actually agree with the āecho chamberā problemāit seems like a number of the other comments more or less say something like this in how they disagree with 80kās conclusions.
However, you might need to elaborate on the āwesternā aspect. What do you think is the shortcoming youāre identifying here? For instance, I expect a western lens is the right one for this problem because most of the frontier labs are western.
AGI delivery by 2030 will failā¦ AGI will surely be much more complex, if possible even
I think this is a controversial callāsome agree, some disagree. But I donāt think anyone can say this with confidence: for instance, the frontier lab leaders all seem to be saying this, in a way that doesnāt seem to be explained by āhypeā or āpandering to investorsā. This would be a good one to elaborate on your views /ā provide a link to the strongest evidence you buy into.
we barely have resources to properly run these current LLM models
I think this is just not true? Maybe we donāt have the resources to run them personally, but plenty of companies seem to be running their latest models for (limited) public access just fine. Are you addressing these inequality considerations or something else?
Congratulations you have now jumped on a trend-wagon that will take us nowhere, while forgetting about actual causes that are relevant right now
I think Iāve mostly responded to this, but Iād like to connect it with the āleft behindā point at the end:
I think itās good, and highly in keeping with EA principles, to (a) respond to information as it comes in and (b) think and work on the margin. This is probably too big to tackle in this commentāmaybe Iāll write something up laterābut
changing advice on the margin (for the next person) isnāt by necessity the same as āleaving people behindā
I would be interested in why or whether people who agree with the advice /ā 80kās judgment feel they should stay where they are, instead of also pivoting?
It seems to me that extinction is the ultimate form of lock-in, while surviving provides more opportunities to increase the value of the future. This moves me very far toward Agree. It seems possible, however, that there could be future that rely on actions today that are so much better than alternatives that it could be worth rolling worse dice, or futures so bad that extinction could be preferable, so this brings me back a bit from very high Agree.
On the margin: I think we are not currently well-equipped to determine whether actions are or arenāt increasing the value of the future[1]. Focusing on protecting what we have seems more prudent, as there are concerningly many concerningly high extinction risks.
- ^
This includes things like concerns about todayās humans vs other forms of intelligence, too.
- ^
Another tentative implication that goes without saying, but Iāll say it anyway: review who youāre listening to.
Who got these developments ārightā and āwrongā? How will you weight what those people say in the next 12 months?
Do the olā career transition into AI safety. Probably governance/āpolicy, but with a technical flavour.
(If you need stats, data science, report writing, or management consultancy skillsāplease hit me up!)
Isnāt mechinterp basically setting out to build tools for AI self-improvement?
One of the things people are most worried about is AIs recursively improving themselves. (Whether all people who claim this kind of thing as a red line will actually treat this as a red line is a separate question for another post.)
It seems to me like mechanistic interpretability is basically a really promising avenue for that. Trivial example: Claude decides that the most important thing is being the Golden Gate Bridge. Claude reads up on Anthropicās work, gets access to the relevant tools, and does brain surgery on itself to turn into Golden Gate Bridge Claude.
More meaningfully, it seems like any ability to understand in a fine-grained way whatās going on in a big model could be co-opted by an AI to ālearnā in some way. In general, I think the case that seems most likely soonest is:
-
Learn in-context (e.g. results of experiments, feedback from users, things like weāve recently observed in scheming papers...)
-
Translate this to appropriate adjustments to weights (identified using mechinterp research)
-
Execute those adjustments
Maybe Iām late to this party and everyone was already conceptualising mechinterp as a very dual-use technology, but Iām here now.
Honestly, maybe it leans more towards āoffenseā (i.e., catastrophic misalignment) than defense! It will almost inevitably require automation to be useful, so weāre ceding it to machines out of the gate. Iād expect tomorrowās models to be better placed to make sense of and use of mechinterp techniques than humans areāpartly just because of sheer compute, but also maybe (and now Iām into speculating on stuff I understand even less) the nature of their cognition is more suited to whatās involved.
-
If someone isnāt already doing so, someone should estimate what % of (self-identified?) EAs donate according to our own principles. This would be useful (1) as a heuristic for the extent to which the movement/ācommunity/āwhatever is living up to its own standards, and (1i) assuming the answer is ādecentlyā it would be useful evidence for PR/āpublicity/āresponding to marginal-faith tweets during bouts of criticism.
Looking at the Rethink survey from 2020, they have some info about which causes EAs are giving to but they seem to note that not many people respond on this? And itās not quite the same question. To do: check GWWC for whether they publish anything like this.
Edit to add: maybe an imperfect but simple and quick instrument for this could be something like āFor what fraction of your giving did you attempt a cost-effectiveness assessment (CEA), read a CEA, or rely on someone else who said they did a CEA?ā. I donāt think it actually has to be about whether the respondent got the ārightā result per se; the point is the principles. Deferring to GiveWell seems like living up to the principles because of how they make their recommendations, etc.
Can you add /ā are you comfortable adding anything on who āusā is and which orgs or what kinds of orgs are hesitant? Is your sense this is universal, or more localised (geographically, politically, cause area...)?
Good point and good fact.
My sense, though, is that if you scratch most āexpand the moral circleā statements you find a bit of implicit moral realism. I think generally thereās an unspoken ā...to be closer to its truly appropriate extentā, and that thereās an unspoken assumption that thereāll be a sensible basis for that extent. Maybe some people are making the statement prima facie though. Could make for an interesting survey.
Love to see these reports!
I have two suggestions/ārequests for ācrosstabsā on this info (which is naturally organised by evaluator, because thatās what the project is!):
As-of-today, which evaluators/ācharities sit where on the recommendation scale. The info for that is mostly on GWWCās website but not quite organised as such. Iām thinking of rows for cause areas, columns for buckets, e.g. āRecommendedā at one end and āMaybe not cost-effectiveā at the other (though maybe youād drop things off altogether). Just something to help visualise whatās moved and by how much, and broadly why are things sitting where they are (e.g. THL corporate campaigns sliding off the recommended list for āproceduralā reasons, so not in the Recommended column but now in a āNearlyā column or something).
Iād love a clear checklist of what you think needs improvement per evaluated program to help with making the list a little more evergreen. I think all that info is in your reporting, but if you called it out I think it would
help evaluated programs and
help donors to
get a sense for how up-to-date that recommendation is (given the rotating/ārolling nature of the evaluation program)
and possibly do their own assessment for whether the charity āshouldā be recommended ānowā.
E.g. moās megaprojects link in another comment!