âEA-Adjacentâ now I guess.
đ¸ 10% Pledger.
Likes pluralist conceptions of the good.
Dislikes Bay Culture being in control of the future.
JWS đ¸
Karma: 4,759
Folding in Responses here
@thoth hermes (or https://ââx.com/ââthoth_iv if someone can get it to them if youâre Twitter friends then pls go ahead.[1] Iâm responding to this thread hereâI am not saying âthat EA is losing the memetic war because of its high epistemic standardsâ, in fact quite the opposite r.e. AI Safety, and maybe because of misunderstanding of how politics work/ânot caring about the social perception of the movement. My reply to Iyngkarran below fleshes it out a bit more, but if thereâs a way for you to get in touch directly, Iâd love to clarify what I think, and also hear your thoughts more. But I think I was trying to come from a similar place that Richard Ngo is, and many of his comments on the LessWrong thread here very much chime with my own point-of-view. What I am trying to push for is the AI Safety movement reflecting on losing ground memetically and then asking âwhy is that? what are we getting wrong?â rather than doubling down into lowest-common denominator communication. I think we actually agree here? Maybe I didnât make that clear enough in my OP though.
@Iyngkarran KumarâThanks for sharing your thoughts, but I must say that I disagree with it. I donât think that the epistemic standards are working against us by being too polite, quite the opposite. I think the epistemic standards in AI Safety have been too low relative to the attempts to wield power. If you are potentialy going to criminalise existing Open-Source models,[2] you better bring the epistemic goods. And for many people in the AI Safety field, the goods have not been brought (which is why I see people like Jeremy Howard, Sara Hooker, Rohit Krishnan etc get increasingly frustrated by the AI Safety field). This is on the field of AI Safety imo for not being more persuasive. If the AI Safety field was right, the arguments would have been more convincing. I think, while itâs good for Eliezer to say what he thinks accurately, the âbomb the datacentersâ[3] piece has probably been harmful for AI Safetyâs cause, and things like it a very liable to turn people away from supporting AI Safety. I also donât think itâs good to say that itâs a claim of âwhat we believeâ, as I donât really agree with Eliezer on much.
(r.e. inside vs outside game, see this post from Holly Elmore)
@anormative/â @David MathersâYeah itâs difficult to manage the exact hypothesis here, especially for falsified preferences. Iâm pretty sure SV is âliberalâ overall, but I wouldnât be surprised if Trump % is greater than 16 and 20, and it definitely seems to be a lot more open this time, e.g. a16z and Musk openly endorsing Trump, Sequoia Capital partners claiming that Biden dropping out was worse than the Jan 6th riot. Things seem very different this time around, different enough to be paid attention to.
- - - - - - - - - - - -
Once again, if you disagree, Iâd love to actually here why. Up/âdown voting is a crude feedback to, and discussion of ideas leads to much quicker sharing of knowledge. If you want to respond but donât want to publicly, then by all means please send a DM :)
- ^
I donât have Twitter and think itâd be harmful for my epistemic & mental health if I did get an account and become immersed in âThe Discourseâ
- ^
This piece from @1a3orn is excellent and to absence of evidence of good arguments against it is evidence of the absence of said arguments. (tl;drâAI Safety people, engage with 1a3orn more!)
- ^
I know thatâs not what it literally says but itâs what people know it as
- ^
Quick[1] thoughts on the Silicon Valley âVibe-Shiftâ
I wanted to get this idea out of my head and into a quick-take. I think thereâs something here, but a lot more to say, and Iâve really havenât done the in-depth research for it. There was a longer post idea I had for this, but honestly diving more than I have here into it is not a good use of my life I think.
The political outlook in Silicon Valley has changed.
Since the attempted assassination attempt on President Trump, the mood in Silicon Valley has changed. There have been open endorsements, e/âacc has claimed political victory, and lots of people have noticed the âvibe shiftâ.[2] I think that, rather than this being a change in opinions, itâs more an event allowing for the beginning of a preference cascade, but at least in Silicon Valley (if not yet reflected in national polling) it has happened.
So it seems that a large section of Silicon Valley is now openly and confidently supporting Trump, and to a greater or lesser extent aligned with the a16z/âe-acc worldview,[3] we know itâs already reached the ears of VP candidate JD Vance.
How did we get here
You could probably write a book on this, so this is a highly opinionated take. But I think this is somewhat, though not exclusively, an own goal of the AI Safety movement.
As ChatGPT starts to bring AI, and AI Safety, into the mainstream discourse, the e/âacc countermovement begins. It positions itself as opposite effective altruism, especially in the wake of SBF.
Guillaume Verdon, under the alias âBeff Jezosâ, realises the memetic weakness of the AI Safety movement and launches a full memetic war against it. Regardless of his rightness or wrongness, you do to some extent got to hand it to him. Heâs like right-wing Ămile Torres, ambitious and relentless and driven by ideological zeal against a hated foe.
Memetic war is total war. This means nuance dies to get it to spread. I donât know if, for example, Marc Andreessen actually thinks antimalarial bednets are a âtriple threatâ of badness, but itâs a war and you donât take prisoners. Does Beff think that people running a uni-group session on Animal Welfare are âbasically terroristsâ, I donât know. But EA is the enemy, and the enemy must be defeated, and the war is total.
The OpenAI board fiasco is, I think, a critical moment here. It doesnât matter what the reasoning weâve come out with at the end of the day was, I think it was perceived as âa doomer coupâ and it did radicalize the valley. In his recent post Richard Ngo called on the AI Safety movement to show more legitimacy and competence. The board fiasco torpedoed my trust in the legitimacy and competence of many senior AI safety people, so god knows how strong the update was for Silicon Valley as a whole.
As some evidence this is known in EA circles, I think this is exactly what Dwarkesh is alluding to when asked âwhat happened to the EA brandâ. For many people in Silicon Valley, I think the answer is that it got thrown in the dustbin of history.
This new movement became increasingly right-wing coded. Partly as a response to the culture wars in America and the increasing vitriol thrown by the left against âtech brosâ, partly as a response to the California Ideology being threatened by any sense of AI oversight or regulation, and partly because EA is the enemy and EA was being increasingly seen by this group as left-wing, woke, or part of the Democratic Party due to the funding patterns of SBF and Moskovitz. I think this has led, fairly predictably, to the right-ward shift in SV and direct political affiliation with a (prospective) second Trump presidency
Across all of this my impression is that, just like with Torres, there was little to no direct pushback. I can understand not wanting to be dragged into a memetic war, or to be involved in the darker parts of Twitter discourse. But the e-acc/âtechnooptimist/âRW-Silicon-Valley movement was being driven by something, and I donât think AI Safety ever really argued against it convincingly, and definitely not in a convincing enough way to âwinâ the memetic war. Like, the a16z cluster literally lied to Congress and to Parliament, but nothing much come of that fact.
I think this is very much linked to playing a strong âinside gameâ to access the halls of power and no âoutside gameâ to gain legitimacy for that use of power. Itâs also I think due to EA not wanting to use social media to make its case, whereas the e-acc cluster was born and lives on social media.
Where are we now?
Iâm not a part of the Bay Area scene and culture,[4] but it seems to me that the AI Safety movement has lost the âmandate of heavenâ to whatever extent it did have it. SB-1047 is a push to change policy that has resulted in backlash, and may result in further polarisation and counter-attempts to fight back in a zero-sum political game. I donât know if itâs constitutional for a Trump/âVance administration to use the Supremacy Clause to void SB-1047 but I donât doubt that they might try. Bidenâs executive order seems certain for the chopping block. I expect a Trump administration to be a lot less sympathetic to the Bay Area/âDC AI Safety movements, and the right-wing part of Silicon Valley will be at the very least energised to fight back harder.
One concerning thing for both Silicon Valley and the AI Safety movement is what happens as a result of the ideological consequences of SV accepting this trend. Already a strong fault-line is the extreme social conservatism and incipient nationalism brought about by this. In the recent a16z podcast, Ben Horowitz literally accuses the Biden administration of breaking the rule of law, and says nothing about Trump literally refusing to concede the 2020 election and declaring that there was electoral fraud. Mike Solana seems to think that all risks of democratic backsliding under a Trump administration were/âare overblown (or at least that people in the Bay agreeing was preference falsification). On the Moments-of-Zen Podcast (which has also hosted Curtis Yarvin twice), Balaji Srinivasan accused the âBlue Tribeâ of ethnically cleansing him out of SF[5] and called on the grey tribe to push all the blues out of SF. e-acc sympathetic people are noting that anti-trans ideas bubbling up in the new movement. You cannot seriously engage with ideas and shape them without those ideas changing you.[6] This right-wing shift will have further consequences, especially under a second Trump presidency.
What next for the AI Safety field?
I think this is a bad sign for the field of AI Safety. Political polarisation has escaped AI for a while. Current polls may lean in support , but polls and political support are fickle, especially in the age of hyper-polarisation.[7] I feel like my fears around the perception of Open Philanthropy are re-occuring here but for the AI Safety movement at large.
I think the consistent defeats to the e-acc school and the fact that the tech sector as a whole seems very much unconvinced by the arguments for AI Safety should at some point lead to a reflection from the movement. Where you stand on this very much depends on your object-level beliefs. While this is a lot of e-acc discourse around transhumanism, replacing humanity, and the AI eschaton, I donât really buy it. I think that they donât think ASI is possible soon, and thus all arguments for AI Safety are bunk. Now, while the tech sector as a whole might not be as hostile, they donât seem at all convinced of the âASI-soonâ idea.
A key point I want to emphasise is that one cannot expect to wield power successfully without also having legitimacy.[8] And to the extent that the AI Safety movementâs strategy is trying to thread this needle it will fail.
Anyway, long ramble over, and given this was basically a one-shot ramble it will have many inaccuracies and flaws. Nevertheless I hope that it can be directionally useful and lead to productive discussion.
- ^
lol, lmao
- ^
- ^
Would be very interested to hear the thoughts of people in the Bay on this
- ^
And if invited to be I would almost certainly decline,
- ^
He literally used the phrase âethnically cleanseâ. This is extraordinarily dangerous language in a political context.
- ^
A good example in fiction is in Warhammer40K, where Horus originally accepts the power of Chaos to fight against Imperial Tyranny, but ends up turning into their slave.
- ^
Due to polarisation, views can dramatically shift on even major topics such as the economy and national security (i know these are messy examples!). Current poll leads for AI regulation should not, in any way, be considered secure
- ^
I guess you could also have overwhelming might and force, but even that requires legitimacy. Caesar needed to be seen as legitimate by Marc Anthony, Alexander didnât have the legitimacy to get his army to cross the Hyphasis etc.
No really appreciated it your perspective, both on SMA and what we mean when we talk about âEAâ. Definitely has given me some good for thought :)
Feels like youâve slightly misunderstood my point of view here Lorenzo? Maybe thatâs on me for not communicating it clearly enough though.
For what itâs worth, Rutger has been donating 10% to effective charities for a while and has advocated for the GWWC pledge many times...So I donât think heâs against that, and lots of people have taken the 10% pledge specifically because of his advocacy
Thatâs great! Sounds like very âEAâ to me đ¤ˇ
I think this mixes effective altruism ideals/âgoals (which everyone agrees with) with EAâs specific implementation, movement, culture and community.
Iâm not sure everyone does agree really, some people have foundational moral differences. But that aside, I think effective altruism is best understand as a set of ideas/âideals/âgoals. Iâve been arguing that on the Forum for a while and will continue to do so. So I donât think Iâm mixing, I think that the critics are mixing.
This doesnât mean that theyâre not pointing out very real problems with the movement/âcommunity. I still strongly think that the movement has lot of growing pains/âreforms/ârecknonings to go through before we can heal the damage of FTX and onwards.
The âwin by ipponâ was just a jokey reference to Michael Nielsenâs âEA judoâ phrase, not me advocating for soldier over scout mindset.
If we want millions of people to e.g. give effectively, I think we need to have multiple âmovementsâ, âflavoursâ or âinterpretationsâ of EA projects.
I completely agree! Like 100000% agree! But thatâs still âEAâ? I just donât understand trying to draw such a big distinction between SMA and EA in the case where they reference a lot of the same underlying ideas.
So I donât know, feels like weâre violently agreeing here or something? I didnât mean to suggest anything otherwise in my original comment, and I even edited it to make it more clear I was more frustrated at the interviewer than anything Rutger said or did (itâs possible that a lot of the non-quoted phrasing were put in his mouth)
Just a general note, I think adding some framing of the piece, maybe key quotes, and perhaps your own thoughts as well would improve this from a bare link-post? As for the post itself:
It seems Bregman views EA as:
a misguided movement that sought to weaponize the countryâs capitalist engines to protect the planet and the human race
Not really sure how donating ~10% of my income to Global Health and Animal Welfare charities matches that framework tbqh. But yeah âweaponizeâ is highly aggressive language here, if you take it out thereâs not much wrong with it. Maybe Rutger or the interviewer think Capitalism is inherently bad or something?
effective altruism encourages talented, ambitious young people to embrace their inner capitalist, maximize profits, and then donate those profits to accomplish the maximum amount of good.
Are we really doing the earn-to-give thing again here? But like apart from the snark there isnât really an argument here, apart from again implicitly associating capitalism with badness. EA people have also warned about the dangers of maximisation before, so this isnât unknown to the movement.
Bregman saw EAâs demise long before the downfall of the movementâs poster child, Sam Bankman-Fried
Is this implying that EA is dead (news to me) or that is in terminal decline (arguable, but knowledge of the future is difficult etc etc)?
he [Rutger] says the movement [EA] ultimately âalways felt like moral blackmailing to me: youâre immoral if you donât save the proverbial child. Weâre trying to build a movement thatâs grounded not in guilt but enthusiasm, compassion, and problem-solving.
I mean, this doesnât sound like an argument against EA or EA ideas? Itâs perhaps why Rutger felt put off by the movement, but then if you want a movement based on âenthusiasm, compassion, and problem-solvingâ (which are still very EA traits to me, btw), then thatâs because it would be doing more good, rather than a movement wracked by guilt. This just falls victim to classic EA Judo, we win by ippon.
I donât know, maybe Rutger has written up more of his criticism somewhere more thoroughly. Feel like this article is such a weak summary of it though, and just leaves me feeling frustrated. And in a bunch of places, itâs really EA! See:
Using Rob Mather founding AMF as a case study (and who has a better EA story than AMF?)
Pointing towards reducing consumption of animals via less meat-eating
Even explicitly admires EAâs support for ânon-profit charity entrepreneurshipâ
So whereâs the EA hate coming from?I think âEA hateâ is too strong and is mostly/âactually coming from the interviewer, maybe more than Rutger. Seems Rutger is very disillusioned with the state of EA, but many EAs feel that way too! Pinging @Rutger Bregman or anyone else from the EA Netherlands scene for thoughts, comments, and responses.
With existential risk from unaligned AI, I donât think anyone has ever told a very clear story about how AI will actually get misaligned, get loose, and kill everyone.
This should be evidence against AI x-risk![1] Even in the atmospheric ignition case in Trinity, they had more concrete models to use. If we canât build a concrete model here, then it implies we donât have a concrete/âconvincing case for why it should be prioritised at all, imo. Itâs similar to the point in my footnotes that you need to argue for both p and p->q, not just the latter. This is what I would expect to see if the case for p was unconvincing/âincorrect.
I donât think this is a problem: we shouldnât expect to know all the details of how things go wrong in advance
Yeah I agree with this. But the uncertainty and cluelessness in the future should decrease oneâs confidence that theyâre working on the most important thing in the history of humanity, one would think.
and it is worthwhile to do a lot of preparatory research that might be helpful so that weâre not fumbling through basic things during a critical period. I think the same applies to digital minds.
Iâm all in favour of research, but how much should that research get funded? Can it be justified above other potential uses of money and general resource? Should it be an EA priority as defined by the AWDW framing? These we (almost) entirely unargued for.
- ^
Not dispositive evidence perhaps, but a consideration
- ^
It also seems like youâre mostly critiquing the tractability of the claim and not the underlying scale nor neglectedness?
Yep, everyone agrees itâs neglected. My strongest critique is the tractability, which may be so low as to discount astronomical value. I do take a lot of issue with the scale as well though. I think that needs to be argued for rather than assumed. I also think trade-offs from other causes need to be taken into account at some point too.
And again, I donât think thereâs no arguments that can make traction on the scale/âtractability that can make AI Welfare look like a valuable cause, but these arguments clearly werenât made (imho) in AWDW
I donât quite know what to respond here.[1] If the aim was to discuss something differently then I guess there should have been a different debate prompt? Or maybe it shouldnât have been framed as a debate at all? Maybe it should have just prioritised AI Welfare as a topic and left it at that. Iâd certainly have less of an issue with the posts that were were that have happened, and certainly wouldnât have been confused by the voting if there wasnât a voting slider.[2]
Thanks for extensive reply Derek :)
Even if you think that AI welfare is important (which I do!), the field doesnât have the existing talent pipelines or clear strategy to absorb $50 million in new funding each year.
Yep completely agree here, and as Siebe pointed out I did got to the extreme end of âmake the changes right nowâ. It could be structured in more gradual way, and potential from more external funding.
The fact that something might have a huge scale and we might be able to do something about it is enough for it to be taken seriously and provides prima facie evidence that it should be a priority.
I agree in principle on the huge scale point, but much less so the âmight be able to do somethingâ. I think we need a lot more than that, we need something tractable to get going, especially for something to be considered a priority. I think the general form of argument Iâve seen this week is that AI Welfare could have a huge scale, therefore it should be an EA priority without much to flesh out the âdo somethingâ part.
AI persons (or things that look like AI persons) could easily be here in the next decade...AI people (of some form or other) are not exactly a purely hypothetical technology,
I think I disagree empirically here. Counterfeit âpeopleâ might be here soon, but I am not moved much by arguments that digital âlifeâ with full agency, self-awareness, autopoiesis, moral values, moral patienhood etc will be here in the next decade. Especially not easily here. I definitely think that case hasnât been made, and I think (contra Chris in the other thread) that claims of this sort should have been made much more strongly during AWDW.
We might have that opportunity now with AI welfare. Perhaps this means that we only need a small core group, but I do think some people should make it a priority.
Some small people should, I agree. Funding Jeff Sebo and Rob Long? Sounds great. Giving them 438 research assistants and $49M in funding taken from other EA causes? Hell to the naw. We werenât discussing whether AI Welfare should be a priority for some EAs, we were discussing specific terms set out in the weekâs statement, and I feel like Iâm the only person during this week who paid any attention to them.
Secondly, the âwe might have that opportunityâ is very unconving to me. Itâs the same convingness to me of saying in 2008 that ââIf CERN is turned on, it make create a black hole that destroys the world. Nobody else is listening. We might only have the opportunity to act now!â Itâs just not enough to be action-guiding in my opinion.
Iâm pretty aware the above is unfair to strong advocates of AI Safety and AI Welfare, but at the moment thatâs where the quality of arguments this week have roughly stood from my viewpoint.
I think itâs very valuable for you to state what the proposition would mean in concrete terms.
Itâs not just concrete terms, itâs the terms weâve all agreed to vote on for the past week!
On the other hand, I think itâs quite reasonable for posts not spend time engaging with the question of whether âthere will be vast numbers of AIs that are smarter than usâ.
I think I just strongly disagree on this point. Not every post has to re-argue everything from the ground up, but I think every post does need at least a link or backing to why it believes that. Are people anchoring on Shulman/âCotra? Metaculus? Cold Takes? General feelings about AI progress? Drawing lines on graphs? Specific claims about the future that making reference only to scaled-up transformer models? These are all very different claims for the proposition, and differ in terms of types of AI, timelines, etc.
AI safety is already one of the main cause areas here and thereâs been plenty of discussion about these kinds of points already.
If someone has something new to say on that topic, then itâd be great for them to share it, otherwise it makes sense for people to focus on discussing the parts of the topic that have not already been covered as part of the discussions on AI safety.
I again disagree, for two slightly different reasons:
Iâm not sure how good the discussion has been about AI Safety. How much have these questions and cruxes actually been internalised? Titotalâs excellent series on AI risk scepticism has been under-discussed in my opinion. There are many anecdotal cases of EAs (especially younger, newer ones) simply accepting the importance of AI causes through deference alone.[1] At the latest EAG London, when I talked about AI risk skepticism I found surprising amounts of agreement with my positions even amongst well-known people working in the field of AI risk. There was certainly an interpretation that the Bay/âAI-focused wing of EA werenât interested in discussing this at all.
Even if something is consensus, it should still be allowed (even encouraged) to be questioned. If EA wants to spend lots of money on AI Welfare (or even AI Safety), it should be very sure that it is one of the best ways we can impact the world. Iâd like to see more explicit red-teaming of this in the community, beyond just Garfinkel on the 80k podcast.
- ^
I also met a young uni organiser who was torn about AI risk, since they didnât really seem to be convinced of it but felt somewhat trapped by the pressure they felt to âtowe the EA lineâ on this issue
Seems needlessly provocative as a title, and almost purposefully designed to generate more heat than light in the resulting discussion.
I think Iâd rather talk about the important topic even if itâs harder? My concern is, for example, that the debate happens and letâs say people agree and start to pressure for moving $ from GHD to AW. But this ignores a third option, move $ from âlongtermistâ work to fund both.
Feels like this is a âlooking under the streetlight because itâs easier effectâ kind of phenomenon.
If Longtermist/âAI Safety work canât even to begin to cash out measurable incomes that should be a strong case against it. This is EA, we want the things weâre funding to be effective.
Just to back this up, since Wei has mentioned it, it does seem like a lot of the Open-Phil-cluster is to varying extents bought into illusionism. I think this is a highly controversial view, especially for those outside of Analytical Philosophy of Mind (and even within the field many people argue against it, I basically agree with Galen Strawsonâs negative take on it as an entire approach to consciousness).
We have evidence here that Carl is somewhat bought in from the original post here and Weiâs comment
The 2017 Report on Consciousness and Moral Patienthood by Muehlhauser assumes illusionism about human consciousness to be true.
Not explicitly in the Open Phil cluster but Keith Frankish was on the Hear This Idea Podcast talking about illusionism (see here). I know itâs about introducing the host and their ideas but I think they could have been more upfront about the radical implications about illusionism.[1]
I donât want to have an argument about phenomenal consciousness in this thread,[2] I just want to point out that it does seem to be potential signs of a consensus on a controversial philosophical premise,[3] perhaps without it being given the scrutiny or justification it deserves.
- ^
It seems to me, to lead to eliminativism, or simply redefine consciousness into something people donât mean in the same way the Dennett redefines âfree willâ into something that many people find unsatisfactory.
- ^
I have cut content and tried to alter my tone to avoid this. If you do want to go 12 rounds of strong illusionism vs qualia realism then by all means send me a DM.
- ^
(that you, dear reader, are not conscious, and that you never have been, and no current or future beings either can or will be)
Why just compare to Global Health here, surely it should be âAnimal Welfare is far more effective per $ than other cause areasâ?
Final final edit: Congrats on the ARC-AGI-PUB results, really impressive :)
This will be my final response on this thread, because life is very time consuming and Iâm rapidly reaching the point where I need to dive back into the technical literature and stress-test my beliefs and intuitions again. I hope Ryan and any readers have found this exchange useful/âenlightening for seeing two different perspectives hopefully have productive disagreement?
If you found my presentation of the scaling-skeptical position highly unconvincing, Iâd recommend following the work and thoughts of Tan Zhi Xuan (find her on X here). One of biggest updates was finding her work after she pushed back on Jacob Steinhardt here, and recently she gave a talk about her approach to Alignment. I urge readers to consider spending much more of their time listening to her than to me about AI.
I feel like this is a pretty strange way to draw the line about what counts as an âLLM solutionâ.
I donât think so? Again, I wouldnât call CICERO an âLLM solutionâ. Surely thereâll be some amount of scaffolding which tips over into the scaffolding being the main thing and the LLM just being a component part? Itâs probably all blurry lines for sure, but I think itâs important to separate âLLM only systemsâ from âsystems that include LLMsâ, because itâs very easy to conceptual scale up the former but harder to do the latter.
Human skeptic: That wasnât humans sending someone to the moon that was Humans + Culture + Organizations + Science sending someone to the moon! You see, humans donât exhibit real intelligence!
I mean, you use this as a reductio, but thatâs basically the theory of Distributed Cognition, and also linked to the ideas of âcollective intelligenceâ, though thatâs definitely not an area Iâm an expert in by any means. Also reminds me a lot Chalmers and Clarksâ thesis of the Extended Mind.[1]
Of course, I think actual LLM skeptics often donât answer âNoâ to the last question. They often do have something that they think is unlikely to occur with a relatively straightforward scaffold on top of an LLM (a model descended from the current LLM paradigm, perhaps trained with semi-supervised learning and RLHF).
So I canât speak for Chollet and other LLM skeptics, and I think again LLMs+extra (or extras+LLMs) are a different beast from LLMs on their own and possibly an important crux. Here are some things I donât think will happen in the near-ish future (on the current paradigm):
I believe an adversarial Imitation Game, where the interrogator is aware of both the AI systemâs LLM-based nature and its failure modes, is unlikely to be consistently beaten in the near future.[2]
Primarily-LLM models, in my view, are highly unlikely to exhibit autopoietic behaviour or develop agentic designs independently (i.e. without prompting/âdirection by a human controller).
I donât anticipate these models exponential increase the rate of scientific research or AI development.[3] Theyâll more likely serve as tools used by scientists and researchers themselves to frame problems, but new and novel problems will still remain difficult and be bottlenecked by the real world + Hofstadterâs law.
I donât anticipate Primarily-LLM models to become good at controlling and manoeuvring robotic bodies in the 3D world. This is especially true in a novel-test-case scenario (if someone could make a physical equivalent of ARC to test this, thatâd be great)
This would be even less likely if the scaffolding remained minimal. For instance, if thereâs no initial sorting code explicitly stating [IF challenge == turing_test GO TO turing_test_game_module].
Finally, as an anti-RSI operationalisation, the idea of LLM-based models assisting in designing and constructing a Dyson Sphere within 15 years seems⌠particularly far-fetched for me.
Iâm not sure if this reply was my best, it felt a little all-over-the-place, but we are touching on some deep or complex topics! So Iâll respectfully bow out now, and thank again for the disucssion and giving me so much to think about. I really appreciate it Ryan :)
- ^
Then you get into ideas like embodiment/âenactivism etc
- ^
I can think of a bunch of strategies to win here, but Iâm not gonna say so it doesnât end up in GPT-5 or 6â˛s training data!
- ^
Of course, with a new breakthrough, all bets could be off, but itâs also definitionally impossible to predict those, and unrobust to draw straight lines and graphs to predict the future if you think breakthroughs will be need. (Not saying you do this, but some other AIXR people definitely seem to be)
(folding in replies to different sub-comments here)
Sure you can have a very smart quadriplegic who is very knowledgable. But they wonât do anything until you let them control some actuator.
I think our misunderstanding here is caused by the word do. Sure, Stephen Hawking couldnât control his limbs, but nevertheless his mind was always working. He kept writing books and papers throughout his life, and his brain was âalways onâ. A transformer model is a set of frozen weights that are only âonâ when a prompt is entered. Thatâs what I mean by âit wonât do anythingâ.
As far as this project, seems extremely implausible to me that the hard part of this project is the scaffolding work I did.
Hmm, maybe weâre differing on what hard works means here! Could be a difference between whatâs expensive, time-consuming, etc. Iâm not sure this holds for any reasonable scheme, and I definitely think that you deserve a lot of credit for the work youâve done, much more than GPT4o.
I think my results are probably SOTA based on more recent updates.
Congrats! I saw that result and am impressed! Itâs definitely clearly SOTA on the ARC-AGI-PUB leaderboard, but the original â34%->50% in 6 days ARC-AGI breakthroughâ claim is still incorrect.
Iâll have to dive into the technical details here I think, but the mystery of in-context learning has certainly shot up my reading list, and I really appreciate that link btw! It seems Blaine has some of the similary a-priori scepticism that I do towards it, but the right way for me to proceed is dive into the empirical side and see if my ideas hold water there.
From the summary page on Open Phil:
In this framework, AGI is developed by improving and scaling up approaches within the current ML paradigm, not by discovering new algorithmic paradigms.
From this presentation about it to GovAI (from April 2023) at 05:10:
So the kinda zoomed out idea behind the Compute-centric framwork is that Iâm assuming something like the current paradigm is going to lead to human-level AI and further, and Iâm assuming that we get there by scaling up and improving the current algorithmic approaches. So itâs going to look like better versions of transformers that are more efficient and that allow for larger context windows...â
Both of these seem to be pretty scaling-maximalist to me, so I donât think the quote seems wrong, at least to me? Itâd be pretty hard to make a model which includes the possibility of the paradigm not getting us to AGI and then needing a period of exploration across the field to find the other breakthroughs needed.
The solution would be much worse without careful optimization and wouldnât work at all without gpt4o (or another llm with similar performance).
I can buy that GPT4o would be best, but perhaps other LLMs might reached âokâ scores on ARC-AGI if directly swapped out? Iâm not sure what you refer to be âcareful optimizationâ here though.
There are different analogies here which might be illuminating:
Suppose that you strand a child out in the woods and never teach them anything. I expect they would be much worse at programming. So, some credit for there abilities goes to society and some to their brain.
If you remove my ability to see (on conversely, use fancy tools to make it easier for a blind person to see) this would greatly affect my ability to do ARC-AGI puzzles.
You can build systems around people which remove most of the interesting intelligence from various tasks.
I think what is going on here is analogous to all of these.
On these analogies:
This is an interesting point actually. I suppose credit-assingment for learning is a very difficult problem. In this case though, the child stranded would (hopefully!) survive and make a life for themselves and learn the skills they need to survive. Theyâre active agents using their innate general intelligence to solve novel problems (per chollet). If I put a hard-drive with gpt4oâs weights in the forest, itâll just rust. And thatâll happen no matter how big we make that model/âhard-drive imo.[1]
Agreed here, will be very interesting to see how improved multimodality affects ARC-AGI scores. I think that we have interesting cases of humans being able to perform these takes in their head presumably without sight? e.g. Blind Chess Players with high ratings or Mathematicians who can reason without sight. I think Cholletâs point in the interview is that they seem to be able to parse the JSON inputs fine in various cases, but still canât perform generalisation.
Yep I think this is true, and perhaps my greatest fear from delegating power to complex AI systems. This is an empirical question weâll have to find out, can we simply automate away everything humans do/âare needed for through a combination of systems even if each individual part/âmodel used in said system is not intelligent?
Separately, this tweet is relevant: https://ââx.com/ââMaxNadeau_/ââstatus/ââ1802774696192246133
Yep saw Maxâs comments and think he did a great job on X bringing some clarifications. I still think the hard part is the scaffolding. Money is easy for SanFran VCs to provide, and we know theyâre all fine to scrape-data-first-ask-legal-forgiveness later.
I think thereâs a separate point where enough scaffolding + LLM means the resulting AI system is not well described by being an LLM anymore. Take the case of CICERO by Meta. Is that a âscaffolded LLMâ? Iâd rather describe it as a system which incorporates an LLM as a particular part. Itâs harder to naturally scale such a system in the way that you can with the transformer architecuter by stacking more layers or pre-training for longer on more data.
My intuition here is that scaffolding to make a system work well on ARC-AGI would make it less useable on other tasks, so sacrificing generality for specific performance. Perhaps in this case ARC-AGI is best used as a suite of benchmarks, where the same model and scaffolding should be used for each? (Just thinking out loud here)
Final point, Iâve really appreciate your original work, comments on substack/âX/âhere. I do apologise if I didnât make clear what parts were my personal reflections/âvibes instead of more technical disagreements on interpretationâthese are very complex topics (at least for me) and Iâm trying my best to form a good explanation of the various evidence and data we have on this. Regardless of our disagreements on this topic, Iâve learned a lot :)
- ^
Similarly, you can pre-train a model to create weights and get to a humongous size. But it wonât do anything until you ask it to generate a token. At least, thatâs my intuition. Iâm quite sceptical of how pre-training a transformer is going to lead to creating a mesa-optimiser
Itâs an unfortunate naming clash, there are different ARC Challenges:
ARC-AGI (Chollet et al) - https://ââgithub.com/ââfchollet/ââARC-AGI
ARC (AI2 Reasoning Challenge) - https://ââallenai.org/ââdata/ââarc
These benchmarks are reporting the second of the two.
LLMs (at least without scaffolding) still do badly on ARC, and Iâd wager Llama 405B still doesnât do well on the ARC-AGI challenge, and itâs telling that all the big labs release the 95%+ number they get on AI2-ARC, and not whatever default result they get with ARC-AGI...
(Or in general, reporting benchmarks where they can go OMG SOTA!!!! and not helpfully advance the general understanding of what models can do and how far they generalise. Basically, traditional benchmark cards should be seen as the AI equivalent of âIN MICEâ)