āEA-Adjacentā now I guess.
šø 10% Pledger.
Likes pluralist conceptions of the good.
Dislikes Bay Culture being in control of the future.
JWS šø
Karma: 4,587
Feels like youāve slightly misunderstood my point of view here Lorenzo? Maybe thatās on me for not communicating it clearly enough though.
For what itās worth, Rutger has been donating 10% to effective charities for a while and has advocated for the GWWC pledge many times...So I donāt think heās against that, and lots of people have taken the 10% pledge specifically because of his advocacy
Thatās great! Sounds like very āEAā to me š¤·
I think this mixes effective altruism ideals/āgoals (which everyone agrees with) with EAās specific implementation, movement, culture and community.
Iām not sure everyone does agree really, some people have foundational moral differences. But that aside, I think effective altruism is best understand as a set of ideas/āideals/āgoals. Iāve been arguing that on the Forum for a while and will continue to do so. So I donāt think Iām mixing, I think that the critics are mixing.
This doesnāt mean that theyāre not pointing out very real problems with the movement/ācommunity. I still strongly think that the movement has lot of growing pains/āreforms/ārecknonings to go through before we can heal the damage of FTX and onwards.
The āwin by ipponā was just a jokey reference to Michael Nielsenās āEA judoā phrase, not me advocating for soldier over scout mindset.
If we want millions of people to e.g. give effectively, I think we need to have multiple āmovementsā, āflavoursā or āinterpretationsā of EA projects.
I completely agree! Like 100000% agree! But thatās still āEAā? I just donāt understand trying to draw such a big distinction between SMA and EA in the case where they reference a lot of the same underlying ideas.
So I donāt know, feels like weāre violently agreeing here or something? I didnāt mean to suggest anything otherwise in my original comment, and I even edited it to make it more clear I was more frustrated at the interviewer than anything Rutger said or did (itās possible that a lot of the non-quoted phrasing were put in his mouth)
Just a general note, I think adding some framing of the piece, maybe key quotes, and perhaps your own thoughts as well would improve this from a bare link-post? As for the post itself:
It seems Bregman views EA as:
a misguided movement that sought to weaponize the countryās capitalist engines to protect the planet and the human race
Not really sure how donating ~10% of my income to Global Health and Animal Welfare charities matches that framework tbqh. But yeah āweaponizeā is highly aggressive language here, if you take it out thereās not much wrong with it. Maybe Rutger or the interviewer think Capitalism is inherently bad or something?
effective altruism encourages talented, ambitious young people to embrace their inner capitalist, maximize profits, and then donate those profits to accomplish the maximum amount of good.
Are we really doing the earn-to-give thing again here? But like apart from the snark there isnāt really an argument here, apart from again implicitly associating capitalism with badness. EA people have also warned about the dangers of maximisation before, so this isnāt unknown to the movement.
Bregman saw EAās demise long before the downfall of the movementās poster child, Sam Bankman-Fried
Is this implying that EA is dead (news to me) or that is in terminal decline (arguable, but knowledge of the future is difficult etc etc)?
he [Rutger] says the movement [EA] ultimately āalways felt like moral blackmailing to me: youāre immoral if you donāt save the proverbial child. Weāre trying to build a movement thatās grounded not in guilt but enthusiasm, compassion, and problem-solving.
I mean, this doesnāt sound like an argument against EA or EA ideas? Itās perhaps why Rutger felt put off by the movement, but then if you want a movement based on āenthusiasm, compassion, and problem-solvingā (which are still very EA traits to me, btw), then thatās because it would be doing more good, rather than a movement wracked by guilt. This just falls victim to classic EA Judo, we win by ippon.
I donāt know, maybe Rutger has written up more of his criticism somewhere more thoroughly. Feel like this article is such a weak summary of it though, and just leaves me feeling frustrated. And in a bunch of places, itās really EA! See:
Using Rob Mather founding AMF as a case study (and who has a better EA story than AMF?)
Pointing towards reducing consumption of animals via less meat-eating
Even explicitly admires EAās support for ānon-profit charity entrepreneurshipā
So whereās the EA hate coming from?I think āEA hateā is too strong and is mostly/āactually coming from the interviewer, maybe more than Rutger. Seems Rutger is very disillusioned with the state of EA, but many EAs feel that way too! Pinging @Rutger Bregman or anyone else from the EA Netherlands scene for thoughts, comments, and responses.
With existential risk from unaligned AI, I donāt think anyone has ever told a very clear story about how AI will actually get misaligned, get loose, and kill everyone.
This should be evidence against AI x-risk![1] Even in the atmospheric ignition case in Trinity, they had more concrete models to use. If we canāt build a concrete model here, then it implies we donāt have a concrete/āconvincing case for why it should be prioritised at all, imo. Itās similar to the point in my footnotes that you need to argue for both p and p->q, not just the latter. This is what I would expect to see if the case for p was unconvincing/āincorrect.
I donāt think this is a problem: we shouldnāt expect to know all the details of how things go wrong in advance
Yeah I agree with this. But the uncertainty and cluelessness in the future should decrease oneās confidence that theyāre working on the most important thing in the history of humanity, one would think.
and it is worthwhile to do a lot of preparatory research that might be helpful so that weāre not fumbling through basic things during a critical period. I think the same applies to digital minds.
Iām all in favour of research, but how much should that research get funded? Can it be justified above other potential uses of money and general resource? Should it be an EA priority as defined by the AWDW framing? These we (almost) entirely unargued for.
- ^
Not dispositive evidence perhaps, but a consideration
- ^
It also seems like youāre mostly critiquing the tractability of the claim and not the underlying scale nor neglectedness?
Yep, everyone agrees itās neglected. My strongest critique is the tractability, which may be so low as to discount astronomical value. I do take a lot of issue with the scale as well though. I think that needs to be argued for rather than assumed. I also think trade-offs from other causes need to be taken into account at some point too.
And again, I donāt think thereās no arguments that can make traction on the scale/ātractability that can make AI Welfare look like a valuable cause, but these arguments clearly werenāt made (imho) in AWDW
I donāt quite know what to respond here.[1] If the aim was to discuss something differently then I guess there should have been a different debate prompt? Or maybe it shouldnāt have been framed as a debate at all? Maybe it should have just prioritised AI Welfare as a topic and left it at that. Iād certainly have less of an issue with the posts that were were that have happened, and certainly wouldnāt have been confused by the voting if there wasnāt a voting slider.[2]
Thanks for extensive reply Derek :)
Even if you think that AI welfare is important (which I do!), the field doesnāt have the existing talent pipelines or clear strategy to absorb $50 million in new funding each year.
Yep completely agree here, and as Siebe pointed out I did got to the extreme end of āmake the changes right nowā. It could be structured in more gradual way, and potential from more external funding.
The fact that something might have a huge scale and we might be able to do something about it is enough for it to be taken seriously and provides prima facie evidence that it should be a priority.
I agree in principle on the huge scale point, but much less so the āmight be able to do somethingā. I think we need a lot more than that, we need something tractable to get going, especially for something to be considered a priority. I think the general form of argument Iāve seen this week is that AI Welfare could have a huge scale, therefore it should be an EA priority without much to flesh out the ādo somethingā part.
AI persons (or things that look like AI persons) could easily be here in the next decade...AI people (of some form or other) are not exactly a purely hypothetical technology,
I think I disagree empirically here. Counterfeit āpeopleā might be here soon, but I am not moved much by arguments that digital ālifeā with full agency, self-awareness, autopoiesis, moral values, moral patienhood etc will be here in the next decade. Especially not easily here. I definitely think that case hasnāt been made, and I think (contra Chris in the other thread) that claims of this sort should have been made much more strongly during AWDW.
We might have that opportunity now with AI welfare. Perhaps this means that we only need a small core group, but I do think some people should make it a priority.
Some small people should, I agree. Funding Jeff Sebo and Rob Long? Sounds great. Giving them 438 research assistants and $49M in funding taken from other EA causes? Hell to the naw. We werenāt discussing whether AI Welfare should be a priority for some EAs, we were discussing specific terms set out in the weekās statement, and I feel like Iām the only person during this week who paid any attention to them.
Secondly, the āwe might have that opportunityā is very unconving to me. Itās the same convingness to me of saying in 2008 that āāIf CERN is turned on, it make create a black hole that destroys the world. Nobody else is listening. We might only have the opportunity to act now!ā Itās just not enough to be action-guiding in my opinion.
Iām pretty aware the above is unfair to strong advocates of AI Safety and AI Welfare, but at the moment thatās where the quality of arguments this week have roughly stood from my viewpoint.
I think itās very valuable for you to state what the proposition would mean in concrete terms.
Itās not just concrete terms, itās the terms weāve all agreed to vote on for the past week!
On the other hand, I think itās quite reasonable for posts not spend time engaging with the question of whether āthere will be vast numbers of AIs that are smarter than usā.
I think I just strongly disagree on this point. Not every post has to re-argue everything from the ground up, but I think every post does need at least a link or backing to why it believes that. Are people anchoring on Shulman/āCotra? Metaculus? Cold Takes? General feelings about AI progress? Drawing lines on graphs? Specific claims about the future that making reference only to scaled-up transformer models? These are all very different claims for the proposition, and differ in terms of types of AI, timelines, etc.
AI safety is already one of the main cause areas here and thereās been plenty of discussion about these kinds of points already.
If someone has something new to say on that topic, then itād be great for them to share it, otherwise it makes sense for people to focus on discussing the parts of the topic that have not already been covered as part of the discussions on AI safety.
I again disagree, for two slightly different reasons:
Iām not sure how good the discussion has been about AI Safety. How much have these questions and cruxes actually been internalised? Titotalās excellent series on AI risk scepticism has been under-discussed in my opinion. There are many anecdotal cases of EAs (especially younger, newer ones) simply accepting the importance of AI causes through deference alone.[1] At the latest EAG London, when I talked about AI risk skepticism I found surprising amounts of agreement with my positions even amongst well-known people working in the field of AI risk. There was certainly an interpretation that the Bay/āAI-focused wing of EA werenāt interested in discussing this at all.
Even if something is consensus, it should still be allowed (even encouraged) to be questioned. If EA wants to spend lots of money on AI Welfare (or even AI Safety), it should be very sure that it is one of the best ways we can impact the world. Iād like to see more explicit red-teaming of this in the community, beyond just Garfinkel on the 80k podcast.
- ^
I also met a young uni organiser who was torn about AI risk, since they didnāt really seem to be convinced of it but felt somewhat trapped by the pressure they felt to ātowe the EA lineā on this issue
MakĀing AI Welfare an EA priĀorĀity reĀquires jusĀtifiĀcaĀtions that have not been given
Seems needlessly provocative as a title, and almost purposefully designed to generate more heat than light in the resulting discussion.
I think Iād rather talk about the important topic even if itās harder? My concern is, for example, that the debate happens and letās say people agree and start to pressure for moving $ from GHD to AW. But this ignores a third option, move $ from ālongtermistā work to fund both.
Feels like this is a ālooking under the streetlight because itās easier effectā kind of phenomenon.
If Longtermist/āAI Safety work canāt even to begin to cash out measurable incomes that should be a strong case against it. This is EA, we want the things weāre funding to be effective.
Just to back this up, since Wei has mentioned it, it does seem like a lot of the Open-Phil-cluster is to varying extents bought into illusionism. I think this is a highly controversial view, especially for those outside of Analytical Philosophy of Mind (and even within the field many people argue against it, I basically agree with Galen Strawsonās negative take on it as an entire approach to consciousness).
We have evidence here that Carl is somewhat bought in from the original post here and Weiās comment
The 2017 Report on Consciousness and Moral Patienthood by Muehlhauser assumes illusionism about human consciousness to be true.
Not explicitly in the Open Phil cluster but Keith Frankish was on the Hear This Idea Podcast talking about illusionism (see here). I know itās about introducing the host and their ideas but I think they could have been more upfront about the radical implications about illusionism.[1]
I donāt want to have an argument about phenomenal consciousness in this thread,[2] I just want to point out that it does seem to be potential signs of a consensus on a controversial philosophical premise,[3] perhaps without it being given the scrutiny or justification it deserves.
- ^
It seems to me, to lead to eliminativism, or simply redefine consciousness into something people donāt mean in the same way the Dennett redefines āfree willā into something that many people find unsatisfactory.
- ^
I have cut content and tried to alter my tone to avoid this. If you do want to go 12 rounds of strong illusionism vs qualia realism then by all means send me a DM.
- ^
(that you, dear reader, are not conscious, and that you never have been, and no current or future beings either can or will be)
Why just compare to Global Health here, surely it should be āAnimal Welfare is far more effective per $ than other cause areasā?
Final final edit: Congrats on the ARC-AGI-PUB results, really impressive :)
This will be my final response on this thread, because life is very time consuming and Iām rapidly reaching the point where I need to dive back into the technical literature and stress-test my beliefs and intuitions again. I hope Ryan and any readers have found this exchange useful/āenlightening for seeing two different perspectives hopefully have productive disagreement?
If you found my presentation of the scaling-skeptical position highly unconvincing, Iād recommend following the work and thoughts of Tan Zhi Xuan (find her on X here). One of biggest updates was finding her work after she pushed back on Jacob Steinhardt here, and recently she gave a talk about her approach to Alignment. I urge readers to consider spending much more of their time listening to her than to me about AI.
I feel like this is a pretty strange way to draw the line about what counts as an āLLM solutionā.
I donāt think so? Again, I wouldnāt call CICERO an āLLM solutionā. Surely thereāll be some amount of scaffolding which tips over into the scaffolding being the main thing and the LLM just being a component part? Itās probably all blurry lines for sure, but I think itās important to separate āLLM only systemsā from āsystems that include LLMsā, because itās very easy to conceptual scale up the former but harder to do the latter.
Human skeptic: That wasnāt humans sending someone to the moon that was Humans + Culture + Organizations + Science sending someone to the moon! You see, humans donāt exhibit real intelligence!
I mean, you use this as a reductio, but thatās basically the theory of Distributed Cognition, and also linked to the ideas of ācollective intelligenceā, though thatās definitely not an area Iām an expert in by any means. Also reminds me a lot Chalmers and Clarksā thesis of the Extended Mind.[1]
Of course, I think actual LLM skeptics often donāt answer āNoā to the last question. They often do have something that they think is unlikely to occur with a relatively straightforward scaffold on top of an LLM (a model descended from the current LLM paradigm, perhaps trained with semi-supervised learning and RLHF).
So I canāt speak for Chollet and other LLM skeptics, and I think again LLMs+extra (or extras+LLMs) are a different beast from LLMs on their own and possibly an important crux. Here are some things I donāt think will happen in the near-ish future (on the current paradigm):
I believe an adversarial Imitation Game, where the interrogator is aware of both the AI systemās LLM-based nature and its failure modes, is unlikely to be consistently beaten in the near future.[2]
Primarily-LLM models, in my view, are highly unlikely to exhibit autopoietic behaviour or develop agentic designs independently (i.e. without prompting/ādirection by a human controller).
I donāt anticipate these models exponential increase the rate of scientific research or AI development.[3] Theyāll more likely serve as tools used by scientists and researchers themselves to frame problems, but new and novel problems will still remain difficult and be bottlenecked by the real world + Hofstadterās law.
I donāt anticipate Primarily-LLM models to become good at controlling and manoeuvring robotic bodies in the 3D world. This is especially true in a novel-test-case scenario (if someone could make a physical equivalent of ARC to test this, thatād be great)
This would be even less likely if the scaffolding remained minimal. For instance, if thereās no initial sorting code explicitly stating [IF challenge == turing_test GO TO turing_test_game_module].
Finally, as an anti-RSI operationalisation, the idea of LLM-based models assisting in designing and constructing a Dyson Sphere within 15 years seemsā¦ particularly far-fetched for me.
Iām not sure if this reply was my best, it felt a little all-over-the-place, but we are touching on some deep or complex topics! So Iāll respectfully bow out now, and thank again for the disucssion and giving me so much to think about. I really appreciate it Ryan :)
- ^
Then you get into ideas like embodiment/āenactivism etc
- ^
I can think of a bunch of strategies to win here, but Iām not gonna say so it doesnāt end up in GPT-5 or 6ā²s training data!
- ^
Of course, with a new breakthrough, all bets could be off, but itās also definitionally impossible to predict those, and unrobust to draw straight lines and graphs to predict the future if you think breakthroughs will be need. (Not saying you do this, but some other AIXR people definitely seem to be)
(folding in replies to different sub-comments here)
Sure you can have a very smart quadriplegic who is very knowledgable. But they wonāt do anything until you let them control some actuator.
I think our misunderstanding here is caused by the word do. Sure, Stephen Hawking couldnāt control his limbs, but nevertheless his mind was always working. He kept writing books and papers throughout his life, and his brain was āalways onā. A transformer model is a set of frozen weights that are only āonā when a prompt is entered. Thatās what I mean by āit wonāt do anythingā.
As far as this project, seems extremely implausible to me that the hard part of this project is the scaffolding work I did.
Hmm, maybe weāre differing on what hard works means here! Could be a difference between whatās expensive, time-consuming, etc. Iām not sure this holds for any reasonable scheme, and I definitely think that you deserve a lot of credit for the work youāve done, much more than GPT4o.
I think my results are probably SOTA based on more recent updates.
Congrats! I saw that result and am impressed! Itās definitely clearly SOTA on the ARC-AGI-PUB leaderboard, but the original ā34%->50% in 6 days ARC-AGI breakthroughā claim is still incorrect.
Iāll have to dive into the technical details here I think, but the mystery of in-context learning has certainly shot up my reading list, and I really appreciate that link btw! It seems Blaine has some of the similary a-priori scepticism that I do towards it, but the right way for me to proceed is dive into the empirical side and see if my ideas hold water there.
From the summary page on Open Phil:
In this framework, AGI is developed by improving and scaling up approaches within the current ML paradigm, not by discovering new algorithmic paradigms.
From this presentation about it to GovAI (from April 2023) at 05:10:
So the kinda zoomed out idea behind the Compute-centric framwork is that Iām assuming something like the current paradigm is going to lead to human-level AI and further, and Iām assuming that we get there by scaling up and improving the current algorithmic approaches. So itās going to look like better versions of transformers that are more efficient and that allow for larger context windows...ā
Both of these seem to be pretty scaling-maximalist to me, so I donāt think the quote seems wrong, at least to me? Itād be pretty hard to make a model which includes the possibility of the paradigm not getting us to AGI and then needing a period of exploration across the field to find the other breakthroughs needed.
The solution would be much worse without careful optimization and wouldnāt work at all without gpt4o (or another llm with similar performance).
I can buy that GPT4o would be best, but perhaps other LLMs might reached āokā scores on ARC-AGI if directly swapped out? Iām not sure what you refer to be ācareful optimizationā here though.
There are different analogies here which might be illuminating:
Suppose that you strand a child out in the woods and never teach them anything. I expect they would be much worse at programming. So, some credit for there abilities goes to society and some to their brain.
If you remove my ability to see (on conversely, use fancy tools to make it easier for a blind person to see) this would greatly affect my ability to do ARC-AGI puzzles.
You can build systems around people which remove most of the interesting intelligence from various tasks.
I think what is going on here is analogous to all of these.
On these analogies:
This is an interesting point actually. I suppose credit-assingment for learning is a very difficult problem. In this case though, the child stranded would (hopefully!) survive and make a life for themselves and learn the skills they need to survive. Theyāre active agents using their innate general intelligence to solve novel problems (per chollet). If I put a hard-drive with gpt4oās weights in the forest, itāll just rust. And thatāll happen no matter how big we make that model/āhard-drive imo.[1]
Agreed here, will be very interesting to see how improved multimodality affects ARC-AGI scores. I think that we have interesting cases of humans being able to perform these takes in their head presumably without sight? e.g. Blind Chess Players with high ratings or Mathematicians who can reason without sight. I think Cholletās point in the interview is that they seem to be able to parse the JSON inputs fine in various cases, but still canāt perform generalisation.
Yep I think this is true, and perhaps my greatest fear from delegating power to complex AI systems. This is an empirical question weāll have to find out, can we simply automate away everything humans do/āare needed for through a combination of systems even if each individual part/āmodel used in said system is not intelligent?
Separately, this tweet is relevant: https://āāx.com/āāMaxNadeau_/āāstatus/āā1802774696192246133
Yep saw Maxās comments and think he did a great job on X bringing some clarifications. I still think the hard part is the scaffolding. Money is easy for SanFran VCs to provide, and we know theyāre all fine to scrape-data-first-ask-legal-forgiveness later.
I think thereās a separate point where enough scaffolding + LLM means the resulting AI system is not well described by being an LLM anymore. Take the case of CICERO by Meta. Is that a āscaffolded LLMā? Iād rather describe it as a system which incorporates an LLM as a particular part. Itās harder to naturally scale such a system in the way that you can with the transformer architecuter by stacking more layers or pre-training for longer on more data.
My intuition here is that scaffolding to make a system work well on ARC-AGI would make it less useable on other tasks, so sacrificing generality for specific performance. Perhaps in this case ARC-AGI is best used as a suite of benchmarks, where the same model and scaffolding should be used for each? (Just thinking out loud here)
Final point, Iāve really appreciate your original work, comments on substack/āX/āhere. I do apologise if I didnāt make clear what parts were my personal reflections/āvibes instead of more technical disagreements on interpretationāthese are very complex topics (at least for me) and Iām trying my best to form a good explanation of the various evidence and data we have on this. Regardless of our disagreements on this topic, Iāve learned a lot :)
- ^
Similarly, you can pre-train a model to create weights and get to a humongous size. But it wonāt do anything until you ask it to generate a token. At least, thatās my intuition. Iām quite sceptical of how pre-training a transformer is going to lead to creating a mesa-optimiser
Oh yeah this wasnāt against you at all! I think youāre a great researcher, and an excellent interlocutor, and I learn a lot (and am learning a lot) from both your work and your reactions to my reaction.[1] Point five was very much a reaction against a āvibeā I saw in the wake of your results being published.
Like letās take Buckās tweet for example. We know now that a) your results arenāt technically SOTA and b) Itās not an LLM solution, itās an LLM + your scaffolding + program search, and I think thatās importantly not the same thing.
- ^
I sincerely hope my post + comments have been somewhat more stimulating than frustrating for you
- ^
At the moment I think ARC-AGI does a good job at showing the limitations of transformer models on simple tasks that they donāt come across in their training set. I think if the score was claimed, weād want to see how it came about. It might be through frontier models demonstrating true understanding, but it might through shortcut learning/ādata leakage/āimpressive but overly specific and intuitively unsatisfying solution.
If ARC-AGI were to be broken (within the constraints Chollet and Knoop place on it) Iād definitely change my opinions, but what theyād change to depends on the matter of how ARC-AGI was solved. Thatās all Iām trying to say in that section (perhaps poorly)
No really appreciated it your perspective, both on SMA and what we mean when we talk about āEAā. Definitely has given me some good for thought :)