At least for a moment in time, I was GWWCâs 10,000th active pledger.
zeshenđ¸
Karma: 228
AgreedâI shouldâve made it clearer in the title that I was referring specifically to the AI safety people in EA, i.e. this excludes other EAs not in AI safety, and also excludes other non-EAs in AI safety.
I would be interested to hear the counterpoints from those who have Disagree-voted on this post.
Likewise!
Do you think the âvery particular worldviewâ you describe is found equally among those working on technical AI safety and AI governance/âpolicy? My impression is that policy inherently requires thinking through concrete pathways of how AGI would lead to actual harm as well as greater engagement with people outside of AI safety.
I think theyâre quite prevalent regardless. While some peopleâs roles indeed require them to analyze concrete pathways more than others, the foundation of their analysis is often implicitly built upon this worldview in the first place. The result is that their concrete pathways tend to be centred around some kind of misaligned AGI, just in much more detail. Conversely, someone with a very different worldview who does such an analysis might end up with concrete pathways centred around severe discrimination of marginalized groups.
I have also noticed a split between the âsuperintelligence will kill us allâ worldview (which you seem to be describing) and âregardless of whether superintelligence kills us all, AGI/âTAI will be very disruptive and we need to manage those risksâ (which seemed to be more along the lines of the Will MacAskill post you linked toâespecially as he talks about directing people to causes other than technical safety or safety governance).
There are indeed many different âsub-worldviewsâ, and I was kind of lumping them all under one big umbrella. To me, the most defining characteristic of this worldview is AI-centrism, and treating the impending AGI as an extremely big deal â not just like any other big deals we have seen before, but this will be unprecedented. Those within this overarching worldview would differ in terms of the details, e.g. will it kill everyone? or will it just lead to gradual disempowerment? are LLMs getting us to AGI? or is it some yet-to-be-discovered architecture? should we focus on getting to AGI safely? or start thinking more about the post-AGI world? I think many people move between these âsub-worldviewsâ as they see evidences that update their priors, but way fewer people move out of this overarching worldview entirely.
AI Safety Has a Very ParÂticÂuÂlar Worldview
(semi-commitment for accountability)
Iâm considering writing more about how a big part of AI safety seem to be implicitly built upon an underlying worldview and we have rarely challenged that worldview.
I think there might be some missing links:
> In the current stage of LLMs, one may reasonably have short timelines for AGI coming in the next 3-5 years, as given here. Here, here, here, and here.
I think GWWC uses the term âactive pledgerâ to refer to the pledgers that are still pledgers. I am #10294, which means I was the 10294th person to sign the pledge, but out of those 10294 people, 294 of them either cancelled their pledges or there were double-counts etc. So at the time when I pledged, there were 9999 existing others who were âactiveâ i.e. had not cancelled their pledge. That doesnât mean the on-paper âactiveâ pledgers actually adhere to their pledges, and I donât know if GWWC has a different term for that.
<brag>
You donât know me, but I wasnât just one of the first 10,000. I was the 10,000th.
</âbrag>
Thank you so much! More importantly, congratulations to you and the GWWC team for this milestone!
Thank you very much! I shouldâve known that the pledge # would be different from the active pledgers, but at Iâm glad to know it now. I just made the pledge and I hope to be the 10,000th active pledger, even if momentarily :D
(apologies in advance, this is a bit of a pointless question)
The pledge count is 9993 nowâIâm curious if this amount is updated live? Iâd like to be the 9999th pledger, but Iâm unsure how the counting system works. Thanks!
[Question] How ofÂten is the GWWC pledge count upÂdated?
I see similarities with this paper. It seems your work focuses more on whatâs feasible for geopolitical rivals?
David Thorstad wrote a similar post in case itâs of interest to anyone here.
On thinkÂing about AI risks concretely
Thanks! I think I was having the impression that the Gates Foundation was struggling to give out money (e.g. this comment from a long time ago), but Iâm now learning that thatâs probably no longer trueâthey set a goal of $9 billion by 2026 and theyâre already having a budget of $8.6 billion this year. Now it makes sense.
Thanks for the link! I vaguely remember reading this but probably didnât really get an answer that I was hoping for. In the case of AMF, reason 1 doesnât apply, because they seem to want the money to do things now instead of building reserves. Reason 4 seems most relevantâmaybe the Gates Foundation is hoping that a Malaria vaccine (which recent developments have shown positive results) could render bed nets futile? But I donât think I buy this eitherâconsidering how effective these vaccines currently are, how long it takes to roll out vaccines in these countries, and that Bill Gates himself has previously vouched for bed nets (albeit before the vaccines were endorsed by WHO). As for reasons 2, 3, and 5, I just donât really see how these reasons are worth killing so many babies forâI canât picture a decision maker in the Foundation saying âyeah we have decided to let a hundred thousand people die of Malaria so that we can diversify our risks and encourage others to donateâ.
I may be missing something, but I only see a few reasonable scenarios:
The Gates Foundation does indeed plan to donate, and they might be the âdonor of last resortâ
They really do not intend to fill the funding gap, perhaps because they donât think additional funding to AMF is as cost-effective as advertised
They are confident that AMF will somehow get funding from other sources
Iâd be interested to understand why there are still huge shortfalls in the supposedly top effective charities.
For example, AMF has a funding gap of $300 milion. The Bill and Melinda Gates Foundation has an endowment of $67 billion, which of course they intend to donate away. Bill Gates also endorses GiveWell, and has an explicit focus on solving Malaria (it also lists 20 organizations that they partner with, but AMF is not one of it).
So why isnât the AMF funding gap plugged yet, by the Gates foundation, or anyone else? As for the Foundation, is it a matter of grant evaluation process? Is there anything else relevant I should know to better understand the whole funding landscape of these issues?
To achieve AGI we will need to program the following:
knowledge creating processes
emotions
creativity
free will
consciousness.
I suspect a large part of the crux is the definition of AGI itself. I donât know many people who think that an agent /â system must fulfill all of the above criteria to qualify as âAGIâ. I personally use the term AGI to refer to systems that have at least human-level capabilities at all tasks that a human is capable of performing, regardless of whether the system posses other properties like consciousness and free will.
On a separate matter, I think it might be a good idea if there is a dual voting system for posts, just like comments, where people can upvote/âdownvote and also agree/âdisagree vote. This is a post that I would upvote but strong disagree on. In the meantime I gave it an upvote anyway, since I like posts that at least attempt to constructively challenge prevailing worldviews, and also to balance out all the downvotes.
Iâm not familiar with how this concept is used in the military, but in safety engineering Iâve never heard of it as a tradeoff between âmany layers, many holesâ vs âone layer, few holesâ. The swiss cheese model is often meant to illustrate the fact that your barriers are often not 100% effective, so even if you think you have a great barrier, you should have more than one of it. From this perspective, the concept of having multiple barriers is straightforwardly good and doesnât imply justifying the use of weaker barriers.