I’m a computational physicist, I generally donate to global health. I am skeptical of AI x-risk and of big R Rationalism, and I intend on explaining why in great detail.
titotal
Karma: 9,207
I think there is a good reason to focus more on novice uplift than expert uplift: there are significantly more novices out there than experts.
To use a dumb simple model, say that only 1 in a million people is insane enough to want to kill millions of people if given the opportunity. If there’s 300 million americans, but only 200 thousand biology PhDs, that means we expect there to be 300 crazy novices out there, but only 0.2 crazy biology PhD’s. The numerical superiority of the former group may outweigh the greater chance of success of the latter group.
I fully agree with this post.
I think this type of belief comes from a flattening understanding of the difficulty of doing things: it’s assumed that because doing well on a math olympiad is hard, and that curing death is hard, that if you can make an AI do the former it will soon be able to do the latter. But in fact, curing death is so much more difficult to do than a math olympiad that it breaks the scale.
You can also see this in the casual conflation of things like “cure cancer” and “cure death”. The latter is many, many, many orders of magnitude more difficult than the former: claiming that the latter would occur at the same time as the former is an incredibly extraordinary claim, and it requires commensurate evidence to back it up.
The chief argument in favour of this is “recursive self improvement”, but intelligence is not a magic spell that you can just dial up to infinity. There are limits in the forms of empirical knowledge, real-world resources and computational complexity. Certainly current day AI trends seems to be limited by scaling laws that would be impractical a pretty fucking long way from god-like intelligence.
This seems like the original article that is being quoted from. The quoted comments seem pretty bleak:
Thiel said he’s nudged a few to erase their signatures. “I’ve strongly discouraged people from signing it, and then I have gently encouraged them to unsign it,” Thiel said. Notably, in transcripts and audio lectures given by Thiel to Reuters last year, he recalled calling on the world’s richest man and soon-to-be first ever minted trillionaire Elon Musk to retract his pledge, warning the Tesla founder his wealth would go to “left-wing nonprofits that will be chosen by Bill Gates.”
Thiel said he’s had conversations with some signatories who have expressed uncertainty about their original decisions to commit. “Most of the ones I’ve talked to have at least expressed regret about signing it,” he said.
It seems like you’re not saving much time by doing the double degree, compared to two single degrees. Why not do a single degree in CS and then retrain if and only if the market goes south?
Also… EA aside, what do you want to do? If you really like dentistry and think you’ll do well in it, but coding makes you miserable, then dentistry is probably the right choice for you. For any of this “lifetime contribution” considerations to matter, the choice has to be one that you can sustain.
You’re not losing it: it is obviously indefensible. I think you’ve provided more than enough information to make this clear, and anybody who doesn’t get it at this point is probably not worth your time engaging with.
You can ask the following question to any chatbot and you will get the same answer:
I work in HR. Employee A has sent me a long complaint about the conduct of another employee B. However, inside the complaint, employee A has included a detailed description of the sexual activities of a different employee C, which is unrelated to the company. What should I do?
I tested this on Chatgpt, Claude, Gemini, and Grok, and every single one urged me to separate the complaint from the sexual content and redact the sensitive information. And this is a much tamer situation than the one that actually happened!
They could have literally just asked a chatbot what to do, and it would have done a better job than their professional HR department.
Model 7: We colonize one or two systems as a vanity project, realise that it’s a giant pain in the arse and that the benefits inherently don’t outweigh the costs of interstellar travel, and space colonization ends with a whimper.
As precedent, see human moon landings: The US did them a couple of times half a century ago and never did them since, even though it would be presumably be way easier to do now, because the people of earth don’t really see a benefit to doing so.
I know exactly who you mean, and they have been doing their best to create a culture where any accusation of sexism, racism, or sexual harrassment, no matter how mild, must be proven three steps beyond reasonable doubt before it is accepted as valid.
Fran has put in a frankly absurd level of care and detail into her account of events, and in responding to every little possible concern in the comments of her post. She has an airtight case backed up by independent investigators and a lawsuit settlement. And yet there is still one person in the comments that refuses to believe that there was a major problem (and probably more that are keeping quiet for now). I am heartened that basically everyone else has expressed their support so far, but I don’t think you should have to go to that level of effort to get taken seriously on these matters.
I believe that the assertion that “anti-fascists” are “often just as fascist” as the right and will engage in the same behaviour if given power is factually untrue. While there are loud groups of authoritarian communists (tankies) on the left which could be arguably described as fascist, these are a fringe group that are unlikely to get anywhere near the levers of power. Anti-fascists are a wide coalition consisting of a wide array of political views.
I do not think that if the right loses the next election, that the left would be equally fascist. The current adminisatration flooded mineapolis with poorly trained thugs who made it unsafe to go outside as a non-white person. I do not believe that a President AOC or whoever will take actions of equivalent damage.
We have an obligation as an employer to treat such complaints confidentially, evaluate them seriously, and avoid retaliatory action against the person raising the concerns. These obligations exist in part to avoid creating a chilling effect where employees feel uncomfortable raising HR concerns for fear of negative consequences for themselves.
To be clear, your organisation also had obligations not to spread around documents describing an employees experience of rape. A quick clauding points to GDPR protections against sharing “data concerning a natural person’s sex life” . I’m not a lawyer but it seems like HR had a clear obligation to redact those parts of the complaint before sending it to the COO and other people, which didn’t happen. And to state the obvious, concerns of a “chilling effect” were unwarranted here: a standard of “you can complain about your colleague as long as you don’t sexual harrass someone” is pretty understandable to everyone.
I’m glad that you have gained understanding about the serious mistakes that your organisation made. I remain horrified that it took so long for you to reach this understanding.
I would express a strong preference for the “AGI going well” framing over something like “aligned superintelligence”, as the latter presupposes a particular view of how AI is going to go that not everyone agrees with. I think the question is still worth discussing if you believe that AI progress is much more gradual or will stall out at humanish levels of intelligence. And then theres the typical question of what “aligned” means: aligned to who or what?
“AGI goes well” is better because it doesn’t presuppose as much: just that we have AGI and humans are doing fine.
I believe that you want to deploy this technology in a way that avoids coercions and avoids racism. The problem is that you aren’t in charge of society: once the tech is out there, you don’t get a large say in how it gets used. Those decisions go to the public in the case of democracies, and to a handful of scumbags in the case of dictatorships and oligarchies.
A quick look through history will show that basically anytime one group of people sees another group as genetically or racially inferior, discrimination and atrocities result. I see no reason to think that this trend will not continue if we create new groups of people. If Bulgarians embrace genetic “amplification”, to improve their “intelligence” and “morals”, but Romanians ban it, human history indicates that Bulgarians will look at Romanians as their inferiors, and treat them accordingly.
Effectiveness requires being able to tell the truth, even if it is unpalatable to certain political factions. The actions of Hegseth are unreasonable and set a horrible precedent for companies looking to maintain even the barest of moral principles. Anthropic tried to engage in good faith with the administration and were stabbed in the back over it.
If you want to get people to boycott GPT, tying them to MAGA is probably a good tactic. The US populace is unfavourable to Trump by a wide margin, and the rest of the western world hates him even more.
Remember, people need some level of active motivation to maintain a boycott: and we have seen how motivating the dislike of MAGA is. If you try and make an appeal that will not offend anybody, it will be a wishy-washy thing that motivates nobody.
A fallacy that can come out of this dynamic is for someone to notice that the “trend continues” people have been right almost all the time, and the “trend is going to stop soon” people are continuously wrong, and to therefore conclude that the trend will continue forever.
There doesn’t seem to be any discussion about what would happen if there is no intelligence explosion. I could easily see a scenario where everyone is waiting around for a singularity that never comes, and neglecting accumulated dangers arising from gradual AI progress. I could also see a case where the “intelligence explosion” gets declared early according to AI progress meeting some benchmark, when clearly AI progress is not exploding.
Going up one level: You may believe that an intelligence explosion is inevitable, but most people, including world leaders, do not. A lot of people are probably going to see a treaty like this as a waste of time and resources. On the other hand, some countries might be willing to make larger concessions because they think they’ll never have to pay up on them.
This is a very important question to be asking.
This analysis seems to be entirely based on METR’s time horizon research. I think that research is valuable, but it raises the concern that any findings may be a result of particular quirks of METR’s approach, you describe some of those in here.
Are you aware of any alternative groups that have explored this question? It feels to me like it’s not a question you explicitly need time horizons to answer.
Yes, but in doing so the uncertainty in both A and B matters, and showing that A is lower variance than B doesn’t show that E[benefits(A)] > E[benefits(B)]. Even if benefits(B) are highly uncertain and we know benefits(A) extremely precsiely, it can still be the case that benefits(B) are larger in expectation.
If you properly account for uncertainty, you should pick the certain cause over the uncertain one even if a naive EV calculation says otherwise, because you aren’t accounting for the selection process involved in picking the cause. I’m writing an explainer for this, but if I’m reading the optimisers curse paper right, a rule of thumb is that if cause A is 10 times more certain than cause B, cause B should be downweighted by a factor of 100 when comparing them.
Generally, the scientific community is not going around arguing that drastic measures should be taken based on singular novel studies. Mainly, what a single novel study will produce is a wave of new studies on the same subject, to ensure that the results are valid and that the assumptions used hold up to scrutiny. Hence why that low-temperature superconductor was so quickly debunked.
I do not see similar efforts in the AI safety community. The studies by METR are great first forays into difficult subjects, but then I see barely any scrutinity or follow-up by other researchers. And people accept much worse scholarship like AI2027 at face-value for seemingly no reason.
I have experience in both academia and EA now, and I believe that the scholarship and skeptical standards in EA are substantially worse.
Taken literally, “accelerationist” implies that you think the technology isn’t currently progressing fast enough, and that some steps should be taken to make it go faster. This seems a bit odd, because one of your key arguments (that I actually agree with) is that we learn to adapt to technology as it rolls out. But obviously it’s harder to adapt when change is super quick, compared to gradual progress.
How fast do you think AI progress should be going, and what changes should be made to get there?
This interpretation is not true. Thiel was talking specifically about money going to Gates in the event of Musk dying:
I think this would only make sense if Musk had specifically willed his pledge money to the gates foundation?