I’m a computational physicist, I generally donate to global health. I am skeptical of AI x-risk and of big R Rationalism, and I intend explaining why in great detail.
titotal
Karma: 7,767
Remember that this is graphing the length of task that the AI can do with an over 50% success rate. The length of task that an AI can do reliably is much shorter than what is shown here (you can look at figure 4 in the paper): for an 80% success rate it’s 30 seconds to a minute.
Being able to do a months work of work at a 50% success rate would be very useful and productivity boosting, of course, but it would really be close to recursive self improvement? I don’t think so. I feel that some part of complex projects needs reliable code, and that will always be a bottleneck.
Welcome to the forum. You are not missing anything: in fact you have hit upon some of the most important and controversial questions about the EA movement, and there is wide disagreement on many of them, both within EA and with EA’s various critics. I can try and give both internal and external sources asking or rebutting similar questions.
In regards to the issue of unintended consequences from global aid, and the global vs local issue. this was an issue raised by Leif Wenar in a hostile critique of EA here. You can read some responses and rebuttals to this piece here and here.
With regards to the merits of Longtermism, this will be a theme of the debate week this coming week, so you should be able to get a feel for the debate within EA there. Plenty of EA’s are not longtermist for exactly the reasons you described. Longtermism the focus of a lot of external critique of EA as well, with some seeing it as a dangerous ideology, although that author has themselves been exposed for dishonest behaviour.
AI safety is a highly speculative subject, and their are a wide variety of views on how powerful AI can be, how soon “AGI” could arrive, how dangerous it is likely to be, and what the best strategy is for dealing with it. To get a feel for the viewpoints, you could try searching for “P doom”, which is a rough estimate for the chance of destruction. I might as well plug my own argument for why I don’t think it’s that likely. For external critics, pivot to AI is a newsletter that compiles articles with the perspective that AI is overhyped and that AI safety isn’t real.
The case for “earning to give” is given in detail here. The argument you raise of working for unethical companies is one of the most common objections to the practice, particularly in the wake of the SBF scandal, however in general EA discourages ETG with jobs that are directly harmful.
“The bell curve” was pilloried by the wider scientific community, and for good reason. I recommend watching this long youtube video summarizing the scientific rebuttals.
As for genetic engineering, I don’t see how you can separate it from ethical implications. As far as I can tell, every time humanity has believed that a group of people was genetically inferior, it has resulted in atrocities against that group of people. Perhaps you can get something working by specifically limiting yourself to preventing diseases and so on, but in general, I don’t think society has the ability to handle having actual “superbabies”.
Again, I’m not sure exactly how to respond to comments like this. Like, yeah, if AI could reliably do everything a top researcher does, it could enable a lot of breakthroughs. But I don’t believe that an AI will be able to do that anytime soon. All I can say is that there is a massive gap between current AI capabilities and what they would need to fully automate a material science job. 30 years sounds like a long time, but AI winters have lasted that long before: there’s no guarantee that because AI has rapidly advanced recently that it will not stall out at some point.
I will say that I just disagree that an AI could suddenly go from “no major effect on research productivity” to “automate everything” in the span of a few years. The scale of difficulty of the latter compared to the former is just too massive, and in all new technologies it takes a lot of time to experiment and figure out how to use it effectively. Ai researchers have done a lot of work to figure out how to optimise and get good at the current paradigm: but by definition, the next paradigm will be different, and will require different things to optimize.
Hey, thanks for weighing in, those seem like interesting papers and I’ll give them a read through.
To be clear, I have very little experience in quantum computing, and haven’t looked into it that much and so I don’t feel qualified to comment on it myself (hence why this was just an aside there). All I am doing is relaying the views of prominent professors in my field, who feel very strongly that it is overhyped and were willing to say so in the panel, although I do not recall them giving much detail on why they felt that way. This matches with the general views I’ve had with other physicists in casual conversations. If I had to guess the source of these views, I’d say it was skepticism of the ability to actually build such large scale fault-tolerant systems.
Obviously this is not strong evidence and should not be taken as such.
AI is not taking over material science (for now): an analysis and conference report
From my (small) experience in climate activist groups, I think this is an excellent article.
Some other points in favour:
Organising for small, early wins allows your organisation to gain experience with how to win, and what to do with said wins. A localised climate campaign will help you understand which messages resonate with people and which are duds, and familiarise yourself with how to deal with media, government, etc.
It’s also helps to scale with your numbers: a few hundred people aren’t going to be enough to stop billion dollar juggernauts, but they can cause local councils to feel the heat.
One counterpoint: you shouldn’t be so unambitious that people feel like you’re wasting their time. If just stop oil had started with a campaign to put flower gardens outside public libraries, they wouldn’t have attracted the committed activist base they needed.
If you look at the previous threads you posted, you’ll see I was a strong defender of giving your project a chance. I think grassroots outreach and support in areas like yours is a very good thing, and I’m glad to see you transparently report on your progress with the project.
That being said, I have to agree with the others here that investing in crypto coins like the one you mentioned is generally a bad idea. I have not heard of either of the people you claim are backing the project. The statement that “most people believe Jelly will soon be the new tiktok in the west” is not at all true. I live in the west and I guarantee you that almost nobody has ever heard of this project, and there has not been significant buzz around crypto projects in the west for a good couple of years now.
If you are skeptical, I recommend you go onto reddit and ask people in non-crypto spaces if they have heard of Jelly or are excited about the idea.
People can make money off crypto: but for the average user it’s more or less a casino, where the odds are not in your favour.
I apologise if this comes off as overly critical, but I have heard of a lot of people who have fallen victims to scammers and scoundrels in the crypto space, and I don’t want you to be one of them.
1-4 is only unreasonable because you’ve written a strawman version of 4. Here is a version that makes total sense:
1. You make a superficially compelling argument for invading Iraq
2. A similar argument, if you squint, can be used to support invading Vietnam
3. This argument for invading vietnam was wrong because it made mistakes X, Y, and Z
4. Your argument for invading Iraq also makes mistakes X, Y and Z
5. Therefore, your argument is also wrong.
Steps 1-3 are not strictly necessary here, but they add supporting evidence to the claims.
As far as I can tell from the article, they are saying that you can make a counting argument that argues that it’s impossible to make a working SGD model. They are using this a jumping off point to explain the mistakes that would lead to flawed counting arguments, and then they spend the rest of the article trying to prove that the AI misalignment counting argument is making these same mistakes.
You can disagree with whether or not they have actually proved that AI misalignment made a comparable mistake, but that’s a different problem to the one you claim is going on here.
This again seems like another “bubble” thing. The vast majority of conservatives do not draw a distinction between USAID and foreign aid in general. And I would guess they do associate foreign aid with “woke”, because “woke” is a word that is usually assigned based on vibes alone, for the things perceived as taking away from the average american to give to some other minority. Foreign aid involves spending american money to help foreigners, it’s absolutely perceieved as “woke”.
Look, I wish we lived in a world where people were rational and actually defined their terms and made their decisions accordingly, but that’s not the world we live in.
The original post is only 700 words, and this is like half that length. Can you not give people the dignity of reading their actual parting words?
I don’t think foreign aid is at risk of being viewed as woke. Even the conservative criticisms of USAID tend to focus on things that look very ideological and very not like traditional foreign aid.
This just isn’t true. Yes, exaggerated claims of “wastefulness” are one of the reasons they are against it, but there are many more who are ideologically opposed to foreign aid altogether.
I can link you to this exchange I had with a conservative, where they explictly stated that saving the lives of a billion foreigners would not be worth increasing the national deficit by 4%, because they are ideologically opposed to american taxpayer money saving foreign lives, no matter how efficiently they do it. Or see the insanely aggressive responses to this seemingly innocuous scott alexander tweet. Or here is a popular right wing meme specifically mocking liberals for having large moral circles.
I suspect that you are in a bubble, where the conservatives you know are fine with foreign aid, so you extend that to the rest of conservatives. But in a broader context, 73% of republicans want to cut foreign aid, while only 33% of democrats do.
I think the term “goodharting” is great. All you have to do is look up goodharts law to understand what is talked about: the AI is optimising for the metric you evaluated it on, rather than the thing you actually want it to do.
Your suggestions would rob this term of the specific technical meaning, which makes thing much vaguer and harder to talk about.
I think employers should be obligated to have at least a kitchen area where people can heat up or make their own food. People should not be forced to buy lunch.
I don’t mind you using LLMs for elucidating discussion, although I don’t think asking it to rate arguments is very valuable.
The additional details of having subfield specific auditors that are opt-in does lessen my objections significantly. Of course, the issue of what counts as a subfield is kinda thorny. It would make most sense for, as claude suggests, journals to have an “auditor verified” badge, but then maybe you’re giving too much power over content to the journals, which usually stick to accept/reject decisions (and even that can get quite political).
Coming back to your original statement, ultimately I just don’t buy that any of this can lead to “incredibly low rates of fraud/bias”. If someone wants to do fraud or bias, they will just game the tools, or submit to journals with weak/nonexistent auditors. Perhaps the black box nature of AI might even make it easier to hide this kind of thing.
Next: there are large areas of science where a tool telling you the best techniques to use will never be particularly useful. On the one hand there is research like mine, where it’s so frontier that the “best practices” to put into such an auditor don’t exist yet. On the other, you have statistics stuff that is so well known that there already exist software tools that implement the best practices: you just have to load up a well documented R package. What does an AI auditor add to this?
If I was tasked with reducing bias and fraud, I would mainly push for data transparency requirements in journal publications, and in beefing up the incentives for careful peer review, which is currently unpaid and unrewarding labour. Perhaps AI tools could be useful in parts of that process, but I don’t see it as anywhere near as important than those other two things.
My obvious answer is that the auditors should be held up to higher standards than the things they are auditing. This means that these should be particularly open, and should be open to other auditing. For example, the auditing code could be open-source, highly tested, and evaluated by both humans and AI systems.
Yeah, I just don’t buy that we could ever establish such a code in a way that would make it viable. Science chases novel projects and experiments, what is “meant” to happen will be different for each miniscule subfield of each field. If you release an open source code that has been proven to work for subfields A,B,C,D,E,F, someone in subfield G will immediately object that it’s not transferable, and they may very well be right. And the only people who can tell if it works on subfield G is people who are in subfield G.
You cannot avoid social and political aspects to this: Imagine if the AI-auditor code starts declaring that a controversial and widely used technique in, say, evolutionary psychology, is bad science. Does the evo-psych community accept this and abandon the technique, or do they say that the auditor code is flawed due to the biases of the code creators, and fork/reject the code? Essentially you are allowing whoever is controlling the auditor code to suppress fields they don’t agree with. It’s a centralization of science that is at odds with what allows science to actually work.
I don’t think Eliezer Yudkowsky and the rationalists should be throwing stones here. Sam Altman himself claimed that “eliezer has IMO done more to accelerate AGI than anyone else”. They’ve spent decades trying to convince people of the miraculous powers of AI, and now are acting shocked that this motivated people to try and build it.
As a working scientist, I strongly doubt that any of this will happen.
First, existing AI’s are nowhere near being able to do any of the things with an accuracy that makes them particularly useful. AI’s are equipped to do things similar to their training set, but all science is on the frontier: it is a much harder task to figure out the correct experimental setup for something that has never been done before in the history of humanity.
Right now I’m finishing up an article about how my field acually uses AI, and it’s nothing like anything you proposed here: LLMs are used for BS grant applications and low-level coding, almost exclusively. I don’t find it very useful for anything else.
The bigger issue here is with the “auditors” themselves: who’s in charge of them? If a working scientist disagrees with what the “auditor” says, what happens? What happens if someone like Elon is in charge, and decides to use the auditors for a political crusade against “woke science”, as is currently literally happening right now?
Catching errors in science is not something that can be boiled down to a formula: a massive part of the process is socio-cultural. You push out AI auditors, people are just going to game them, like they have with p-values, etc. This is not a problem with a technological solution.
If I’m reading this right, there was a ~40% drop in the number of respondents in 2024 compared to 2022?
I think this gives cause to be careful when interpreting the results: for example, from the graphs it might look like EA has succesfully recruited a few more centre-left people to the movement: but in absolute terms the number of centre leftists responding to the survey has decreased by about 300 people.
I think the decline in left identifying people is unlikely to be a case of mass political conversion: I think it’s more likely that a large number of leftists have left the movement due to it’s various recent scandals.
I’m not sure the “passive” finding should be that reassuring.
I’m imagining someone googling “ethical career” 2 years from now and finding 80k, noticing that almost every recent article, podcast, and promoted job is based around AI, and concluding that EA is just an AI thing now. If they have no interest in AI based careers (either through interest or skillset), they’ll just move on to somewhere else. Maybe they would have been a really good fit for an animal advocacy org, but if their first impressions don’t tell them that animal advocacy is still a large part of EA they aren’t gonna know.
It could also be bad even for AI safety: There are plenty of people here who were initially skeptical of AI x-risk, but joined the movement because they liked the malaria nets stuff. Then over time and exposure they decided that the AI risk arguments made more sense than they initially thought, and started switching over. In hypothetical future 80k, where malaria nets are de-emphasised, that person may bounce off the movement instantly.