I don’t know of any such stats, but I also don’t know much about CFAR.
I was excluding governance papers, because it seems like the relevant question is “will AI development happen in Europe or elsewhere”, and governance papers provide ~no evidence for or against that.
My lived experience is that most of the papers I care about (even excluding safety-related papers) come from the US. There are lots of reasons that both of these could be true, but for the sake of improving AGI-related governance, I think my lived experience is a much better measure of the thing we actually care about (which is something like “which region does good AGI-related thinking”).
From my current read, psychedelics have a stronger evidence base than rationality training programs
I agree if for CFAR you are looking at the metric of how rational their alumni are. If you instead look at CFAR as a funnel for people working on AI risk, the “evidence base” seems clearer. (Similarly to how we can be quite confident that 80K is having an impact, despite there not being any RCTs of 80K’s “intervention”.)
Sorry, I’m claiming government is supposed to spend money to achieve outcomes the public wants. (That felt self-evident to me, but maybe you disagree with it?) Given that, it’s weird to say that it is better to give the money to the public than to let the government spend it.
I think the claim “philanthropic spending can do more good than typical government spending” usually works because we agree with the philanthropist’s values more so than “government’s values”. But I wouldn’t expect that “public’s values” would be better than “government’s values”, and I do expect that “government’s competence” would be better than “public’s competence”.
Not necessarily disagreeing, but I wanted to point out that this relies on a perhaps-controversial claim:
Claim: Even though government is supposed to spend money to achieve outcomes the public wants, it is better to give the money to the public so that they can achieve outcomes that they want.
To me, the most relevant of these impossibility theorems is the Arrhenius paradox (relevant to population ethics). Unfortunately, I don’t know of any good public explanation of it.
Even with the astronomical waste argument, which is the most extreme version of this argument, at some point you have astronomical numbers of people living, and the rest of the future isn’t tremendously large in comparison, and so focusing on flourishing at that point makes more sense. Of course, this would be quite far in the future.
In practice, I expect the bar comes well before that point, because if everyone is focusing on x-risks, it will become harder and harder to reduce x-risks further, while staying equally as easy to focus on flourishing.
Note that in practice many more people in the world focus on flourishing than on x-risks, so maybe the few long-term focused people might end up always prioritizing x-risks because everyone else picks the low-hanging fruit in flourishing. But that’s different from saying “it’s never important to work on animal suffering”, it’s saying “someone else will fix animal suffering, and so I should do the other important thing of reducing x-risk”.
I’m pretty sure all the people you’re thinking about won’t make claims any stronger than “All of EA’s resources should currently be focused on reducing extinction risks”. Once extinction risks are sufficiently small, I would expect them to switch to focusing on flourishing.
The random chance argument is harder to make if the studies have large effect sizes. If the true effect is 0, it’s unlikely we’ll observe a large effect by chance.
This is exactly what p-values are designed for, so you are probably better off looking at p-values rather than effect size if that’s the scenario you’re trying to avoid.
I suppose you could imagine that p-values are always going to be just around 0.05, and that for a real and large effect size people use a smaller sample because that’s all that’s necessary to get p < 0.05, but this feels less likely to me. I would expect that with a real, large effect you very quickly get p < 0.01, and researchers would in fact do that.
(I don’t necessarily disagree with the rest of your comment, I’m more unsure on the other points.)
See also my summary and Richard Ngo’s comments.
Yeah, I’ve been doing this occasionally (though that started recently).
From my present vantage, the AI alignment newsletter is becoming a pretty prominent clearinghouse for academic AI alignment research updates. (I wouldn’t be surprised if it were the primary source of such for a sizable portion of newsletter subscribers.)
To the extent that’s true, the amplification effects seem possibly strong.
I agree that’s true and that the amplification effects for AI safety researchers are strong; it’s much less strong of an amplification effect for any other category. My current model is that info hazards are most worrisome when they spread outside the AI safety community.
On confidentiality, the downsides of the newsletter failing to preserve confidentiality seem sufficiently small that I’m not worried (if you ignore info hazards). Failures of confidentiality seem bad in that they harm your reputation and make it less likely that people are willing to talk to you—it’s similar to the reason you wouldn’t break a promise even if superficially the consequences of the thing you’re doing seem slightly negative. But in the case of the newsletter, we would amplify someone else’s failure to preserve confidentiality, which shouldn’t reflect all that poorly on us. (Obviously if we knew that the information was supposed to be confidential we wouldn’t publish it.)
This was in response to “the growing amount of AI safety research.”
Yeah, I think I phrased that question poorly. The question is both “should all of it be summarized” and “if yes, how can that be done”.
Presumably as there is more research, it takes more time to read & assess the forthcoming literature to figure out what’s important / worth including in the newsletter.
I feel relatively capable of that—I think I can figure out for any given reading whether I want to include it in ~5 minutes or so with relatively high accuracy. It’s actually reading and summarizing it that takes time.
Interesting to think about what governance the newsletter should have in place re: info hazards, confidentiality, etc.
Currently we only write about public documents, so I don’t think these concerns arise. I suppose you could imagine that someone writes about something they shouldn’t have and we amplify it, but I suspect this is a rare case and one that should be up to my discretion.
What did you guys do for GPT-2?
Not sure what specifically you’re asking about here. You can see the relevant newsletter here.
My intuition is that this would be a good time to formalize the structure of the newsletter somewhat, especially given that there are multiple contributors & you are starting to function more as an editor.
Certainly more systems are being put into place, which is kind of like “formalizing the structure”. Creating an organization feels like a high fixed cost for not much benefit—what do you think the main benefits would be? (Maybe this is combined with paying content writers and editors, in which case an organization might make more sense?)
Plausibly it’s fine to keep it as an informal research product, but I’d guess that “AI alignment newsletter editor” could basically be (or soon become) a full-time job.
If I were to make this my full-time job, the newsletter would approximately double in length (assuming I found enough content to cover), and I’d expect that people wouldn’t read most of it. (People already don’t read all of it, I’m pretty sure.) What do you think would be the value of more time put into the newsletter?
My first guess is that there’s significant value in someone maintaining an open, exhaustive database of AIS research.
Yeah, I agree. But there’s also significant value in doing more AIS research, and I suspect that on the current margin for a full-time researcher (such as myself) it’s better to do more AIS research compared to writing summaries of everything.
Note that I do intend to keep adding all of the links to the database, it’s the summaries that won’t keep up.
It is plausible to me that an org with a safety team (e.g. DeepMind/OpenAI) is already doing this in-house, or planning to do so.
I’m 95% confident that no one is already doing this, and if they were seriously planning to do so I’d expect they would check in with me first. (I do know multiple people at all of these orgs.)
More broadly, these labs might have some good systems in place for maintaining databases of new research in areas with a much higher volume than AIS, so could potentially share some best-practices.
You know, that would make sense as a thing to exist, but I suspect it does not. Regardless that’s a good idea, I should make sure to check.
Comment thread for the question: What is the value of the newsletter for you?
Comment thread for the question: What is the value of the newsletter for other people?
Comment thread for the question: How should I deal with the growing amount of AI safety research?