I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me).
I have a website: https://mdickens.me/ Most of the content on my website gets cross-posted to the EA Forum.
My favorite things that I’ve written: https://mdickens.me/favorite-posts/
I used to work as a software developer at Affirm.
I think Matthew and Tamay think this is positive, since they think AI is positive.
I don’t see how this alleviates concern. Sure they’re acting consistently with their beliefs*, but that doesn’t change the fact that what they’re doing is bad.
*I assume, I don’t really know
I think you are one of the few people who disregards x-risk and has a well-considered probability estimate for which it makes sense to disregard x-risk. (Modulo some debate around how to handle tiny probabilities of enormous outcomes.)
I was more intending to critique the sort of people who say “AI risk isn’t a concern” without having any particular P(doom) in mind, which in my experience is almost all such people.
Yeah I agree with that.
So is it fair to say that the takeaway is that we have pretty strong evidence for the efficacy of very large protests in the US, but very little evidence about smaller protest activities?
I think that’s correct. On priors, if large-scale protests work, I would expect smaller protests to work too, but there’s minimal supporting evidence. In this section I gave an argument for why small-scale protests might not work even if nationwide protests do.
I wonder if protests about animals can expect to have similar results, given that baseline consideration for animals as relevant stakeholders seems to be quite a bit lower.
I don’t know, but there’s some evidence about this. Orazani et al. (2021) included some animal welfare protests. It would be possible to do a subgroup analysis comparing the animal vs. human protests.
I wonder if any studies have looked at patterns of backlash? E.g., BLM protest succeeds in the short term, but then DEI is cancelled by the Trump administration.
There are indeed studies on this, but I didn’t review them because none of them are high-quality. Well, Wasow (2020) has high-quality evidence that violent protests backlash, but I think that’s not what you’re talking about; you’re talking about a short-term success followed by a long-term backlash.
Survey evidence from the BLM study actually found the reverse order: protests appeared to have a negative effect on BLM favorability in the weeks following the protests, but by the time of the 2020 election, BLM appeared to have a net positive effect—although I don’t think the study succeeded at establishing causality, so I’m not confident that this is a real effect.
Here’s an example of what I would consider high-quality evidence on this question:
Study uses rainfall method, finds that BLM protests increased support for Democrats in 2020.
However, the same counties showed decreased support for Democrats in 2022.
There are no studies like that, as far as I know.
I don’t know what “election by jury” means or how it’s supposed to be a democratic safeguard. I was interested to know what it meant, but the first ~500 words of the article did not explain it, so I gave up.
This comment reads to me like it was written by an AI. If I’m right, it sounds like you did not take my criticism to heart.
The writing style of your post, with all the AI-generated text, makes it hard for me to tell what you actually believe, and what was generated by AI. You should skip all the AI text and write what you actually believe.
Another writing technique you might use, if you have something you want to say but you’re not sure how to say it:
Write a list of bullet points summarizing what you want the post to say.
Ask an AI to turn the bullet points into a full post.
(Most important step) Throw away the AI-generated text and just post your bullet points, they will be more useful to readers.
I would recommend including a summary at the beginning of the article. It took quite a long time to get to the point and I gave up on reading before it got there.
Asking AIs about something and then posting their responses is not a useful sort of post.
Beyond that, this post seems to be addressing several things at once and it’s not clear to me what it’s trying to say.
Thanks, this comment gives me a much better sense of where you’re coming from. I agree and disagree with various specific points, but I won’t get into that since I don’t think we will resolve any disagreements without an extended discussion.
What I will say is that I found this comment to be much more enlightening than your original post. And whereas I said before that the original article didn’t feel like the output of a reasoning process, this comment did feel like that. At least for me personally, I think whatever mental process you used to write this comment is what you should use to write these sorts of articles, because whatever process you used to write this comment, it worked.
I don’t know what’s going on inside your head, but if I were to guess, perhaps you didn’t want to write an article in the style of this comment because it’s too informal or personal or un-authoritative. Those qualities do make it harder to (say) get a paper published in an academic journal, but I prefer to read articles that have those qualities. If your audience is the EA Forum or similar, then I think you should lean into them.
However I did occasionally feel like I was ~LARPing research writing when I was trying to articulate my thoughts, and suspect some of that never got resolved!
I don’t think you were LARPing research, your comment shows a real thought process behind it. After reading all your line item responses, I feel like I understand what you were trying to say. Like on the few parts I quoted as seeming contradictory, I can now see why they weren’t actually contradictory and they were part of a coherent stance.
I agree with a good portion of your comment but I still don’t think increasing government competence (on AI) is worth prioritizing:
SB-1047 was adequately competently written (AFAICT). If we get more regulations at a similar level of competence, that would be reasonable.
Good AI regulations will make things harder on AI companies. AI leaders / tech accelerationists will be unhappy about regulations regardless of how competently written they are. On the other hand, the general population mostly supports AI regulations (according to AIPI polls). Getting regulators on board with what people want seems to me to be the best path to getting regulations in place.
Like, the US government uses software a lot, but I don’t see them “funding/helping software development”
Suppose it turned out Microsoft Office was dangerous. Surely the fact that Office is so embedded in government procedures would make it less likely to get banned?
IIRC you see similar phenomena (although I can’t recall any examples off hand) where some government-mandated software has massive security flaws but nobody does anything about it because the software is too entrenched.
I agree.
A big reason why I think timeline forecasting gets too much attention—which you alluded to— is that no matter how much forecasting you do, you’ll never have that much confidence about how AI is going to go. And certain plans only work under a narrow set of future scenarios. You need to have a plan that works even if your forecast is wrong, because there will always be a good chance that your forecast is wrong.
Slowing down AI has downsides, but it seems to me that slowing down AI is the plan that works under the largest number of future scenarios. Particularly an international treaty to globally slow down AI, so that all developers slow down simultaneously. That seems hard to achieve, but I think peaceful protests increase the chance of success by cultivating political will for a pause/slowdown treaty.
Epidemiology! I hadn’t really thought about epidemiology as a career but it strikes me as potentially very high impact, especially if you’re going into it with an attention to impact. My basic thinking is that the field of health tends to have some of the lowest-hanging fruit in terms of improving people’s lives, and epidemiology can have a leveraged impact by benefiting many people simultaneously (which is also why being a doctor is maybe less good—the number of people you can help is much smaller).
If you have thoughts, I am interested in what you think about where are the big problems in epidemiology, or at least where are the big problems that you personally can contribute to. It’s not a space I know much about. (You did say the problems are complex which seems true to me so I don’t think I am really in a position to understand epidemiology lol.)
Thank you for this article. I’ve read some of the stuff you wrote in your capacity at CEA, which I quite enjoyed, your comments on slow vs. quick mistakes changed my thinking. This is the first thing I’ve read since you started at Forethought. I have some comments, which are mostly critical, I tried using ChatGPT and Claude to make my comment more even-handed but they did a bad job so you’re stuck with reading my overly critical writing. Some of my criticism may be misguided due to me not having a good understanding of the motivation behind writing the article so it might help me if you explained more about the motivation. Of course you’re not obligated to explain anything to me or to respond at all, I’m just writing this because I think it’s generally useful to share criticisms.
I think this article would benefit from a more thorough discussion of the downside risks of its proposed changes—off the top of my head:
Increasing government dependency on AI systems could make policy-makers more reluctant to place restrictions on AI development because they would be hurting themselves by doing so. This is a very bad incentive.
The report specifically addresses how the fact that Microsoft Office is so embedded in government means the company can get away with bad practices, but seemingly doesn’t connect this to how AI companies might end up in the same position.
Government contracts to buy LLM services increases AI company revenue, which shortens timelines.
The government does not always work in the interests of the people (in fact it frequently works against them!) so making the government more effective/powerful is not pure upside.
The article does mention some downsides, but with no discussion of tradeoffs, and it says we should focus on “win-wins” but doesn’t actually say how we can avoid the downsides (or, if it did, I didn’t get that out of the article).
To me the article reads like you decided the conclusion and then wrote a series of justifications. It is not clear to me how you arrived at the belief that the government needs to start using AI more, and it’s not clear to me whether that’s true.
For what it’s worth, I don’t think government competence is what’s holding us back from having good AI regulations, it’s government willingness. I don’t see how integrating AI into government workflow will improve AI safety regulations (which is ultimately the point, right?[^1]), and my guess is on balance it would make AI regulations less likely to happen because policy-makers will become more attached to their AI systems and won’t want to restrict them.
I also found it odd that the report did not talk about extinction risk. In its list of potential catastrophic outcomes, the final item on the list was “Human disempowerment by advanced AI”, which IMO is an overly euphemistic way of saying “AI will kill everyone”.
By my reading, this article is meant to be the sort of Very Serious Report That Serious People Take Seriously, which is why it avoids talking about x-risk. I think that:
you won’t get people to care about extinction risks by pretending they don’t exist;
the market is already saturated with AI safety people writing Very Serious Reports in which they pretend that human extinction isn’t a serious concern;
AI x-risk is mainstream enough at this point that we can probably stop pretending not to care about it.
There are some recommendations in this article that I like, and if I think it should focus much more on them:
investing in cyber security, pre-deployment testing of AI in high-stakes areas, and advancing research on mitigating the risks of advanced AI
Without better compliance tools, AI companies and AI systems might start taking increasingly consequential actions without regulators’ understanding or supervision
[Without oversight], the government may be unable to verify AI companies’ claims about their testing practices or the safety of their AI models.
Steady AI adoption could backfire if it desensitizes government decision-makers to the risks of AI in government, or grows their appetite for automation past what the government can safely handle.
I also liked the section “Government adoption of AI will need to manage important risks” and I think it should have been emphasized more instead of buried in the middle.
I don’t really know how to organize this so I’m just going to write a list of lines that stood out to me.
invest in AI and technical talent
What does that mean exactly? I can’t think of how you could do that without shortening timelines so I don’t know what you have in mind here.
Streamline procurement processes for AI products and related tech
I also don’t understand this. Procurement by whom, for what purpose? And again, how does this not shorten timelines? (Broadly speaking, more widespread use of AI shortens timelines at least a little bit by increasing demand.)
Gradual adoption is significantly safer than a rapid scale-up.
This sounds plausible but I am not convinced that it’s true, and the article presents no evidence, only speculation. I would like to see more rigorous arguments for and against this position instead of taking it for granted.
And in a crisis — e.g. after a conspicuous failure, or a jump in the salience of AI adoption for the administration in power — agencies might cut corners and have less time for security measures, testing, in-house development, etc.
This line seems confused. Why would a conspicuous failure make government agencies want to suddenly start using the AI system that just conspicuously failed? Seems like this line is more talking about regulating AI than adopting AI, whereas the rest of the article is talking about adopting AI.
Frontier AI development will probably concentrate, leaving the government with less bargaining power.
I don’t think that’s how that works. Government gets to make laws. Frontier AI companies don’t get to make laws. This is only true if you’re talking about an AI company that controls an AI so powerful that it can overthrow the government, and if that’s what you’re talking about then I believe that would require thinking about things in a very different way than how this article presents them.
And: would adopting AI (i.e. paying frontier companies so government employees can use their products) reduce the concentration of power? Wouldn’t it do the opposite?
It’s natural to focus on the broad question of whether we should speed up or slow down government AI adoption. But this framing is both oversimplified and impractical
Up to this point, the article was primarily talking about how we should speed up government AI adoption. But now it’s saying that’s not a good framing? So why did the article use that framing? I get the sense that you didn’t intend to use that framing, but it comes across as if you’re using it.
Hire and retain technical talent, including by raising salaries
I would like to see more justification for why this is a good idea. The obvious upside is that people who better understand AI can write more useful regulations. On the other hand, empirically, it seems that people with more technical expertise (like ML engineers) are on average less in favor of regulations and more in favor of accelerating AI development (shortening timelines, although they usually don’t think “timelines” are a thing). So arguably we should have fewer such people in positions of government power. I can see the argument either way, I’m not saying you’re wrong, I’m just saying you can’t take your position as a given.
And like I said before, I think by far the bigger bottleneck to useful AI regulations is willingness, not expertise.
Explore legal or other ways to avoid extreme concentration in the frontier AI market
(this isn’t a disagreement, just a comment:)
You don’t say anything about how to do that but it seems to me the obvious answer is antitrust law.
(this is a disagreement:)
The linked article attached to this quote says “It’s very unclear whether centralizing would be good or bad”, but you’re citing it as if it definitively finds centralization to be bad.
If the US government never ramps up AI adoption, it may be unable to properly respond to existential challenges.
What does AI adoption have to do with the ability to respond to existential challenges? It seems to me that once AI is powerful enough to pose an existential threat, then it doesn’t really matter whether the US government is using AI internally.
Map out scenarios in which AI safety regulation is ineffective and explore potential strategies
I don’t think any mapping is necessary. Right now AI safety regulation is ineffective in every scenario, because there are no AI safety regulations (by safety I mean notkilleveryoneism). Trivially, regulations that don’t exist are ineffective. Which is one reason why IMO the emphasis of this article is somewhat missing the mark—right now the priority should be to get any sort of safety regulations at all.
Build emergency AI capacity outside of the government
I am moderately bullish on this idea (I’ve spoken favorably about Sentinel before) although I don’t actually have a good sense of when it would be useful. I’d like to see more projection of under exactly what sort of scenarios “emergency capacity” would be able to prevent catastrophes. Not that that’s within the scope of this article, I just wanted to mention it.
[^1] Making government more effective in general doesn’t seem to me to qualify as an EA cause area, although perhaps a case could be made. The thing that matters on EA grounds (with respect to AI) is making the government specifically more effective at, or more inclined to, regulate the development of powerful AI.
Perhaps most immediately jarring is the recommendation to add olive oil to smoothies—a culinary choice that defies both conventional wisdom and basic palatability.
I’ve tried putting olive oil in smoothie-adjacent concoctions (calling the things I’ve made “smoothies” would be an insult to smoothies) and it always makes me nauseous.
One time, due to poor planning, the only thing I had available to eat all day was an olive-oil-based smoothie-adjacent beverage, and I still couldn’t manage to choke it down.
I’m a bit late to the party but:
On “backfire”—do you have any view on backfire of BLM protests? I’ve been concerned with the pattern of protest → police stop enforcing in a neighborhood → murder rates go up.
I wouldn’t consider this a “backfire”, although murder rates going up is definitely a bad thing. In the context of protests, a backfire isn’t when anything bad happens, it’s when the protests hurt the protesters’ goals. If “police stop enforcing in a neighborhood” is a goal of BLM protests (which it basically is), then this is a success, not a backfire, and the increase in murder rate is an unfortunate consequence.
A backfire effect would be something like: protest → protests make people feel unsafe → city allocates more funding to the police.
I will not be upvoting, downvoting, agree-voting, disagree-voting, or reacting to this piece, and I will not be leaving any comment except to say that I have no comment.
My lack of feedback should NOT be construed as an endorsement of your anti-feedback position.
I just think Anthropic leaders being un-candid about their connection to EA is pretty weak evidence that they’re doing fraud or something like what FTX did. (It’s positive evidence, but weak.)
Increasing the amount of animal-friendly content that is likely to feature in AI training data
My understanding is that current AIs’ (professed) values are largely determined by RLHF, not by training data. Therefore it would be more effective to persuade the people in charge of RLHF policies to make them more animal-friendly.
But I have no idea whether RLHF will continue to be relevant as AI gets more powerful, or if RLHF affects AI’s actual values rather than merely its professed values.
I think it’s a good thing that you’re open about your motivations and I appreciate it.