I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me). Lately, I mainly write about EA investing strategy, but my attention span is too short to pick just one topic.
I have a website: https://mdickens.me/ Most of the content on my website gets cross-posted to the EA Forum.
My favorite things that I’ve written: https://mdickens.me/favorite-posts/
I used to work as a software developer at Affirm.
$100 million/year and 600 people = $167,000 per person-year
$1M buys 10 person-years = $100,000 per person-year
These numbers are approximately the same. I don’t understand how you get that 5⁄6 of the work comes from volunteering / voluntary underpayment, did I do it wrong?
I plan on donating to PauseAI, but I’ve put considerable thought into reasons not to donate.
I gave some arguments against slowing AI development (plus why I disagree with them) in this section of my recent post, so I won’t repeat those.
There’s not that much evidence that protests are effective. There’s some evidence, including a few real-world natural experiments and some lab experiments, but this sort of research doesn’t have a good replication record.
Research generally suggests that peaceful protests work, while violent protests reduce public support. If violent protests backfire, maybe some types of peaceful protests also backfire. And we don’t have enough granularity in the research to identify under what specific conditions peaceful protests work, so maybe PauseAI-style protests will backfire.
Polls suggest there is broad public support for a pause. If there’s already public support, doesn’t that weaken the case for protesting? Perhaps that means we should be doing something else instead.
Protesting might not be a good use of resources. It might be better to lobby policy-makers.
Yes that’s also fair. Conflicts of interest are a serious concern and this might partially explain why big funders generally don’t support efforts to pause AI development.
I think it’s ok to invest a little bit into public AI companies, but not so much that you’d care if those companies took a hit due to stricter regulations etc.
I think the position I’m arguing for is basically the standard position among AI safety advocates so I haven’t really scrutinized it. But basically, (many) animals evolved to experience happiness because it was evolutionarily useful to do so. AIs are not evolved so it seems likely that by default, they would not be capable of experiencing happiness. This could be wrong—it might be that happiness is a byproduct of some sort of information processing, and sufficiently complex reinforcement learning agents necessarily experience happiness (or something like that).
Also: According to the standard story where an unaligned AI has some optimization target and then kills all humans in the interest of pursuing that target (e.g. a paperclip maximizer), it seems unlikely that this AI would experience much happiness (granting that it’s capable of happiness) because its own happiness is not the optimization target.
(Note: I realize I am ignoring some parts of your comment, I’m intentionally only responding to the central point so my response doesn’t get too frayed.)
I want to add that I think the argument you present here is better than any of the arguments I’d considered in the relevant section.
I think it’s quite unlikely that a misaligned AI would create an AI utopia. Much more likely that it would create something that resembles a paperclip maximizer / something that has either no conscious experience, or experiences with no valance, or experiences with random valences.
I had remembered that the pause letter talked about extinction. Reading again, it doesn’t use the word extinction; it does say “Should we risk loss of control of our civilization?” which is similar but somewhat ambiguous. CAIS’ Statement on AI Risk would have been a better example.
On that framing, I agree that that’s something that happens and that we should be able to anticipate will happen.
Thanks for the comment! Disagreeing with my proposed donations is the most productive sort of disagreement. I also appreciate hearing your beliefs about a variety of orgs.
A few weeks ago, I read your back-and-forth with Holly Elmore about the “working with the Pentagon” issue. This is what I thought at the time (IIRC):
I agree that it’s not good to put misleading messages in your protests.
I think this particular instance of misleadingness isn’t that egregious, it does decrease my expectation of the value of PauseAI US’s future protests but not by a huge margin. If this was a recurring pattern, I’d be more concerned.
Upon my first reading, it was unclear to me what your actual objection was, so I’m not surprised that Holly also (apparently) misunderstood it. I had to read through twice to understand.
Being intentionally deceptive is close to a dealbreaker for me, but it doesn’t look to me like Holly was being intentionally deceptive.
I thought you both could’ve handled the exchange better. Holly included misleading messaging in the protest and didn’t seem to understand the problem, and you did not communicate clearly and then continued to believe that you had communicated well in spite of contrary evidence. Reading the exchange weakly decreased my evaluation of both your work and PauseAI US’s, but not by enough to change my org ranking. You both made the sorts of mistakes that I don’t think anyone can avoid 100% of the time. (I have certainly made similar mistakes.) Making a mistake once is evidence that you’ll make it more, but not very strong evidence.
I re-read your post and its comments just now and I didn’t have any new thoughts. I feel like I still don’t have great clarity on the implications of the situation, which troubles me, but by my reading, it’s just not as big a deal as you think it is.
General comments:
I think PauseAI US is less competent than some hypothetical alternative protest org that wouldn’t have made this mistake, but I also think it’s more competent than most protest orgs that could exist (or protest orgs in other cause areas).
I reviewed PauseAI’s other materials, although not deeply or comprehensively, and they seemed good to me. I listened to a podcast with Holly and my impression was that she had an unusually clear picture of the concerns around misaligned AI.
I believe the “consciousness requires having a self-model” is the only coherent model for rejecting animals’ moral patienthood, but I don’t understand the argument for why the model is supposedly true. Why would consciousness (or moral patienthood) require having a self-model? I have never seen Eliezer or anyone attempt to defend this position.
+1 for doing a Fermi estimate, I would like to see more of those.
I looked thru the congressional commission report’s list of testimonies for plausibly EA-adjacent people. The only EA-adjacent org I saw was CSET, which had two testimonies (1, 2). From a brief skim, neither one looked clearly pro- or anti-arms race. They seemed vaguely pro-arms race on vibes but I didn’t see any claims that look like they were clearly encouraging an arms race—but like I said, I only briefly skimmed them, so I could have missed a lot.
I would like to see this. I have considerable uncertainty about whether to prioritize (longtermism-oriented) animal welfare or AI safety.
I did it in my head and I haven’t tried to put it into words so take this with a grain of salt.
Pros:
Orgs get time to correct misconceptions.
(Actually I think that’s pretty much the only pro but it’s a big pro.)
Cons:
It takes a lot longer. I reviewed 28 orgs; it would take me a long time to send 28 emails and communicate with potentially 28 people. (There’s a good chance I would have procrastinated on this and not gotten my post out until next year, which means I would have had to make my 2024 donations without publishing this writeup first.)
Communicating beforehand would make me overly concerned about being nice to the people I talked to, and might prevent me from saying harsh but true things because I don’t want to feel mean.
Orgs can still respond to the post after it’s published, it’s not as if it’s impossible for them to respond at all.
Here are some relevant EA Forum/LW posts (the comments are relevant too):
It depends. I think investing in publicly-traded stocks has a smallish effect on helping the underlying company (see Harris (2022), Pricing Investor Impact). I think investing in private companies is probably much worse and should be avoided.
I think that’s not a reasonable position to hold but I don’t know how to constructively argue against it in a short comment so I’ll just register my disagreement.
Like, presumably China’s values include humans existing and having mostly good experiences.
Scott’s last sentence seems to be claiming that avoiding an arms race is easier than solving alignment (and it would seem to follow from that that we shouldn’t race). But I can see how a politician reading this article wouldn’t see that implication.
That’s the claim I made, yes. Looking again at GovAI’s publications, I’m not sure why I thought that at the time since they do look more like policy research. Perhaps I was taking a strict definition of “policy research” where it only counts if it informs policy in some way I care about.
Right now it looks like my past self was wrong but I’m going to defer to him because he spent more time on it than I’m spending now. I’m not going to spend more time on it because this issue isn’t decision-relevant, but there’s a reasonable chance I was confused about something when I wrote that.
Thank you for the comment! I’ve added a correction to the post.
I don’t know why (I thought it was a good post) but I have some guesses:
Maybe people don’t like the framing of o1 as “lying” when it’s not clear that it lied. All that’s clear is that o1 gave a false justification, which isn’t necessarily lying.
You tend to write in an alarmist rhetorical style that I think turns off a lot of people. I think you are very much correct to be alarmed about AI x-risk, but also I don’t think it’s a good persuasive strategy (for an EA/rationalist audience) to convey this through emotionally charged rhetoric. I didn’t think this particular post was alarmist, but you have a history of writing alarmist posts/comments, so maybe people downvoted based on the title.