Iâm a former software engineer from Brisbane, Australia who successfully pivoted into AI safety. I spent eighteen months at the UK AISI, and now work at Arcadia Impact.
Jay Baileyđ¸
Karma: 1,152
My p(doom) went down slightly (From around 30% to around 25%) mainly as a result of how GPT-4 caused governments to begin taking AI seriously in a way I didnât predict. My timelines havenât changedâthe only capability increase of GPT-4 that really surprised me was its multimodal nature. (Thus, governments waking up to this was a double surprise, because it clearly surprised them in a way that it didnât surprise me!)
Iâm also less worried about misalignment and more worried about misuse when it comes to the next five years, due to how LLMâs appear to behave. It seems that LLMâs arenât particularly agentic by default, but can certainly be induced to perform agent-like behaviourâGPT-4â˛s inability to do this well seems to be a capability issue that I expect to be resolved in a generation or two. Thus, Iâm less worried about the training of GPT-N but still worried about the deployment of GPT-N. It makes me put more credence in the slow takeoff scenario.
This also makes me much more uncertain about the merits of pausing in the short-term, like the next year or two. I expect that if our options were âPause nowâ or âPause after another year or twoâ, the latter is better. In practice, I know the world doesnât work that way and slowing down AI now likely slows down the whole timeline, which complicates things. I still think that government efforts like the UKâs AISI are net-positive (Iâm joining them for a reason, after all) but I think a lot of the benefit to reducing x-risk here is building a mature field around AI policy and evaluations before we need itâif we wait until I think the threat of misaligned AI is imminent, that may be too late.
This is exactly right, and the main reason I wrote this up in the first place. I wanted this to serve as a data point for people to be able to say âOkay, things have gone a little off the rails, but things arenât yet worse than they were for Jay, so weâre still probably okay.â Note that it is good to have a plan for when you should give up on the field, tooâit should just allow for some resilience and failures baked in. My plan was loosely âIf I canât get a job in the field, and I fail to get funded twice, I will leave the fieldâ.
Also contributing to positive selection effects is that youâre more likely to see the more impressive results in the field, because theyâre more impressive. That gives your brain a skewed idea of what the median person in the field is doing. Our brain thinks âAverage piece of alignment research we seeâ is âAverage output of alignment researchersâ.
The counterargument to this is âWell, shouldnât we be aiming for better than median? Shouldnât these impressive pieces be our targets to reach?â I think so, yes, but I believe in incremental ambition as wellâif one is below-average in the field, aiming to be median first, then good, then top-tier rather than trying to immediately be top-tier seems to me a reasonable approach.
ReflecÂtions on my first year of AI safety research
Welcome to the Forum!
This post falls into a pretty common Internet failure mode, which is so ubiquitous outside of this forum that itâs easy to not realise that any mistake has even been madeâafter all, everyone talks like this. Specifically, you donât seem to consider whether your argument would convince someone who genuinely believes these views. I am only going to agree with your answer to your trolley problem if I am already convinced invertebrates have no moral value...and in that case, I donât need this post to convince me that invertebrate welfare is counterproductive. There isnât any argument for why someone who does not currently agree with you should change their mind.
It is worth considering what specific reasons people who care about invertebrate reasoning have, and trying to answer those views directly. This requires putting yourself in their shoes and trying to understand why they might consider invertebrates to have actual moral worth.
âSo whatâs the problem? Why donât I just let the invertebrate-lovers go do their thing, while I do mine? The problem is that those arguing for the invertebrate cause as an issue of moral importance have brought bad arguments to the table.â
This is much more promising, and Iâd like to see actual discussion of what these arguments are, and why theyâre bad.
Great post! I definitely feel similar regarding givingâwhile giving cured my guilt about my privileged position in the world, I donât feel as amazing as I thought I would when givingâit is indeed a lot like taxes. I feel like a better person in the background day-to-day, but the actual giving now feels pretty mundane.
Iâm thinking I might save up my next donation for a few months and donate enough to save a full life in one goâbecause of a quirk in human brains I imagine that would be more satisfying than saving 20% of a life 5 times.
For the Astra Fellowship, what considerations do you think people should be thinking about when deciding to apply for SERI MATS, Astra Fellowship, or both? Why would someone prefer one over the other, given theyâre both happening at similar times?
âAll leading labs coordinate to slow during crunch time: great. This delays dangerous AI and lengthens crunch time. Ideally the leading labs slow until risk of inaction is as great as risk of action on the margin, then deploy critical systems.
All leading labs coordinate to slow now: bad. This delays dangerous AI. But it burns leading labsâ lead time, making them less able to slow progress later (because further slowing would cause them to fall behind, such that other labs would drive AI progress and the slowed labsâ safety practices would be irrelevant).â
I would be more inclined to agree with this if there was a set of criteria we had that indicated we were in âcrunch timeâ which we are very likely to meet before dangerous systems and havenât met now. Have people generated such a set? Without that, how do we know when âcrunch timeâ is, or for that matter, if weâre already here?
- Timelines are short, p(doom) is high: a global stop to fronÂtier AI deÂvelÂopÂment unÂtil x-safety conÂsenÂsus is our only reaÂsonÂable hope by (12 Oct 2023 11:24 UTC; 78 points)
- 's comment on Timelines are short, p(doom) is high: a global stop to fronÂtier AI deÂvelÂopÂment unÂtil x-safety conÂsenÂsus is our only reaÂsonÂable hope by (13 Oct 2023 8:48 UTC; -2 points)
Great post! Another thing worth pointing out is another advantage of giving yourself capacity. I try to operate at around 80-90% capacity. This allows me time to notice and pursue better opportunities as they arise, and imo this is far more valuable to your long-term output than a flat +10% multiplier. As we know from EA resources, working on the right thing can multiply your effectiveness by 2x, 10x, or more. Giving yourself extra slack makes you less likely to get stuck in local optima.
Thanks Elle, I appreciate that. I believe your claimsâI fully believe itâs possible to safely go vegan for an extended period, Iâm just not sure how difficult it is (i.e, whatâs the default outcome, if one tries without doing research first) and what ways there are to prevent that outcome if the outcome is not good.
I shall message you, and welcome to the forum!
With respect to Point 2, I think that EA is not large enough that a large AI activist movement would be comprised mostly of EA aligned people. EA is difficult and demandingâI donât think youâre likely to get a âOne Million EAâ march anytime soon. I agree that AI activists who are EA aligned are more likely to be in the set of focused, successful activists (Like many of your friends!) but I think youâll end up with either:
- A small group of focused, dedicated activists who may or may not be largely EA aligned
- A large group of unfocused-by-default, relatively casual activists, most of whom will not be EA aligned
If either of those two would be effective at achieving goals, then I think that makes AI risk activism a good idea. If you need a large group of focused, dedicated activistsâI donât think weâre going to get that.
As for Point 1, itâs certainly possibleâespecially if having a large group of relatively unfocused people would be useful. I have no idea if this is true, so I have no idea if raising awareness is an impactful idea at this point. (Also, there are those that have made the point that raising AI risk awareness tends to make people more likely to race for AGI, not lessâsee OpenAI)
I think thereâs a bit of an âugh fieldâ around activism for some EAâs, especially the rationalist types in EA. At least, thatâs my experience.
My first instinct, when I think of activism, is to think about people who:
- Have incorrect, often extreme beliefs or ideologies.
- Are aggressively partisan.
- Are more performative than effective with their actions.
This definitely does not describe all activists, but it does describe some activists, and may even describe the median activist. That said, this shouldnât be a reason for us to discard this idea immediately out of handâafter all, how good is the median charity? Not that great compared to what EAâs actually do.
Perhaps thereâs a mass-movement issue here thoughâactivism tends to be best with a large groundswell of numbers. If you have a hundred thousand AI safety activists, youâre simply not going to have a hundred thousand people with a nuanced and deep understanding of the theory of change behind AI safety activism. Youâre going to have a few hundred of those, and ninety nine thousand people who think AI is bad for Reason X, and thatâs the extent of their thinking, and X varies wildly in quality.
Thus, the question isâwould such a movement be useful? For such a movement to be useful, it would need to be effective at changing policy, and it would need to be aimed at the correct places. Even if the former is true, I find myself skeptical that the latter would occur, since even AI policy experts are not yet sure where to aim their own efforts, let alone how to communicate where to aim so well that a hundred thousand casually-engaged people can point in the same useful direction.
I am one of those meat-eating EAâs, so I figured Iâd give some reasons why Iâm not vegan, to aid the goals of this post in finding out about these things.
Price: While I can technically afford it, I still prefer to save money when possible.
Availability: A lot of food out there, especially frozen foods which I buy a lot of since I donât like cooking, involves meat. Itâs simply easier to decide on meals when meat is an option.
Knowledge: If I were to go vegan, I would be unsure how to go vegan safely for an extended period, and how to make sure I got a decent variety rather than eating the same foods over and over (which comes into tasteâI donât mind vegan food but thereâs much more variety I can find in meat-based dishes)
Convenience: Similarly to aboveâit takes resources to seek out vegan options, more resources than to just eat normally.
The harms are real, but the harms are far away and abstract. So when I feel vaguely guilty about eating meat, I think about all the hassle and cost it would take to swap diets, and I shy away from it and donât do it.
Iâm not quite sure why those harms are far away and abstract, whereas the harms caused by malaria or AI risk donât invoke the same feelings in me. I think itâs because I can use maths to determine the number of humans impacted and then put myself in the place of one of those humansâitâs harder to do that with chickens. Also, giving away 10% of my income is actually less of a day-to-day drain on my resources than going vegan would be. I feel aversion to spending money, but I only give away money once a month, and it doesnât cause me financial hardship. By contrast, veganism requires daily effort.
As a micro-example of where these considerations donât applyâthere are some plant meat based strips that I can get at my local supermarket. I find them tastier than actual meat when put into curry, and theyâre just as cheap when on special. So whenever theyâre on special, I pick a bunch of them up and they become my default option for a while. I know how to cook them, I know where to get them, theyâre just as cheap (sometimes) and I enjoy the taste. So I end up avoiding meat by default. I hope plant-based meat will eventually reach that saturation point for all kinds of dishes too.
Seems like a pretty incredible opportunity for those interested! What level of time commitment do you expect reading and understanding the book to take, in addition to the meetings?
Looking at the two critiques in reverse order:
I think itâs true that itâs easy for EAâs to lose sight of the big picture, but to me, the reason for that is simpleâhumans, in general, are terrible at seeing the bigger picture. If anything, it seems to me that the EA frameworks are better than most altruistic endeavours at seeing the big pictureâmost altruistic endeavors donât get past the stage of âSee good thing and do itâ, whereas EAâs tend to be asking if X is really the most effective thing they can do, which invariably involves looking at a bigger picture than the immediate thing. In my own field of AI safety, thinking about the big picture is an idea that people are routinely exposed to. Researchers often do exercises like backchaining (ask what the main goal is, like âMake AI go wellâ and figure out how to move backwards from that to what you should be doing now) and theory of change (Writing out specifically what problem you want to help, what you want to achieve, and how that will help)
Do you think there are specific vulnerabilities that EAâs have that make them lose sight of the bigger picture, that non-EA altruistic people avoid?
For the point of foregoing fulfillmentâIâm not sure exactly what fulfillment you think people are foregoing, here. Is it the fulfillment of having lots of money? The fulfillment of working directly on the worldâs biggest problems?
I was using unidentifiability in the Hubinger way. I do believe that if you try to get an AI trained in the way you mention here to follow directions subject to ethical considerations, by default, the things it considers âmaximally ethicalâ will be approximately as strange as the sentences from above.
That said, this is not actually related to the problem of deceptive alignment, so I realise now that this is very much a side point.
I donât understand why you believe unidentifiability will be prevented by large datasets. Take the recent SolidGoldMagikarp work. It was done on GPT-2, but GPT-2 nevertheless was trained on a lot of dataâa quick Google search suggests eight million web pages.
Despite this, when people tried to find the sentences that maximally determined the next token, what we got was...strange.
This is exactly the kind of thing I would expect to see if unidentifiability was a major problemâwhen we attempt to poke the bounds of extreme behaviour of the AI and take it far off distribution as a result, what we get is complete nonsense and not at all correlated with what we actually want. Clearly it understands the concepts of âgirlâ, âUSAâ, and âevilâ very differently to us, and not in a way we would endorse.
This is far from a guarantee that unidentifiability will remain a problem, but considering your position is under 1%, things like this seem to add much more credence to unidentifiability in my world model than you give it.
Thanks for this! One thing I noticed is there is an assumption youâll continue to donate 10% of your current salary even after retirementâit would be worth having that as a toggle to turn that off, since the GWWC pledge does say âuntil I retireâ. That may make giving more appealing as well, because giving 10% forever requires longer timelines than giving 10% until retirementâwhen I did the calcs in my own spreadsheet I only increased my working timeline by about 10% by committing to give 10% until retiring.
Admittedly, now Iâm rethinking the whole âretire earlyâ thing entirely given the impact of direct work, but this outside the scope of one spreadsheet :P
This came from going through AGI Safety Fundamentals (and to a lesser extent, Alignment 201) with a discussion group and talking through the various ideas. I also read more extensively in most weeks in AGISF than the core readings. I think the discussions were a key part of this. (Though itâs hard to tell since I donât have access to a world where I didnât do thatâthis is just intuition)
Great stuff! Thanks for running this!
Minor point: The Discovering Latent Knowledge Github appears empty.
Also, regarding the data poisoning benchmark, This Is Fine, Iâm curious if this is actually a good benchmark for resistance to data poisoning. The actual thing we seem to be measuring here is speed of transfer learning, and declaring that slower is better. While slower speed of learning does increase resistance to data poisoning, it also seems bad for everything else we might want our AI to do. To me, this is basically a fine-tuning benchmark that weâve then inverted. (After all, if a neural network always outputted the number 42 no matter what, it would score the maximum on TIFâthere is no sequence of wrong prompts that can cause it to elicit the number 38 instead, because it is incapable of learning anything. Nevertheless, this is not where we want LLMâs to go in the future.)
A better benchmark would probably be to take data poisoned examples and real fine-tuning, fine-tune the model on each, and compare how much it learns in both cases. With current capabilities, it might not be possible to score above baseline on this benchmark since I donât know if we actually have ways for the model to filter out data poisoned examplesânevertheless, this would bring awareness to the problem and actually measure what we want to measure more accurately.
I live in Australia, and am interested in donating to the fundraising efforts of MIRI and Lightcone Infrastructure, to the tune of $2,000 USD for MIRI and $1,000 USD for Lightcone. Neither of these are tax-advantaged for me. Lightcone is tax advantaged in the US, and MIRI is tax advantaged in a few countries according to their website.
Anyone want to make a trade, where I donate the money to a tax-advantaged charity in Australia that you would otherwise donate to, and you make these donations? As I understand it, anything in Effective Altruism Australia would work. Since my tax bill is expected to be about 1/â3rd this year, Iâm open to matching up to 4.5k USD for this instead of 3k, which will cost me about 3k in the long run. This will not funge against my existing 10% donations to global health, itâs on top of them, so you get all that sweet, sweet counterfactual impact.
If heâs still around, @Mitchell Laughlinđ¸ can confirm that Iâve successfully done a match like this before across a longer timeframe and larger total amount.