Lukas_Gloor

Karma: 6,840

Lukas_Gloor Nov 20, 2023, 11:09 AM
5 points
0 ∶ 0
in reply to: MathiasKB🔸’s comment on: Sam Altman / Open AI Discussion Thread
Hm, very good point! I now think that could be his most immediate motivation. Would feel sad to build something and then see it implode (and also the team to be left in limbo). On reflection, that makes me think maybe Sam doesn’t necessarily look that bad here. I’m sure Microsoft tried to use their leverage to push for changes, and the OpenAI board stood its ground, so it couldn’t have been easy to find a solution that isn’t the company falling apart over disagreements and stuff.

Lukas_Gloor Nov 20, 2023, 10:41 AM
17 points
4 ∶ 0
on: Sam Altman / Open AI Discussion Thread
[Edit: I no longer feel confident about this comment; see thread right below]

Hm, I don’t think Altman looks good here either.

We have to wait and see how public or the media will react, but to me, this looks like it casts doubts on some things he said previously about his specific motivations for building AGI. It’s hard to square that working under Microsoft’s leadership (and their need to compete with other companies like Alphabet) is a good environment for making AGI breakthroughs and thinking that it’ll likely go really well.

Although, maybe he’s planning to only build specific apps for Microsoft rather than intends to build AGI there? That would seem like an atypical reduction of ambition/scope to me. Or maybe the plan is “amass more money and talent and then go back to OpenAI if possible, or otherwise start a new AGI thing with more independence from profit-driven structure. That would be more more understandable, but also feels like he’d be being very agentic about this goal in a way that’s scary and like I have to trust this one person’s judgment about pulling the brakes when it becomes necessary, even though there’s now evidence that many people think he’s not been cautious enough already recently.

I guess we have to wait and see.

Lukas_Gloor Nov 20, 2023, 1:16 AM
8 points
3 ∶ 0
in reply to: Sharmake’s comment on: JWS’s Shortform
This is admittedly a less charitable take than say, Lukas Gloor’s take.
Haha, I was just going to say that I’d be very surprised if the people on the OpenAI board didn’t have access to a lot more info than the people on the EA forum or Lesswrong, who are speculating about the culture and leadership at AI labs from the sidelines.

TBH, if you put a randomly selected EA from a movement of 1,000s of people in charge of the OpenAI board, I would be very concerned that a non-trivial fraction of them probably would make decisions the way you describe. That’s something that EA opinion leaders could maybe think about and address.
But I don’t think most people who hold influential positions within EA (or EA-minded people who hold influential positions in the world at large, for that matter) are likely to be that superficial in their analysis of things. (In particular, I’m strongly disagreeing with the idea that it’s likely that the board “basically had no evidence except speculation from the EA/LW forum”. I think one thing EA is unusually good at – or maybe I should say “some/many parts of EA are unusually good at” – is hiring people for important roles who think for themselves and have generally good takes about things and acknowledge the possibility of being wrong about stuff. [Not to say that there isn’t any groupthink among EAs. Also, “unusually good” isn’t necessarily that high of a bar.])
I don’t know for sure what they did or didn’t consider, so this is just me going off of my general sense for people similar to Helen or Tasha. (I don’t know much about Tasha. I’ve briefly met Helen but either didn’t speak to her or only did small talk. I read some texts by her and probably listened to a talk or two.)

Lukas_Gloor Nov 20, 2023, 12:12 AM
21 points
9 ∶ 0
in reply to: Kevin Lacker’s comment on: Sam Altman fired from OpenAI
This could end up also having really bad consequences for the goals of EA, so it’s perhaps similar to FTX in that way (but things are still developing and it might somehow turn out well).
Or maybe you feel like the board displayed inexperience and that they were in over their heads. I can probably get behind that based on how things look right now (but there’s a chance we learn more details later that put things into a different light).
Still, I feel like inexperience is only unforgivable if it comes combined with hubris. Many commenters seem to think that there must have been hubris involved on the board’s part. To me, that feels like it’s why people seem so mad about this. “Why else would the board have the audicity to oust such a successful and respected CEO, if they cannot point to any smoking-gun-type thing that justifies firing him to the world?”
But notice how that attitude – being risk averse and inclined to just let the experienced tech CEO do his thing without pushback (and possibly amass leverage over the rest of the company and the board by starting or investing into compute startups or stuff like that, as some of the rumors seem to indicate) – is also dangerous and potentially “irresponsible.” It’s not the by-default safe option, after forming concerns about his suitability, to let Sam Altman continue to cash in from the reputational benefits of running OpenAI with the seal of approval from this public good, non-profit, beneficial-mission-focused board structure that OpenAI has installed. This board structure has, from the very start, served as a kind of seal of approval that guarantees a significant amount of goodwill to people who would look at OpenAI skeptically and think “these tech people put the world at risk to attain power/money/the top spot in history.” EAs were arguably quite crucial (via getting Elon Musk to think about AI risk as well as some other pathways) in helping to set up OpenAI with such a board structure and the reputational protection against scrutiny from a concerned public (especially now that AI risk is gaining traction after chat-gpt spooked a bunch of people) that comes with that. So, I mainly want to point out that it’s not obviously “the responsible choice” to not step in when Sam Altman would otherwise de facto keep benefitting from this board structure’s seal of approval, especially if the board that was put in place no longer feels comfortable with his leadership.
To be clear, even if I’m right about the above, this isn’t saying that there wouldn’t have been better ways to handle this. Also, I want to flag that I don’t know what the board members were actually thinking – maybe they did think of this more as coup and less as a “if we put on our board members hats and try to serve our role as well as possible, what should we do?.” In that case, I would disapprove. I don’t know which one applies.

Lukas_Gloor Nov 19, 2023, 3:19 AM
8 points
0 ∶ 0
in reply to: Jelle Donders’s comment on: Sam Altman fired from OpenAI
There’s some related discussion here on LW.

Lukas_Gloor Nov 18, 2023, 11:39 AM
12 points
3 ∶ 0
in reply to: titotal’s comment on: Sam Altman fired from OpenAI
Yeah, now that more information has come to light, it seems to be clearly about disagreements about how to pursue the OpenAI mission. I wonder if the board can point to at least one objectively outrageous thing that Altman was deceptive about, or whether it was more subtle stuff that added up but is hard to convey to outsiders. For instance, I could imagine that they got “empty promises” vibes from Altman where he was placating the most safety-concerned voices at OpenAI by saying he’ll take such and such precautions later in the future, but then kept doing things that are at odds with taking safety seriously, until people had enough and felt deceived and like they could no longer trust his assurances. In this scenario, it’s going to be difficult for the board and for Sutskever to convey that their decision wasn’t some overreaction. (FWIW, I think it can be totally justifiable to fire someone over weasel-like assurances about mission alignment that never led to any visible actions – it’s just tricky that there’s always some plausible deniability where the CEO can say “I was going to take action later, like I said; it’s just that you people are insufficiently pragmatic and don’t have experience dealing with investors like Microsoft; and anyway, the tech isn’t risky enough yet and you all are freaking out.”)

Lukas_Gloor Nov 18, 2023, 12:09 AM
16 points
9 ∶ 0
in reply to: Dave Cortright 🔸’s comment on: Sam Altman fired from OpenAI
This doesn’t seem impossible given the timing, but I’d still be very surprised if this was what the board’s decision was about. (I’m especially skeptical that it would be exclusively about this.) For one thing, the board announcement uses the wording “hindering [the board’s] ability to exercise its responsibilities.” This doesn’t seem like the wording someone would choose if their decision was prompted by investigating events that happened more than twenty years ago and which don’t directly relate to beneficial use of AI or running a company. (Even in the unlikely case where the board decided to open an investigation into abuse allegations and then caught Sam Altman lying about details related to that, it’s not apparent why they would describe these hypothetical lies as “hindering [the board’s] ability to exercise its responsibilities,” as opposed to using wording that’s more just about “lost the board’s trust.”) Besides, I struggle to picture board members starting an investigation solely based on one accusation from when the person in question was still a teenager. I’m not saying that these accusations are for sure unimportant – in fact, I said the opposite on that LW comment thread. It’s just that… Despite the good advice here about how boards should keep a close eye on leadership, I don’t think it’s a board’s role or comparative advantage to focus on investigating stuff like that. Especially once they already have confirmed their standing CEO and in the absence of more direct red flags. (It would maybe be a bit different if this was a CEO selection process and Sam Altman was a new applicant that board members had only little information about.) One option I can see is that, maybe if the board already had other reasons to be concerned, then learning about the accusations could give them further fuel for investigations. Alternatively, though, it seems much more likely to me that this was about other things entirely. (Perhaps something related to publicly announcing that OpenAI “created AGI internally” and then backpedaling it, while also saying that short AI timelines are best for humanity even though an alignment solution is far from in sight?)

Lukas_Gloor Oct 20, 2023, 8:20 AM
2 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: What, exactly, did SBF do?
I see. So, you’re making the argument I described in my earlier reply:
Maybe your point is based on treating FTX and Alameda as two properly segregated entities (even if just for the sake of argument). But then the “allow negative” feature is obviously problematic/in tension with that assumption, as is the fact that FTX no longer has the assets (they paid back some lenders, as opposed to saying “we would be short if you treated us as connected to Alameda, but obviously we aren’t connected to them and we’re not giving Alameda our customers’ money to pay any of their debts”).
You replied to that, “The point of the example is that there were never any assets, it was a naked option.”
But my point was that, even if it was initially a naked option by Alameda, the FTX customer money ended up (partly) gone. Where did it go? (Edit: When I said “as is the fact that FTX no longer has the assets,” I obviously meant their customer’s assets, not “assets put up for collateral.”) It seems that prosecution has evidence that some of it was used to pay off some of Alameda’s debtors. So, the two entities aren’t properly segregated/distinct, and something about Levine’s summary therefore seems misleading. Your account would be accurate if FTX refused to pay any of Alameda’s loans to third parties and therefore still had enough assets to make customers whole.
Or maybe you’re saying that FTX is a bit like the London Metals Exchange in your initial analogy and the fact that its biggest customer made such a big trading loss also affects other customers, because all the money was “numbers on an exchange” and therefore intertwined. In that case, it seems like FTX must have lied about its liquidation engine! They kept advertizing how they have an advanced liquidation engine that prevents situations where large customers making massive trading losses due to fast market movements (and liquidation not happening fast enough) puts other people’s customer funds at risk.
So, I keep believing that the fact that the money is no longer there in a liquid state cannot be explained without FTX lying about at least some things. We know this from FTX’s ToS + their talk about their advanced liquidation engine. It’s as simple as that: you cannot explain that the customer money is no longer there if the liquidation engine is “on” for every customer (including Alameda as the largest customer) and if they didn’t use customer funds to bail out Alameda when Alameda was in the red.
(And we indeed have evidence now that they lied about the liquidation engine; Alameda was exempt from it, but this was not disclosed.)

Lukas_Gloor Oct 19, 2023, 11:58 PM
4 points
1 ∶ 3
in reply to: Ben_West🔸’s comment on: What, exactly, did SBF do?
I feel like that distinction isn’t what’s relevant. I mean, I don’t know what the law is here, but it would be a ridiculous law if it allowed for this sort of loophole.
Say I’m sharing my finances with my wife and I’m buying naked call options on some exchanges without telling her (and then go massively into minus). In that scenario, I’m putting her part of our finances at risk. Call it what you want but she wouldn’t (and shouldn’t!) be happy with the “excuse” that I never explicitly posted her share of the money as collateral.
[Edit: Either way, it seems like our discussion might be about a scenario that didn’t happen. Fermi–Dirac Distribution points out in the comments that, according to the Prosecution, there seem to have been borrowed assets: it looks like Alameda withdrew actual assets with the allow negative feature.]

Lukas_Gloor Oct 19, 2023, 10:58 PM
2 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: What, exactly, did SBF do?
[Edit]: This comment was meant as a reply to your:
Do you think that I am “putting up customer funds as potential collateral” in the example? If so, I’m confused how, because there was no customer funds. If not, then it seems like Alameda could do the same.
I accidentally replied to your earlier comment higher up in the thread.
[/edit]
I’m not sure I’m understanding this right, but:
In this story, if FTX/Alameda (assuming for the moment that they’re acting as the same entity, as is alleged by prosecutors) is analogous to Ben, then it seems like Ben did put customer accounts on the hook by agreeing to a trade that could result in outsized debt.
Maybe your point is based on treating FTX and Alameda as two properly segregated entities (even if just for the sake of argument). But then the “allow negative” feature is obviously problematic/in tension with that assumption, as is the fact that FTX no longer has the assets (they paid back some lenders, as opposed to saying “we would be short if you treated us as connected to Alameda, but obviously we aren’t connected to them and we’re not giving Alameda our customers’ money to pay any of their debts”).
(Relatedly, according to some testimonies for the prosecution, there was a point where lenders asked for their money back, and someone from Alameda – don’t remember who it was but probably Caroline Ellison – asked SBF whether to pay back these lenders knowing that Alameda cannot pay back what it owes FTX, and SBF said to go ahead with some of these payments (but wait with others). That couldn’t be allowed to happen if FTX was properly segregated from Alameda and if the debt was already exceeding FTX’s assets by that point.)

Lukas_Gloor Oct 19, 2023, 8:48 PM
4 points
1 ∶ 3
in reply to: Ben_West🔸’s comment on: What, exactly, did SBF do?
I see what you (and Matt Levine) mean here, but then that still means that Alameda was putting up FTX customer funds as potential collateral in case their trades go poorly. So, in this scenario, they would have been putting customer funds at risk, which IMO isn’t relevantly different from misappropriating them for investments or directly trading with them.

Lukas_Gloor Oct 19, 2023, 3:56 PM
15 points
2 ∶ 2
in reply to: basil.halperin’s comment on: What, exactly, did SBF do?
As others have commented, this strikes me as a misleading summary.
But this story is not exactly right. There was not money sitting in customer accounts that was then transferred to Alameda accounts and squandered. FTX was a futures exchange; it did not keep money in a box for customers. The money in your FTX account was just money that FTX owed you. Nor did Alameda need to steal the money; FTX was a leveraged futures exchange, and traders like Alameda could, in the ordinary course of business, borrow money from FTX based on their crypto positions. The problem, ultimately, at FTX, was that it owed customers a lot of money, but it couldn’t pay them, because Alameda owed FTX a lot of money, but it couldn’t pay it, because it had squandered the money on dumb crypto trades and endorsement deals.
The passage makes it seem like FTX customers agreed to terms of service (ToS) that their money could be lent out. Most customers, for most of FTX customer money, did not.
So, the only way that Alameda can end up with a massive negative balance that’s larger than FTX’s assets and the pool of money where customers agreed that it could lend out, is through breaking of ToS. [Edit: I agree, after the discussion in the comments below, that this paragraph is false without added context. What I should have added is: “I’m operating on the assumption that FTX is bound to their previous advertizing of their advanced liquidation engine that prevents situations where a liquidation of one big customer can be detrimental to other customers who weren’t engaged in trading.”]
In theory, SBF’s defense could argue that they had poor accounting and the position ended up getting slowly bigger and bigger until at some point they overstepped the threshold where they are now clearly breaking ToS. That would still be poor management, but maybe not intentional fraud? However, the prosecution already brought up a bunch of convincing evidence that SBF authorized/decided to increase Alameda’s lending position even at a point where they were already breaking ToS, so unless all that evidence somehow turns out to be based on misrepresentations, that narrative is no longer plausible.
[...] squandered the money on dumb crypto trades and endorsement deals.
And illiquid venture investments. It’s not even clear to me that the crypto trades were a huge problem by themselves. From following the trial, I’m getting the impression that SBF had fear of missing out about investment opportunities (or endorsement deals/advertizing, which is similarly “investing into growth of his empire”) and seemed okay taking big risks and doing dodgy stuff to grow his portfolio faster.
(There is a similar theme in Michael Lewis’s book: SBF chooses the Hoard Dragon in the game Storybook Brawl, who has poor stats in the early game but gets benefits whenever it collects treasure, so it does well in the endgame from its earlier investments and boosts from treasure chests.)

Lukas_Gloor Oct 11, 2023, 12:17 PM
5 points
1 ∶ 0
on: A New Model for Compute Center Verification
Cool post; researching these issues seems like one of the most important things in AI governance to me!
Some questions I have (for future research) are:
1. How hard is it to distinguish approved from unapproved training runs with these snapshots that the chips would provide? Is this just about establishing that the length of the training run is below a certain threshold, or does it assess whether the training run follows a previously submitted recipe that was approved to be safe by an authorizing body?
2. How long would it take to implement these mechanisms at the hardware level, and who would have to be on board to make this happen? (E.g., if the US govt simply passed legislation that prohibited future chip innovation unless these machanisms are installed, would that be enough to get it done?)

Lukas_Gloor Oct 2, 2023, 2:26 PM
2 points
0 ∶ 0
in reply to: AnonResearcherMajorAILab’s comment on: Aim for conditional pauses
Oh we disagree much more straightforwardly. I think the 89% should be going up, not down. That seems by far the most important disagreement.
I think we agree (at least as far as my example model with the numbers was concerned). The way I meant it, 82% goes up to 89%.
(My numbers were confusing because I initially said 89% was my all-things-considered probability, but in this example model, I was giving 89% as the probability for a scenario where we take a (according to your view) suboptimal action. In the example model, 82% is the best chance we can get with optimal course of action, but it comes at the price of way higher risk of death in the first five years.)
In any case, my assumptions for this example model were:
(1) Public advocacy is the only way to install an ambitious pause soon enough to reduce risks that happen before 5 years.
(2) If it succeeds at the above, public advocacy will likely also come with negative side effects that increase the risks later on.
And I mainly wanted to point out how, from a person-affecting perspective, the difference between 82% and 89% isn’t necessarily huge, whereas getting 5 years of zero risk vs 5 years of 55% cumulative risks feels like something that could matter a lot.
But one can also discuss the validity of (1) and (2). It sounds like you don’t buy (1) at all. By contrast, I think (1) is plausible but I’m anot confident in my stance here, and you raise good points.
Regarding (2), I probably agree that if you achieve an ambitious pause via public advocacy and public pressure playing a large role, this makes some things harder later.
But why do evals matter? What’s an example story where the evals prevent Molochian forces from leading to us not being in control? I’m just not seeing how this scenario intervenes on your threat model to make it not happen.
Evals prevent the accidental creation of misaligned transformative AI by the project that’s authorized to go beyond the compute cap for safety research (if necessary; obviously they don’t have to go above the cap if the returns from alignment research are high enough at lower levels of compute).
Molochian forces are one part of my threat model, but I also think alignment difficulty is high and hard takeoff is more likely than soft takeoff. (Not all these components to my worldview are entirely independent. You could argue that being unusually concerned about Molochian forces and expecting high alignment difficulty is both produced by the same underlying sentiment. Arguably, most humans aren’t really aligned with human values, for Hansonian reasons, which we can think of as a subtype of Moloch problems. Likewise, if it’s already hard to align humans to human values, it’ll probably also be hard to align AIs to those values [or at least create an AI that is high-integrity friendly towards humans, while perhaps pursuing some of its own aims as well – I think that would be enough to generate a good outcome, so we don’t necessarily have to create AIs that care about nothing else besides human values].)

Lukas_Gloor Oct 1, 2023, 11:48 AM
2 points
0 ∶ 0
in reply to: AnonResearcherMajorAILab’s comment on: Aim for conditional pauses
But I also wish you’d say what exactly your alternative course of action is, and why it’s better. E.g. the worry of “algorithmic progress gets you to the threshold” also applies to unconditional pauses. Right now your comments feel to me like a search for anything negative about a conditional pause, without checking whether that negative applies to other courses of action.
The way I see it, the main difference between conditional vs unconditional pause is that the unconditional pause comes with a bigger safety margin (as big as we can muster). So, given that I’m more worried about surprising takeoffs, that position seems prima facie more appealing to me.

In addition, as I say in my other comment, I’m open to (edit: or, more strongly, I’d ideally prefer this!) some especially safety-conscious research continuing onwards through the pause. I gather that this is one of your primary concerns? I agree that an outcome where that’s possible requires nuanced discourse, which we may not get if public reaction to AI goes too far in one direction. So, I agree that there’s a tradeoff around public advocacy.

Lukas_Gloor Oct 1, 2023, 11:37 AM
2 points
0 ∶ 0
in reply to: AnonResearcherMajorAILab’s comment on: Aim for conditional pauses
That sounds like a rephrasing of what I said that puts a positive spin on it. (I don’t see any difference in content.)
Yeah, I just wanted to say that my position doesn’t require public discourse to turn out to be surprisingly nuanced.
To put it another way—you’re critiquing “optimistic EAs” about their attitudes towards institutions, but presumably they could say “we get lucky in terms of the institutional response being a good one given the circumstances”. What’s the difference between your position and theirs?
I’m hoping to get lucky but not expecting it. The more optimistic EAs seem to be expecting it (otherwise they would share more pessimism).
I don’t think people would be on board with the principle “we’ll reduce the risk of doom in 2028, at the cost of increasing risk of doom in 2033 by a larger amount”.
If no one builds transformative AI in a five-year period, the risk of doom can be brought down to close to 0%. By contrast, once we do build it (with that five years delay), if we’re still pessimistic about the course of civilization, as I am and I was taking as an assumption for my comments about how worldviews would react to these tradeoffs, then the success chances five years later will still be bad. (EDIT:) So, the tradeoff is something like 55% death spread over a five-year period vs no death for five years, for an eventual reward of reducing total chance of death (over a 10y period or whatever) from 89% to 82%. (Or something a bit lower; I probably place >20% on us somehow not getting to TAI even in 10 years.)
In that scenario, I could imagine that many people would go for the first option. Many care more about life as they know it than for chances of extreme longevity and awesome virtual worlds. (Some people would certainly change their minds if they thought about the matter a lot more under ideal reasoning conditions, but I expect many [including me when I think self-orientedly] wouldn’t.)
(I acknowledge that, given your more optimistic estimates about AI going well and about low imminent takeoff risk, person-affecting concerns seem highly aligned with long-termist concerns.)
Your threat model seems to be “Moloch will cause doom by default, but with AI we have one chance to prevent that, but we need to do it very carefully”. But Molochian forces grow much stronger as you increase the number of actors! The first intervention would be to keep the number of actors involved as small as possible, which you do by having the few leaders race forward as fast as possible, with as much secrecy as possible. If this were my main threat model I would be much more strongly against public advocacy and probably also against both conditional and unconditional pausing.
That’s interesting. I indeed find myself thinking that our best chances of success come from a scenario where most large-model AI research shuts down, but a new org of extremely safety-conscious people is formed where they make progress with a large lead. I was thinking of something like the scenario you describe as “Variant 2: In addition to this widespread pause, there is a tightly controlled and monitored government project aiming to build safe AGI.” It doesn’t necessarily have to be government-led, but maybe the government has talked to evals experts and demands a tight structure where large expenditures of compute always have to be approved by a specific body of safety evals experts.
>>But my sense is that it’s unlikely we’d succeed at getting ambitious measures in place without some amount of public pressure.
Why do you believe that?
I don’t know. Maybe I’m wrong: If the people who are closest to DC are optimistic that lawmakers would be willing to take ambitious measures soon enough, then I’d update that public advocacy has fewer upsides (while the downsides remain). I was just assuming that the current situation is more like “some people are receptive, but there’s also a lot of resistance from lawmakers.”

Lukas_Gloor Sep 30, 2023, 1:23 PM
17 points
14 ∶ 1
in reply to: aprilsun’s comment on: The Lighthaven Campus is open for bookings
So I’m curious if community opinion is generally that the EVF purchase was “bad” and the Lightcone purchase was “good”?
I didn’t get the sense that there’s a community-consensus about the castle as a convenient event venue that you can resell at some point later to make up parts of the costs. Some people were very vocal in their outrage, but many thought it might be totally fine or is at least defensible even if it was a mistake.
It could be that “third wave EA” will contain a norm of accepting that people have different takes on things like that. For example, Lightcone has lately been critical of the direction EA is going, but them advertizing their new spaces here on the forum suggests that they’re happy to collaborate with parts of the community that shares enough of their worldview/assumptions.

Lukas_Gloor Sep 30, 2023, 12:53 PM
2 points
0 ∶ 0
in reply to: AnonResearcherMajorAILab’s comment on: Aim for conditional pauses
A conditional pause fails to prevent x-risk if either:
- The AI successfully exfiltrates itself (which is what’s needed to defeat rollback mechanisms) during training or evaluation, but before deployment.
- The AI successfully sandbags the evaluations. (Note that existing conditional pause proposals depend on capability evaluations, not alignment evaluations.)
Another way evals could fail is if they work locally but it’s still too late in the relevant sense because even with the pause mechanism kicking in (e.g., “from now on, any training runs that use 0.2x the compute of the model that failed the eval will be prohibited”), algorithmic progress at another lab or driven by people on twitter(/X) tinkering with older, already-released models, will get someone to the same capability threshold again soon enough. There’s probably a degree to which algorithmic insights can substitute for having more compute available. So, even if a failed eval triggers a global pause, if cognitive takeoff dynamics are sufficiently fast/”discontinuous,” a failed eval may just mean that we’re now so close to takeoff that it’s too hard to prevent. Monitoring algorithmic progress is particularly difficult (I think compute is the much better lever).
Of course, you can still incorporate this concern by adding enough safety margin to when your evals trigger the pause button. Maybe that’s okay!
But then we’re back to my point of “Are we so sure that the evals (with the proper safety margin) shouldn’t have triggered today already?”
(It’s not saying that I’m totally confident about any of this. Just flagging that it seems like a tough forecasting question to determine what an appropriate safety margin is on the assumption that algorithmic progress is hard to monitor/regulate and the assumption that fast takeoff into and through the human range is very much a possibility.)

Lukas_Gloor Sep 30, 2023, 12:12 PM
4 points
1 ∶ 0
in reply to: AnonResearcherMajorAILab’s comment on: Aim for conditional pauses
From my perspective you’re assuming we’re in an unlikely world where the public turns out to be saner than our baseline would suggest.
Hm, or that we get lucky in terms of the public’s response being a good one given the circumstances, even if I don’t expect the discourse to be nuanced. It seems like a reasonable stance to think that a crude reaction of “let’s stop this research before it’s too late” is appropriate as a first step, and that it’s okay to worry about other things later on. The negatives you point out are certainly significant, so if we could get a conditional pause setup through other channels, that seems clearly better! But my sense is that it’s unlikely we’d succeed at getting ambitious measures in place without some amount of public pressure. (For what it’s worth, I think the public pressure is already mounting, so I’m not necessarily saying we have to ramp up the advocacy side a lot – I’m definitely against forming PETA-style anti-AI movements.)
As a first pass heuristic, I think it still seems pretty unreasonable all the way up to p(doom) of < 90%,
It also matters how much weight you give to person-affecting views (I’ve argued here for why I think they’re not unreasonable). If we can delay AI takeoff for five years, that’s worth a lot from the perspective of currently-existing people! (It’s probably also weakly positive or at least neutral from a suffering-focused longtermist perspective because everything seems uncertain from that perspective and a first-pass effect is delaying things from getting bigger; though I guess you could argue that particular s-risks are lower if more alignment-research-type reflection goes into AI development.) Of course, buying a delay that somewhat (but not tremendously) worsens your chances later on is a huge cost to upside-focused longtermism. But if we assume that we’re already empirically pessimistic on that view to begin with, then it’s an open question how a moral parliament between worldviews would bargain things out. Certainly the upside-focused longtermist faction should get important concessions like “try to ensure that actually-good alignment research doesn’t fall under the type of AI research that will be prohibited.”
What is your p(doom)?
My all-things-considered view (the one I would bet on) is maybe 77%. My private view (what to report to avoid double-counting the opinions of people the community updates towards) is more like 89%. (This doesn’t consider scenarios where AI is misaligned but still nice towards humans for weird decision-theoretic reasons where the AI is cooperating with other AIs elsewhere in the multiverse – not that I consider that particularly likely, but I think it’s too confusing to keep track of that in the same assessment.)
Some context on that estimate: When I look at history, I don’t think of humans being “in control” of things. I’m skeptical of Stephen Pinker’s “Better Angels” framing. Sure, a bunch of easily measurable metrics got better (though some may even be reversing for the last couple of years, e.g., life expectancy is lower now than it used to be before Covid). However, at the same time, technological progress introduced new problems of its own that don’t seem anywhere close to being addressed (e.g., social media addiction, increases in loneliness, maybe polarization via attention-grabbing news). Even if there’s an underlying trend of “Better Angels,” there’s also a trend of “new technology increases the strength of Molochian forces.” We seem to be losing that battle! AI is an opportunity to gain control for the first time via superhuman intelligence/rationality/foresight to bail us out and reign in Molochian forces once and for all, but to get there, we have to accomplish an immense feat of coordination. I don’t see why people are by-default optimistic about something like that. If anything, my 11% that humans will gain control over history for the first time ever seems like the outlandish prediction here! The 89% p(doom) is more like what we should expect by default: things get faster and out of control and then that’s it for humans.

Lukas_Gloor 30 Sep 2023 9:49 UTC
3 points
1 ∶ 0
in reply to: AnonResearcherMajorAILab’s comment on: Aim for conditional pauses
I don’t think this depends on takeoff speeds at all, since I’d expect a conditional pause proposal to lead to a pause well before models are automating 20% of tasks (assuming no good mitigations in place by that point).
I should have emphasized that I’m talking about cognitive AI takeoff, not economic takeoff.
I don’t have a strong view whether there are ~20% of human tasks that are easy and regular/streamlined enough to automate with stochastic-parrot AI tools. Be that as it may, what’s more important is what happens once AIs pass the reliability threshold that makes someone a great “general assistant” in all sorts of domains. From there, I think it’s just a tiny step further to also being a great CEO. Because these capability levels are so close to each other on my model, the world may still look similar to ours at that point.
All of that said, it’s not like I consider it particularly likely that a system would blow past all the evals you’re talking about in a single swoop, especially since some of them will be (slightly) before the point of being a great “general assistant.” I also have significant trust that the people designing these evals will be thinking about these concerns. I think it’s going to be very challenging to make sure evals organizations (or evals teams inside labs in case it’s done lab-internally) have enough political power and stay uncorrupted by pressures to be friendly towards influential lab leadership. These problems are surmountable in theory, but I think it’ll be hard, so I’m hoping the people working on this are aware of all that could go wrong. I recently wrote up some quick thoughts on safety evals here. Overall, I’m probably happy enough with a really well-thought out “conditional pause” proposal, but I’d need to be reassured that the people who decide in favor of that can pass the Ideological Turing test for positions like fast takeoff or the point that economic milestones like “20% of tasks are automated” are probably irrelevant.