Now: TYPE III AUDIO.
Previously: 80,000 Hours (2014-15; 2017-2021) Worked on web development, product management, strategy, internal systems, IT security, etc.
Before that: My CV.
Side-projects: Inbox When Ready; Radio Bostrom; The Valmy; Comment Helper for Google Docs.
Andrew Mayne points out that “the base model for ChatGPT (GPT 3.5) had been publicly available via the API since March 2022”.
On (1): it’s very unclear how ownership could be compatible with no financial interest.
Maaaaaybe (2) explains it. That is: while ownership does legally entail financial interest, it was agreed that this was only a pragmatic stopgap measure, such that in practice Sam had no financial interest.
For context:
OpenAI claims that while Sam owned the OpenAI Startup Fund, there was “no personal investment or financial interest from Sam”.
In February 2024, OpenAI said: “We wanted to get started quickly and the easiest way to do that due to our structure was to put it in Sam’s name. We have always intended for this to be temporary.”
In April 2024 it was announced that Sam no longer owns the fund.
Sam didn’t inform the board that he owned the OpenAI Startup Fund, even though he constantly was claiming to be an independent board member with no financial interest in the company.
Sam has publicly said he has no equity in OpenAI. I’ve not been able to find public quotes where Sam says he has no financial interest in OpenAI (does anyone have a link?).
From the interview:
When ChatGPT came out, November 2022, the board was not informed in advance about that. We learned about ChatGPT on Twitter.
Several sources have suggested that the ChatGPT release was not expected to be a big deal. Internally, ChatGPT was framed as a “low-key research preview”. From The Atlantic:
The company pressed forward and launched ChatGPT on November 30. It was such a low-key event that many employees who weren’t directly involved, including those in safety functions, didn’t even realize it had happened. Some of those who were aware, according to one employee, had started a betting pool, wagering how many people might use the tool during its first week. The highest guess was 100,000 users.
If that’s true, then perhaps it wasn’t ex ante above the bar to report to the board.
You wrote:
[OpenAI do] very little public discussion of concrete/specific large-scale risks of their products and the corresponding risk-mitigation efforts (outside of things like short-term malicious use by bad API actors, where they are doing better work).
This doesn’t match my impression.
For example, Altman signed the CAIS AI Safety Statement, which reads:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
The “Preparedness” page—linked from the top navigation menu on their website—starts:
The study of frontier AI risks has fallen far short of what is possible and where we need to be. To address this gap and systematize our safety thinking, we are adopting the initial version of our Preparedness Framework. It describes OpenAI’s processes to track, evaluate, forecast, and protect against catastrophic risks posed by increasingly powerful models.
The page mentions “cybersecurity, CBRN (chemical, biological, radiological, nuclear threats), persuasion, and model autonomy”. The framework itself goes into more detail, proposing scorecards for assessing risk in each category. They define “catastrophic risk” as “any risk which could result in hundreds of billions of dollars in economic damage or lead to the severe harm or death of many individuals—this includes, but is not limited to, existential risk”. The phrase “millions of deaths” appears in one of the scorecards.
Their “Planning for AGI & Beyond” blog post describes the risks as “existential”, I quote the relevant passage in another comment.
On their “Safety & Alignment” blog they highlight recent posts called Reimagining secure infrastructure for advanced AI and Building an early warning system for LLM-aided biological threat creation.
My sense is that there are many other examples, but I’ll stop here for now.
OpenAI’s “Planning for AGI & Beyond” blog post includes the following:
As our systems get closer to AGI, we are becoming increasingly cautious with the creation and deployment of our models. Our decisions will require much more caution than society usually applies to new technologies, and more caution than many users would like. Some people in the AI field think the risks of AGI (and successor systems) are fictitious; we would be delighted if they turn out to be right, but we are going to operate as if these risks are existential.
At some point, the balance between the upsides and downsides of deployments (such as empowering malicious actors, creating social and economic disruptions, and accelerating an unsafe race) could shift, in which case we would significantly change our plans around continuous deployment.
[...]
The first AGI will be just a point along the continuum of intelligence. We think it’s likely that progress will continue from there, possibly sustaining the rate of progress we’ve seen over the past decade for a long period of time. If this is true, the world could become extremely different from how it is today, and the risks could be extraordinary. A misaligned superintelligent AGI could cause grievous harm to the world; an autocratic regime with a decisive superintelligence lead could do that too.
[...]
Successfully transitioning to a world with superintelligence is perhaps the most important—and hopeful, and scary—project in human history. Success is far from guaranteed, and the stakes (boundless downside and boundless upside) will hopefully unite all of us.
Altman signed the CAIS AI Safety Statement, which reads:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
In 2015 he wrote a blog post which begins:
Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity.
Yes. (4) and (11) are also very much “citation needed”. My sense is that they would need to be significantly moderated to fit the facts (e.g. the profit cap is still a thing).
On short stories, two notable examples: Narrative Ark by Richard Ngo and the regular “Tech Tales” section at the end of Jack Clark’s Import AI newsletter.
I read the paper, then asked Claude 3 to summarise. I endorse the following summary as accurate:
The key argument in this paper is that Buddhism is not necessarily committed to pessimistic views about the value of unawakened human lives. Specifically, the author argues against two possible pessimistic Buddhist positions:
The Pessimistic Assumption—the view that any mental state characterized by dukkha (dissatisfaction, unease, suffering) is on balance bad.
The Pessimistic Conclusion—the view that over the course of an unawakened life, dukkha will always outweigh the good.
Against the Pessimistic Assumption, the author contends that even if an experience contains some element of dukkha, it can still be good overall if its positive features outweigh the negative ones. The mere presence of dukkha does not make an experience bad on balance.
Against the Pessimistic Conclusion, the author argues that once we reject the Pessimistic Assumption and allow that some discrete unawakened experiences can be good overall, then given the right causes and conditions, it’s possible for a whole unawakened life to be good on balance, even if it necessarily contains a nontrivial amount of dukkha.
The core claim is that while Buddhism regards dukkha as pervasive in unawakened existence, this is compatible with unawakened lives sometimes being good overall. Dukkha is an inevitable bad-making feature, but other good-making features can potentially outweigh it. Thus, Buddhism need not be committed to a pessimistic assessment of unawakened human lives.
Thanks for this.
I take the central claim to be:
Even if an experience contains some element of dukkha, it can still be good overall if its positive features outweigh the negative ones. The mere presence of dukkha does not make an experience bad on balance.
I agree with this, and also agree that it’s often overlooked.
The completion rate at BlueDot Impact averaged out at about 75%
How do you define completion?
I think so. I’ll put a note about this at the top of the post.
Perhaps it’s just the case that the process of moral reflection tends to cause convergence among minds from a range of starting points, via something like social logic plus shared evolutionary underpinnings.
Yes. And there are many cases where evolution has indeed converged on solutions to other problems[1].
Some examples:
(Copy-pasted from Claude 3 Opus. They pass my eyeball fact-check.)
Wings: Birds, bats, and insects have all independently evolved wings for flight, despite having very different ancestry.
Eyes: Complex camera-like eyes have evolved independently in vertebrates (like humans) and cephalopods (like octopuses and squids).
Echolocation: Both bats and toothed whales (like dolphins) have evolved the ability to use echolocation for navigation and hunting, despite being unrelated mammals.
Venomous spines: Both porcupines (mammals) and hedgehogs (also mammals, but not closely related to porcupines) have evolved sharp, defensive spines.
Fins: Sharks (cartilaginous fish) and dolphins (mammals) have independently evolved similar fin shapes and placement for efficient swimming.
Succulence: Cacti (native to the Americas) and euphorbs (native to Africa) have independently evolved similar water-storing, fleshy stems to adapt to arid environments.
Flippers: Penguins (birds), seals, and sea lions (mammals) have all evolved flipper-like limbs for swimming, despite having different ancestries.
Ant-eating adaptations: Anteaters (mammals), pangolins (mammals), and numbats (marsupials) have independently evolved long snouts, sticky tongues, and strong claws for eating ants and termites.
My own attraction to a bucket approach (in the sense of (1) above) is motivated by a combination of:
(a) reject the demand for commensurability across buckets.
(b) make a bet on plausible deontic constraints e.g. duty to prioritise members of the community of which you are a part.
(c) avoid impractical zig-zagging when best guess assumptions change.
Insofar as I’m more into philosophical pragmatism than foundationalism, I’m more inclined to see a messy collection of reasons like these as philosophically adequate.
I think there are two things to justify here:
The commitment to a GHW bucket, where that commitment involves “we want to allocate roughly X% of our resources to this”.
The particular interventions we fund within the GHW resource bucket.
I think the justification for (1) is going to look very different to the justification for (2).
I’m not sure which one you’re addressing, it sounds like more (2) than (1).
Would you be up for spelling out the problem of “lacks adequate philosophical foundations”?
What criteria need to be satisfied for the foundations to be adequate, to your mind?
Do they e.g. include consequentialism and a strong form of impartiality?
Some examples of the kinds of thing I might share, were there an obvious place to do so:
Bret Taylor and Larry Summers (members of the current OpenAI board) have responded to Helen Toner and Tasha McCauley in The Economist.
The key passages: