Peter S. Park

Karma: 564

Peter S. Park 26 Sep 2023 16:27 UTC
3 points
0 ∶ 0
on: Inside the Mind of an Aspiring Charity Entrepreneur [Follow Along] #1 - From Layoff to Co-founding in a Breathtaking Two Months
Fantastic writeup on your (and our) steps in founding an evidence-based nonprofit, Harry! Your insights are extremely helpful and broadly applicable, and I am similarly optimistic and excited about our next steps.

The lessons “celebrate successes big and small” and “apply early” seem especially important.

Really looking forward to our continued collaboration on StakeOut.AI, Harry!

Peter S. Park 1 Jun 2023 13:03 UTC
5 points
3 ∶ 3
in reply to: havequick’s comment on: A moral backlash against AI will probably slow down AGI development
Thank you so much for this extremely insightful comment! I strongly agree with all of your points.

“‘AI took my job’ is low-status (‘can’t adapt? skill issue’) to admit seriously thinking about, but even in the dream AGI/ASI-is-aligned scenarios, the catastrophic consequences of AGI/ASI will likely look like ‘AI took my job’ extrapolated to the entire human species…”

My guess: The point at which “AI took my job” changes from low-status to an influential rallying cry is the point when a critical mass of people “wake up” to the fact that AGI is going to take their jobs (in fact, everyone’s) and that this will happen in the near future.

Peter S. Park 1 Jun 2023 2:33 UTC
13 points
6 ∶ 2
on: A moral backlash against AI will probably slow down AGI development
Thank you so much, Geoffrey, for this compelling argument! I completely agree that a moral backlash against AI is very plausible (and in my estimation, imminent), especially from people whose career paths are or will be indefinitely automated away by AI, and who will not thrive in the new niche of ‘startup founder who only hires AGI employees.’

OpenAI’s mission is to create “highly autonomous systems that outperform humans at most economically valuable work” (their definition of AGI).

I cannot overstate how broadly unpopular this mission is. In fact, in my experience, whenever I told someone occupationally far-removed from AI about OpenAI’s mission, they immediately diagnosed this mission as dystopian. They were also very open-minded about the plausibility of catastrophic and existential dangers in such an AGI-led future, with very few exceptions.

The only bottleneck is that most people currently don’t believe that AGI is imminent. This is, of course, starting to change: for example, with the recent CAIS statement signed by leading AI researchers and notable figures.

We should tirelessly prepare for a future in which AGI leaves large numbers of people jobless, purposeless, and angry. I don’t know all the answers to how we should prepare for this dystopia, but I’m confident that we should prioritize the following two things:

(1) Truthful, frequent, and high-quality communication to angry people. e.g., communication about the true cause of their hardships—AGI companies—so that they don’t blame a scapegoat.

(2) Make preparations for the prevention and resolution of the first (near-catastrophic or catastrophic) AI disaster, as well as for proposing and implementing effective AI-disaster-prevention policies after the first AI disaster.

Peter S. Park 1 Jun 2023 2:04 UTC
3 points
1 ∶ 0
on: Statement on AI Extinction—Signed by AGI Labs, Top Academics, and Many Other Notable Figures
Truly brilliant coalition-building by CAIS and collaborators. It is likely that the world has become a much safer place as a result. Congratulations!

Peter S. Park 2 May 2023 17:26 UTC
4 points
1 ∶ 0
in reply to: richard_ngo’s comment on: AGI safety career advice
Richard, I hope you turn out to be correct that public support for AI governance ideas will become less of a bottleneck as more powerful AI systems are released!

But I think it is plausible that we should not leave this to chance. Several of the governance ideas you have listed as promising (e.g., global GPU tracking, data center monitoring) are probably infeasible at the moment, to say the least. It is plausible that these ideas will only become globally implementable once a critical mass of people around the world become highly aware of and concerned about AGI dangers.

This means that timing may be an issue. Will the most detrimental of the AGI dangers manifest before meaningful preventative measures are implemented globally? It is plausible that before the necessary critical mass of public support builds up, a catastrophic or even existential outcome may already have occurred. It would then be too late.

The plausibility of this scenario is why I agree with Akash that identifying and raising public awareness of AGI’s dangers is an underrated approach.

Peter S. Park 2 May 2023 16:14 UTC
4 points
4 ∶ 4
on: AGI safety career advice
Thank you so much for your insightful and detailed list of ideas for AGI safety careers, Richard! I really appreciate your excellent post.

I would propose explicitly grouping some of your ideas and additional ones under a third category: “identifying and raising public awareness of AGI’s dangers.” In fact, I think this category may plausibly contain some of the most impactful ideas for reducing catastrophic and existential risks, given that alignment seems potentially difficult to achieve in a reasonable period of time (if ever) and the implementation of governance ideas is bottlenecked by public support.

For a similar argument that I found particularly compelling, please check out Greg Colbourn’s recent post: https://forum.effectivealtruism.org/posts/8YXFaM9yHbhiJTPqp/agi-rising-why-we-are-in-a-new-era-of-acute-risk-and

Peter S. Park 20 Dec 2022 5:33 UTC
6 points
1 ∶ 0
on: Why mechanistic interpretability does not and cannot contribute to long-term AGI safety (from messages with a friend)
Brilliant and compelling writeup, Remmelt! Thank you so much for sharing it. (And thank you so much for your kind words about my post! I really appreciate it.)
I strongly agree with you that mechanistic interpretability is very unlikely to contribute to long-term AI safety. To put it bluntly, the fact that so many talented and well-meaning people sink their time into this unpromising research direction is unfortunate.
I think we AI safety researchers should be more open to new ideas and approaches, rather than getting stuck in the same old research directions that we know are unlikely to meaningfully help. The post “What an actually pessimistic containment strategy looks like” has potentially good ideas on this front.

Peter S. Park 10 Dec 2022 6:59 UTC
4 points
0 ∶ 0
on: Lessons Learned from EA Hackathon following EAGxBerkeley
Thank you so much for organizing this, Nicole! I attended and it was extremely fun.

Peter S. Park 4 Dec 2022 17:06 UTC
2 points
0 ∶ 0
in reply to: Jay Bailey’s comment on: AI can exploit safety plans posted on the Internet
Thank you so much for the clarification, Jay! It is extremely fair and valuable.
I don’t really understand how this is supposed to be an update for those who disagreed with you. Could you elaborate on why you think this information would change people’s minds?
The underlying question is: does the increase in the amount of AI safety plans resulting from coordinating on the Internet outweigh the decrease in secrecy value of the plans in EV? If the former effect is larger, then we should continue the status-quo strategy. If the latter effect is larger, then we should consider keeping safety plans secret (especially those whose value lies primarily in secrecy, such as safety plans relevant to monitoring).
The disagreeing commenters generally argued that the former effect is larger, and therefore we should continue the status-quo strategy. This is likely because their estimate of the latter effect was quite small and perhaps far-into-the-future.
I think ChatGPT provides evidence that the latter should be a larger concern than many people’s prior. Even current-scale models are capable of nontrivial analysis about how specific safety plans can be exploited, and even how specific alignment researchers’ idiosyncrasies can be exploited for deceptive misalignment.
For this to be a threat, we would need an AGI that was

- Misaligned
- Capable enough to do significant damage if it had access to our safety plans
- Not capable enough to do a similar amount of damage without access to our safety plans

I see the line between 2 and 3 to be very narrow. I expect almost any misaligned AI capable of doing significant damage using our plans to also be capable of doing significant damage without needing them.
I am uncertain about whether the line between 2 and 3 will be narrow. I think the argument of the line between 2 and 3 being narrow often assumes fast takeoff, but I think there is a strong empirical case that takeoff will be slow and constrained by scaling, which suggests the line between 2 and 3 might be larger than one might think. But I think this is a scientific question that we should continue to probe and reduce our uncertainty about!

Peter S. Park 20 Nov 2022 21:56 UTC
6 points
5 ∶ 0
in reply to: Evan R. Murphy’s comment on: What happened to the “Women and Effective Altruism” post?
No, “Women and Effective Altruism” by Keerthana preceded “Brainstorming ways to make EA safer and more inclusive” by Richard.

Peter S. Park 17 Nov 2022 1:28 UTC
26 points
18 ∶ 2
on: Stop Overreacting re: FTX / SBF
I agree that in many cases, when bad-faith critics say “EA should not do X” and we stop doing X, the world would actually become worse off.
But making SBF the face of the EA movement was a really bad decision. Especially given that he was unilaterally gambling with the whole EA movement’s credibility.
There are robust lessons to be learned in this saga, which will allow us EAs to course-correct and prevent future catastrophic outcomes (to our movement and in general).
What links here?
- Ubuntu's comment on Time Article Discussion—“Effective Altruist Leaders Were Repeatedly Warned About Sam Bankman-Fried Years Before FTX Collapsed” by Nathan Young (18 Mar 2023 11:30 UTC; 4 points)

Peter S. Park 16 Nov 2022 20:34 UTC
17 points
5 ∶ 1
on: Some important questions for the EA Leadership
I think we EAs need to increasingly prioritize speaking up about concerns like the ones Habryka mentioned.

Even when positive in-group feelings, the fear of ostracism, and uncertainty/risk aversion internally influences one to not bring up these concerns, we should fight back against this urge because the concerns, if true, will likely grow larger and larger until they blow up.

There is very high EV in course correction before the catastrophic failure point.

Peter S. Park 14 Nov 2022 3:04 UTC
34 points
18 ∶ 2
on: Wrong lessons from the FTX catastrophe
Strongly agree with all of these points.
On point 2: The EA movement urgently needs more earners-to-give, especially now. One lesson that I think is correct, however, is that we should be wary of making any one billionaire donor the face of the EA movement. The downside risk—a loss of credibility for the whole movement due to unknown information about the billionaire donor—is generally too high.

Peter S. Park 11 Nov 2022 22:47 UTC
2 points
1 ∶ 0
in reply to: SaraAzubuike’s comment on: Another FTX post: suggestions for change
Mostly it was about Point 3. I think an unconditional norm of only accepting anonymous donations above a certain threshold would be too blunt.
I think a version of Point 3 I would agree with is to have high-contributing donor names not be publicized as a norm (with some possible exceptions). I think this captures most of the benefits of an anonymous donation, and most potential donors who might not be willing to make an anonymous donation would be willing to make a discreet, non-publicized donation.

Peter S. Park 11 Nov 2022 20:15 UTC
11 points
4 ∶ 0
on: Another FTX post: suggestions for change
I strongly agree with the spirit of the reforms being suggested here (although I might have some different opinions on how to implement it). We need large-scale reforms of the EA community’s social norms to prevent future risks to movement-wide credibility.
1. Strongly agree. The fact that net upvotes are the only concrete metric by which EA forum posts and LessWrong forum posts are judged has indeed been suboptimal for one of EA’s main goals: to reflect on and adapt our previous beliefs based on new evidence. Reforms designed to increase the engagement of controversial posts would be very helpful for our pursuit of this goal. (Disclaimer: Most of my EA forum posts would rank highly on the “controversial” scale, in that many people upvote and many people downvote them, and the top comment is usually critical and has a lot of net upvotes. I think that we EAs need to increasingly prioritize both posting and engaging with controversial arguments that run contrary to status-quo beliefs, even if it’s hard! This is especially true for LessWrong, which arguably doubles as a scientific venue for AI safety research in addition to an EA-adjacent discussion forum.)
2. Agree, although I think EAs should be more willing to write and engage with controversial arguments non-anonymously as well.
3. Strongly agree in spirit. While a norm of unconditionally refusing non-anonymous donations above a certain threshold might be too blunt, I do think we need to have better risk-management about tying our EA movement’s credibility to a single charismatic billionaire, or a single charismatic individual in general. Given how important our work is, we probably need better risk-management practices in general. (And we EAs already care earnestly about this! I do think this is a question not of earnest desire but of optimal implementation.) I also think that many billionaires would actually prefer to donate anonymously or less publicly, because they agree with the bulk of but not all of EA’s principles. Realistically, leaving room for case-by-case decision-making seems helpful.

Peter S. Park 5 Nov 2022 21:52 UTC
3 points
1 ∶ 1
on: Instead of technical research, more people should focus on buying time
Thank you so much for your excellent post on the strategy of buying time, Thomas, Akash, and Olivia! I strongly agree that this strategy is necessary, neglected, and tractable.
For practical ideas of how to achieve this (and a productive debate in the comments section of the risks from low-quality outreach efforts), please see my related earlier forum post: https://forum.effectivealtruism.org/posts/juhMehg89FrLX9pTj/a-grand-strategy-to-recruit-ai-capabilities-researchers-into

Peter S. Park 4 Nov 2022 23:49 UTC
1 point
0 ∶ 0
in reply to: Charles He’s comment on: Why do we post our AI safety plans on the Internet?
Thanks so much for your detailed response, Charles! I really appreciate your honest feedback.
It seems likely that our cruxes are the following. I think that (a) we probably cannot predict the precise moment the AGI becomes agentic and/or dangerous, (b) we probably won’t have a strong credence that a specific alignment plan will succeed (say, to the degree of solving ELK or interpretability in the short time we have), and (c) AGI takeoff will be slow enough that secrecy can be a key difference-maker in whether we die or not.
So, I expect we will have alignment plan numbers 1, 2, 3, and so on. We will try alignment plan 1, but it will probably not succeed (and hopefully we can see signs of it not succeeding early enough that we shut it down and try alignment plan 2). If we can safely empirically iterate, we will find an alignment plan N that works.
This is risky and we could very well die (although I think the probability is not unconditionally 100%). This is why I think not building AGI is by far the best strategy. (Corollary: I place a lot of comparative optimism on AI governance and coordination efforts.) The above discussion is conditional on trying to build an aligned AGI.
Given these cruxes, I think it is plausible that a Manhattan-Project-esque shift in research norms can help achieve secrecy value in our generated AI safety plans, but also keep ease-of-research high.

Peter S. Park 4 Nov 2022 23:13 UTC
1 point
0 ∶ 0
in reply to: Charles He’s comment on: Why do we post our AI safety plans on the Internet?
This would definitely be a useful tag for a hostile AI.
Moreover, the fact that everyone can access tagged posts right now means that there are many ways by which tagged content may still be made accessible by the AGI, even in the event that the AGI corporation cooperates with us.
Secrecy is only likely to be kept if the information is known by a small set of people. (e.g., Conjecture’s infohazard document discusses this quite well)

Peter S. Park 3 Nov 2022 21:10 UTC
1 point
0 ∶ 0
in reply to: Michael Einhorn’s comment on: Why do we post our AI safety plans on the Internet?
Thanks so much, Michael, for your detailed and honest feedback! I really appreciate your time.
I agree that both threat models are real. AI safety research can lose its value when not kept secret, and humanity’s catastrophic and extinction risk increases if AI capabilities advances faster than valuable AI safety research.
Regarding your point that the Internet has significantly helped accelerate AI safety research, I would say two things. First, what matters is the Internet’s effect on valuable AI safety research. If much of the value in AI safety plans requires them being kept secret from the Internet (e.g., one’s plan in a rock-paper-scissors-type interaction), then the current Internet-forum-based research norms may not be increasing the rate of value generation by that much. In fact, it may plausibly be decreasing the rate of value generation, in light of the discussion in my post. So, we should vigorously investigate how much of the value in AI safety plans in fact requires secrecy.
Second, the fact that AI safety researchers (or more generally, people in the economy) are extensively using the Internet does not falsify the claim that the Internet may be close to net-zero or net-negative for community innovation. In this claim, the Internet is great at enticing people to use it and spend money on it, but it is not great at improving real innovation as measured by productivity gains. So, it has a redistributive rather than productive effect on how people spend their time and money. So, we should expect to see people (e.g., AI safety researchers) extensively use the Internet even if the Internet has not increased the innovation rate of real productivity gains.
What matters is the comparison with the counterfactual: if there were an extensive, possibly expensive, and Manhattan-Project-esque change in research norms for the whole community, could ease-of-research be largely kept the same even with secrecy gains? I think the answer may be plausibly “yes.”
How do we estimate the real effect of the Internet on the generation of valuable AI safety research? First, we would need to predict the value of AI safety research, particularly how its value is affected by its secrecy. This effort would be aided by game theory, past empirical evidence of real-world adversarial interactions, and the resolution of scientific debates about what AGI training would look like in the future.
Second, we would need to estimate how much the Internet affects the generation of this value. This effort would be aided by progress studies and other relevant fields of economics and history.
Edit: Writing to add a link for the case of why the Internet may be overrated for the purposes of real community innovation: https://www.bloomberg.com/news/articles/2013-06-20/what-the-web-didnt-deliver-high-economic-growth

Peter S. Park 31 Oct 2022 18:57 UTC
1 point
0 ∶ 0
in reply to: alex lawsen’s comment on: Why do we post our AI safety plans on the Internet?
I didn’t mean to imply that the cost will be trivial. The cost will either be a significant reduction in communication between AI safety researchers who are far apart (which I agree harms our x-risk reduction efforts), or a resource cost paid by a collaboration of AI safety researchers and EAs with a variety of skillsets to create the infrastructure and institutions needed for secure AI safety research norms. The latter is what I had in mind, and it probably cannot be a decentralized effort like “don’t use Google Docs or gmail.”