Every post, comment, or Wiki edit I authored is hereby licensed under a Creative Commons Attribution 4.0 International License.
Pablo
Karma: 10,784
Why do you keep referring to Ives Parr as “Ives Parr”? I don’t know if that is his real name or a nom de plume, but in either case it is the name Ives Parr has chosen to use, and I think you should respect that choice; enclosing his name in scare quotes seems disrespectful.
There is a list by Sandberg here. (The other items in that post may also be of interest.)
There is now a dedicated FHI website with lists of selected publications and resources about the institute. (Thanks to @Stefan_Schubert for bringing this to my attention.)
Future of Humanity Institute 2005-2024: Final Report
I have taken the liberty of reinstating the images and removing the notice. @Mark Xu, I assume you are okay with this?
the result of a PR-management strategy which seems antithetical to the principles of Effective Altruism to me
This view has been asserted many times before, but to my knowledge, it has never been explicitly defended. Why is it antithetical to the principles of effective altruism to be concerned with the reputational damage certain decisions can cause, when such damage can often severely impact one’s ability to do good?
In a comment explaining his decision to seek funding for the Wytham Abbey project, Owen Cotton-Barratt expresses a similar view. Owen writes that it is “better to let decisions be guided less by what we think looks good, and more by what we think is good.” But, to state the obvious, deciding based on what is good will sometimes require giving a lot of weight to how good the decision will look to others, because those perceptions are among the circumstances affecting the impact of our actions.
There may be a more sophisticated justification for the decision procedure to never, or almost never, allow PR concerns to influence one’s decision-making. Empirically, this doesn’t look true to me, though. For better or worse, we live in a world where PR “scandals” can harm people or movements involved in them to an extreme degree. I think we should take notice of this fact, and act accordingly.
In particular, superforecasters are generally more sceptical of such short AI timelines.
Note that this is true only of some subset of superforecasters. Samotsvety’s forecasters (many of whom are superforecasters) have shorter timelines than both domain experts and general x-risk experts:
P(AGI by 2030)[4] P(AGI by 2050) P(AGI by 2100) P(AGI by this year) = 10% P(AGI by this year) = 50% P(AGI by this year) = 90% mean: 0.31 0.63 0.81 2026 2041 2164 stdev: 0.07 0.11 0.09 1.07 8.99 79.65 50% CI: [0.26, 0.35] [0.55, 0.70] [0.74, 0.87] [2025.3, 2026.7] [2035, 2047] [2110, 2218] 80% CI: [0.21, 0.40] [0.48, 0.77] [0.69, 0.93] [2024.6, 2027.4] [2030, 2053] [2062, 2266] 95% CI[5]: [0.16, 0.45] [0.41, 0.84] [0.62, 0.99] [2023.9, 2028.1] [2024, 2059] [2008, 2320] geomean: 0.30 0.62 0.80 2026.00 2041 2163 geo odds[6]: 0.30 0.63 0.82
I guess most people sympathetic to existential risk reduction think the extinction risk from AI is much higher that those from other risks (as I do). In addition, existential risk as a fraction of extinction risk is arguably way higher for AI than other risks, so the consideration you mentioned will tend to make AI existential risk even more pressing? If so, people may be more interested in either tackling AI risk, or assessing its interactions with other risks.
Yes, this seems right.
As a semi-tangential observation: your comment made me better appreciate an ambiguity in the concept of importance. When I said that this was an important consideration, I meant that it could cause us to significantly revise our estimates of impact. But by ‘important consideration’ one could also mean a consideration that could cause us to significantly alter our priorities.[1] “X-risks to all life v. to humans” may be important in the first sense but not in the second sense.
- ^
Perhaps one could distinguish between ‘axiological importance’ and ‘deontic importance’ to disambiguate these two notions.
- ^
I agree that in the absence of specific examples the criticism is hard to understand. But I would go further and argue that the NB at the beginning is fundamentally misguided and that well-meaning and constructive criticism of EA orgs or people should very rarely be obscured to make it seem less antagonistic.
I would like someone write a post expanding on X-risks to all life v. to humans. Despite the importance of this consideration, it seems to have been almost completely neglected in EA discussion.
If I were to write on this, I’d reframe the issue somewhat differently than the author does in that post. Instead of a dichotomy between two types of risks, one could see it as a gradation of risks that push things back an increasing number of possible great filters. Risks to all life and risks to humans would then be two specific instances of this more general phenomenon.
Thanks for the clarification.
Yes, I agree that we should consider the long-term effects of each intervention when comparing them. I focused on the short-term effects of hastening AI progress because it is those effects that are normally cited as the relevant justification in EA/utilitarian discussions of that intervention. For instance, those are the effects that Bostrom considers in ‘Astronomical waste’. Conceivably, there is a separate argument that appeals to the beneficial long-term effects of AI capability acceleration. I haven’t considered this argument because I haven’t seen many people make it, so I assume that accelerationist types tend to believe that the short-term effects dominate.
I was trying to hint at prima facie plausible ways in which the present generation can increase the value of the long-term future by more than one part in billions, rather than “assume” that this is the case, though of course I never gave anything resembling a rigorous argument.
I do agree that the “washing out” hypothesis is a reasonable default and that one needs a positive reason for expecting our present actions to persist into the long-term. One seemingly plausible mechanism is influencing how a transformative technology unfolds: it seems that the first generation that creates AGI has significantly more influence on how much artificial sentience there is in the universe a trillion years from now than, say, the millionth generation. Do you disagree with this claim?
I’m not sure I understand the point you make in the second paragraph. What would be the predictable long-term effects of hastening the arrival of AGI in the short-term?
I think it remains the case that the value of accelerating AI progress is tiny relative to other apparently available interventions, such as ensuring that AIs are sentient or improving their expected well-being conditional on their being sentient. The case for focusing on how a transformative technology unfolds, rather than on when it unfolds,[1] seems robust to a relatively wide range of technologies and assumptions. Still, this seems worth further investigation.
- ^
Indeed, it seems that when the transformation unfolds is primarily important because of how it unfolds, insofar as the quality of a transformation is partly determined by its timing.
- ^
Recently, an OP grantee spent 5 hours of staff time opening this Chase Platinum Business Checking account which swept into this JPMorgan US Treasury Plus Money Market Fund.
I tried to open a Chase Platinum Business Checking account for a 501(c)(3) that I created recently, but it appears that they are not available for nonprofits. If one selects “Corporation” in the dropdown menu on the left, one is forced to select either “S-Corporation” or “C-Corporation” in the dropdown menu on the right, neither of which, I believe, is appropriate for a nonprofit.
Separately, I successfully opened a Wise Business account, but it has two drawbacks: (1) three months ago, Wise “temporarily” deactivated Business cards; and (2) their interest feature is unavailable for non-US-based customers (like me).
Other options mentioned in the post or comments are also, unfortunately, not suitable for my case: e.g. Mercury is not available for nonprofits, and Monzo is only available for UK-based businesses.
I would be grateful for any recommendations.
Thank you for the ping; I’ll take a look shortly.
Great post. A few months I wrote a private comment that makes a very similar point but frames it somewhat differently; I share it below in case it is of any interest.
Victoria Krakovna usefully defines the outer and inner alignment problems in terms of different “levels of specification”: the outer alignment problem is the problem of aligning the ideal specification (the goals of the designer) with the design specification (the goal implemented in the system), while the inner alignment problem is the problem of aligning this design specification with the revealed specification (the goal the system actually pursues). I think this model could be extended to define a third subcomponent of the alignment problem, next to the inner and outer alignment problems. This would be the problem of moving from what we may call the normative specification (the goals that ought to be pursued) to the ideal specification (though it would be clearer to call the latter “human specification”).
This “third alignment problem” is rarely formulated explicitly, in part because “AI alignment” is ambiguously defined to mean either “getting AI systems to do what we want them to do” and “getting AI systems to do what they ought to do”. But it seems important to distinguish between normative and human specifications, not only because (arguably) “humanity” may fail to pursue the goals it should, but also because the team of humans that succeeds in building the first AGI may not represent the goals of “humanity”. So this should be relevant both to people (like classical and negative utilitarians) with values that deviate from humanity’s in ways that could matter a lot, and to “commonsense moralists” who think we should promote human values but are concerned that AI designers may not pursue these values (because these people may not be representative members of the population, because of self-interest, or because of other reasons).
It’s unclear to me how important this third alignment problem is relative to the inner or outer alignment problems. But it seems important to be aware that it is a separate problem so that one can think about it explicitly and estimate its relative importance.
This deck includes some EAA-related numbers, which may be of interest.
No longer working.
Retaliation is bad.
People seem to be using “retaliation” in two different senses: (1) punishing someone merely in response to their having previously acted against the retaliator’s interests, and (2) defecting against someone who has previously defected in a social interaction analogous to a prisoner’s dilemma, or in a social context in which there is a reasonable expectation of reciprocity. I agree that retaliation is bad in the first sense, but Will appears to be using ‘retaliation’ in the second sense, and I do not agree that retaliation is bad in this sense.
(I haven’t followed this thread closely and I do not have object-level views about the Nonlinear dispute. Sharing just in case it helps clear unnecessary misunderstandings.)
See also Anders’s more personal reflections: