Jacy

Ar­tifi­cial In­tel­li­gence, Mo­ral­ity, and Sen­tience (AIMS) Sur­vey: 2021

• JacyJul 1, 2022, 4:59 AM

  1 point

  0 ∶ 0

  in reply to: Ben_West🔸’s comment on: The Fu­ture Might Not Be So Great.

  Thanks! Fixed, I think.

• JacyJul 1, 2022, 12:10 AM

  9 points

  0 ∶ 0

  in reply to: Mau’s comment on: The Fu­ture Might Not Be So Great.

  It’s great to know where your specific weights differ! I agree that each of the arguments you put forth are important. Some specifics:

  • I agree that differences in the future (especially the weird possibilities like digital minds and acausal trade) is a big reason to discount historical evidence. Also, by these lights, some historical evidence (e.g., relations across huge gulfs of understanding and ability like from humans to insects) seems a lot more important than others (e.g., the fact that animal muscle and fat happens to be an evolutionarily advantageous food source).

  • I’m not sure if I’d agree that historical harms have occurred largely through divergence; there are many historical counterfactuals that could have prevented many harms: the nonexistence of humans, an expansion of the moral circle, better cooperation, discovery of a moral reality, etc.. In many cases, a positive leap in any of these would have prevented the atrocity. What makes divergence more important? I would make the case based on something like “maximum value impact from one standard deviation change” or “number of cases where harm seemed likely but this factor prevented it.” You could write an EA Forum post going into more detail on that. I would be especially excited for you to go through specific historical events and do some reading to estimate the role of (small changes in) each of these forces.

  • As I mention in the post, reasons to put negative weight on DMPS include the vulnerability of digital minds to intrusion, copying, etc., the likelihood of their instrumental usefulness in various interstellar projects, and the possibility of many nested minds who may be ignored or neglected.

  • I agree moral trade is an important mechanism of reasoned cooperation.

  I’m really glad you put your own numbers in the spreadsheet! That’s super useful. The ease of flipping the estimates from negative to positive and positive to negative is one reason I only make the conclusion “not highly positive” or “close to zero” and not going with the mean estimate from myself and others (which would probably be best described as moderately negative, e.g., the average at an EA meetup where I presented this work was around −10).

  I think your analysis is on the right track to getting us better answers to these crucial questions :)

• JacyJun 30, 2022, 3:20 PM

  1 point

  0 ∶ 0

  in reply to: Oliver Sourbut’s comment on: The Fu­ture Might Not Be So Great.

  Whoops! Thanks!

• JacyJun 30, 2022, 2:35 PM

  8 points

  0 ∶ 0

  on: Fa­nat­i­cal EAs should sup­port very weird projects

  This is super interesting. Thanks for writing it. Do you think you’re conflating several analytically distinct phenomena when you say (i) “Fanaticism is the idea that we should base our decisions on all of the possible outcomes of our actions no matter how unlikely they are … base our decisions on all of the possible outcomes of our actions no matter how unlikely they are EA fanatics take a roughly maximize expected utility approach” and (ii) “Fanaticism is unreasonable”?
  
  For (i), I mainly have in mind two approaches “fanatics” could be defined by: (ia) “do a quick back-of-the-envelope calculation of expected utility and form beliefs based solely on its output,” and (ib) “do what you actually think maximizes expected utility, no matter whether that’s based on a spreadsheet, heuristic, intuition, etc.” I think (ia) isn’t something basically anyone would defend, while (ib) is something I and many others would (and it’s how I think “fanaticism” tends to be used). And for (ib), we need to account for heuristics like, (f) quick BOTE calculations tend to overestimate the expected utility of low probabilities of high impact, and (g) extremely large and extremely small numbers should be sandboxed (e.g., capped in the influence they can have on the conclusion). This is a (large) downside of these “very weird projects,” and I think it makes the “should support” case a lot weaker.
  
  For (ii), I mainly have in mind three claims about fanaticism: (iia) “Fanaticism is unintuitive,” (iib) “Fanaticism is absurd (a la reductio ad absurdum,” and (iic) “Fanaticism breaks some utility axioms.” These each have different evidence . For example, (iia) might not really matter if we don’t think our intuitions—which have been trained through evolution and life experience—are reliable for such unusual questions like maximizing long-run aggregate utility.
  
  Did you have some of these in mind? Or maybe other operationalizations?

• JacyJun 30, 2022, 2:05 PM

  12 points

  0 ∶ 0

  on: Jacy’s Shortform

  Brief Thoughts on the Prioritization of Quality Risks

  This is a brief shortform post to accompany “The Future Might Not Be So Great.” These are just some scattered thoughts on the prioritization of quality risks not quite relevant enough to go in the post itself. Thanks to those who gave feedback on the draft of that post, particularly on this section.

  People ask me to predict the future, when all I want to do is prevent it. Better yet, build it. Predicting the future is much too easy, anyway. You look at the people around you, the street you stand on, the visible air you breathe, and predict more of the same. To hell with more. I want better. ⸻ Ray Bradbury (1979)

  I present a more detailed argument for the prioritization of quality risks (particularly moral circle expansion) over extinction risk reduction (particularly through certain sorts of AI research) in Anthis (2018), but here I will briefly note some thoughts on importance, tractability, and neglectedness. Two related EA Forum posts are “Cause Prioritization for Downside-Focused Value Systems” (Gloor 2018) and “Reducing Long-Term Risks from Malevolent Actors” (Althaus and Baumann 2020). Additionally, at this early stage of the longtermist movement, the top priorities for population and quality risk may largely intersect. Both issues suggest foundational research of topics such as the nature of AI control and likely trajectories of the long-term future, community-building of thoughtful do-gooders, and field-building of institutional infrastructure to use for steering the long-term future.

  Importance

  One important application of the EV of human expansion is to the “importance” of population and quality risks. Importance can be operationalized as the good done if the entire cause succeeded in solving its corresponding problem, such as the good done by eliminating or substantially reducing extinction risk, which is effectively zero if the EV of human expansion is zero and effectively negative if the EV of human expansion is negative.

  The importance of quality risk reduction is clearer, in the sense that the difference in quality between possible futures is clearer than the difference in extinction and non-extinction, and larger, in the sense that while population risk entails only the range of zero-to-positive difference between human extinction and non-extinction (or population risk between zero population and some positive number of individuals) across quality risk entails the difference between the best quality humans could engender and the worst, across all possible population sizes. This is arguably a weakness of the framework because we could categorize the quality risk cause area as smaller in importance (say, an increase of 1 trillion utils, i.e., units of goodness), and it would tend to become more tractable as we narrow the category.

  Tractability

  The tractability difference between population and quality risk seems the least clear of the three criteria. My general approach is thinking through the most likely “theories of change” or paths to impact and assessing them step-by-step. For example, one commonly discussed extinction risk reduction path to impact is “agent foundations,” building mathematical frameworks and formally proving claims about the behavior of intelligent agents, which would then allow us to build advanced AI systems more likely to do what we tell them to do, and then using these frameworks to build AGI or persuading the builders of AGI to use them. Quality-risk-focused AI safety strategies may be more focused on the outer alignment problem, ensuring that an AI’s objective is aligned with the right values, rather than just the inner alignment problem, ensuring that all actions of the AI are aligned with the objective.^[1] Also, we can influence quality by steering the “direction” or “speed” of the long-term future, approaches with potentially very different impact, hinging on factors such as the distribution of likely futures across value and likelihood (e.g., Anthis 2018c; Anthis and Paez 2021).

  One argument that I often hear on the tractability of trajectory changes is that changes need to “stick” or “persist” over long periods. It is true that there needs to be a persistent change in the expected value (i.e., the random variable or time series regime of value in the future), but I frequently hear the claim that there needs to be a persistent change in the realization of that value. For example, if we successfully broker a peace deal between great powers, neither the peace deal itself nor any other particular change in the world has to persist in order for this to have high long-term impact. The series of values itself can have arbitrarily large variance, such as it being very likely that the peace deal is broken within a decade.

  For a sort of change to be intractable, it needs to not just lack persistence, but to rubber band (i.e., create opposite-sign effects) back to its counterfactual. For example, if brokering a peace deal causes an equal and opposite reaction of anti-peace efforts, then that trajectory change is intractable. Moreover, we should not only consider rubber banding but dominoing (i.e., create same-sign effects), perhaps because of how this peace deal inspires other great powers to follow suit even if this particular deal is broken. There is much of this potential energy in the world waiting to be unlocked by thoughtful actors.

  The tractability of trajectory change has been the subject of research at Sentience Institute, including our historical case studies and “Harris’ (2019)” How Tractable Is Changing the Course of History?”

  Neglectedness

  The neglectedness difference between population and quality risk seems the most clear. There are far more EAs and longtermists working explicitly on population risks than on quality risks (i.e., risks to the moral value of individuals in the long-term future). Two nuances for this claim are first that it may not be true for other relevant comparisons: For example, many people in the world are trying to change social institutions, such as different sides of the political spectrum trying to pull public opinion towards their end of the spectrum. This group seems much larger than people focused explicitly on extinction risks, and there are many other relevant reference classes. Second, it is not entirely clear whether extinction risk reduction and quality risk reduction face higher or lower returns to being less neglected (i.e., more crowded). It may be that so few people are focused on quality risks that marginal returns are actually lower than they would be if there were more people working on them (i.e., increasing returns).

  ---

  1. ↩︎

     In my opinion, there are many different values involved in developing and deploying an AI system, so the distinction between inner and outer alignment is rarely precise in practice. Much of identifying and aligning with “good” or “correct” values can be described as outer alignment. In general, I think of AI value alignment as a long series of mechanisms from the causal factors that create human values (which themselves can be thought of as objective functions) to a tangled web of objectives in each human brain (e.g., values, desires, preferences) to a tangled web of social objectives aggregated across humans (e.g., voting, debates, parliaments, marketplaces) to a tangled web of objectives communicated from humans to machines (e.g., material values in game-playing AI, training data, training labels, architectures) to a tangled web of emergent objectives in the machines (e.g., parametric architectures in the neural net, (smoothed) sets of possible actions in domain, (smoothed) sets of possible actions out of domain) and finally to the machine actions (i.e., what it actually does in the world). We can reasonably refer to the alignment of any of these objects with any of the other objects in this long, tangled continuum of values. Two examples of outer alignment work that I have in mind here are Askell et al. (2021) “A General Language Assistant as a Laboratory for Alignment” and Hobbhan et al. (2022) “Reflection Mechanisms as an Alignment Target: A Survey.”

  What links here?

  • The Fu­ture Might Not Be So Great by Jacy (Jun 30, 2022, 1:01 PM; 145 points)
  • Longter­mism and An­i­mal Farm­ing Trajectories by MichaelDello (Dec 27, 2022, 12:58 AM; 51 points)

Jacy’s Quick takes

The Fu­ture Might Not Be So Great

• JacyJun 25, 2022, 4:28 PM

  2 points

  0 ∶ 0

  in reply to: quinn’s comment on: quinn’s Shortform

  Jamie Harris at Sentience Institute authored a report on “Social Movement Lessons From the US Anti-Abortion Movement” that may be of interest.

• JacyJun 16, 2022, 11:28 AM

  8 points

  0 ∶ 0

  in reply to: Black Box’s comment on: Steer­ing AI to care for an­i­mals, and soon

  That’s right that we don’t have any ongoing projects exclusively on the impact of AI on nonhuman biological animals, though much of our research includes that, especially the outer alignment idea of ensuring an AGI or superintelligence accounts for the interests about all sentient beings, including wild and domestic nonhuman biological animals. We also have several empirical projects where we collect data on both moral concern for animals and for AI, such as on perspective-taking, predictors of moral concern, and our recently conducted US nationally representative survey on Artificial Intelligence, Morality, and Sentience (AIMS).
  
  For various reasons discussed in those nonhumans and the long-term future posts and in essays like “Advantages of Artificial Intelligences, Uploads, and Digital Minds” (Sotala 2012), biological nonhuman animals seem less likely to exist in very large numbers in the long-term future than animal-like digital minds. That doesn’t mean we shouldn’t work on the impact of AI on those biological nonhuman animals, but it has made us prioritize laying groundwork on the nature of moral concern and the possibility space of future sentience. I can say that we have a lot of researcher applicants propose agendas focused more directly on AI and biological nonhuman animals, and we’re in principle very open to it. There are far more promising research projects in this space than we can fund at the moment. However, I don’t think Sentience Institute’s comparative advantage is working directly on research projects like CETI or Interspecies Internet that wade through the detail of animal ethology or neuroscience using machine learning, though I’d love to see a blog-depth analysis of the short-term and long-term potential impacts of such projects, especially if there are more targeted interventions (e.g., translating farmed animal vocalizations) that could be high-leverage for EA.

• JacyJun 14, 2022, 11:12 AM

  54 points

  0 ∶ 0

  on: Steer­ing AI to care for an­i­mals, and soon

  Good points! This is exactly the sort of work we do at Sentience Institute on moral circle expansion (mostly for farmed animals from 2016 to 2020, but since late 2020, most of our work has been directly on AI—and of course the intersections), and it has been my priority since 2014. Also, Peter Singer and Yip Fai Tse are working on “AI Ethics: The Case for Including Animals”; there are a number of EA Forum posts on nonhumans and the long-term future; and the harms of AI and “smart farming” for farmed animals is a common topic, such as this recent article that I was quoted in. My sense from talking to many people in this area is that there is substantial room for more funding; we’ve gotten some generous support from EA megafunders and individuals, but we also consistently get dozens of highly qualified applicants whom we have to reject every hiring round, including people with good ideas for new projects.

Apology

• JacyFeb 27, 2019, 2:07 AM

  18 points

  0 ∶ 0

  in reply to: Jonas V’s comment on: After one year of ap­ply­ing for EA jobs: It is re­ally, re­ally hard to get hired by an EA organisation

  Same perspective here! Thank you for sharing.

• JacyDec 30, 2018, 8:36 PM

  5 points

  1 ∶ 0

  in reply to: JanB’s comment on: The ex­pected value of ex­tinc­tion risk re­duc­tion is positive

  Oh, sorry, I was thinking of the arguments in my post, not (only) those in your post. I should have been more precise in my wording.

• JacyDec 27, 2018, 12:22 AM

  15 points

  1 ∶ 0

  in reply to: JanB’s comment on: The ex­pected value of ex­tinc­tion risk re­duc­tion is positive

  Thank you for the reply, Jan, especially noting those additional arguments. I worry that your article neglects them in favor of less important/​controversial questions on this topic. I see many EAs taking the “very unlikely that [human descendants] would see value exactly where we see disvalue” argument (I’d call this the ‘will argument,’ that the future might be dominated by human-descendant will and there is much more will to create happiness than suffering, especially in terms of the likelihood of hedonium over dolorium) and using that to justify a very heavy focus on reducing extinction risk, without exploration of those many other arguments. I worry that much of the Oxford/​SF-based EA community has committed hard to reducing extinction risk without exploring those other arguments.

  It’d be great if at some point you could write up discussion of those other arguments, since I think that’s where the thrust of the disagreement is between people who think the far future is highly positive, close to zero, and highly negative. Though unfortunately, it always ends up coming down to highly intuitive judgment calls on these macro-socio-technological questions. As I mentioned in that post, my guess is that long-term empirical study like the research in The Age of Em or done at Sentience Institute is our best way of improving those highly intuitive judgment calls and finally reaching agreement on the topic.

  What links here?

  • The Fu­ture Might Not Be So Great by Jacy (Jun 30, 2022, 1:01 PM; 145 points)

• JacyDec 16, 2018, 11:19 PM

  25 points

  1 ∶ 0

  on: The ex­pected value of ex­tinc­tion risk re­duc­tion is positive

  Thanks for posting on this important topic. You might be interested in this EA Forum post where I outlined many arguments against your conclusion, the expected value of extinction risk reduction being (highly) positive.

  I do think your “very unlikely that [human descendants] would see value exactly where we see disvalue” argument is a viable one, but I think it’s just one of many considerations, and my current impression of the evidence is that it’s outweighed.

  Also FYI the link in your article to “moral circle expansion” is dead. We work on that approach at Sentience Institute if you’re interested.

• JacyDec 9, 2018, 3:43 PM

  15 points

  0 ∶ 0

  on: Why I’m fo­cus­ing on in­ver­te­brate sentience

  I remain skeptical of how much this type of research will influence EA-minded decisions, e.g. how many people would switch donations from farmed animal welfare campaigns to humane insecticide campaigns if they increased their estimate of insect sentience by 50%? But I still think the EA community should be allocating substantially more resources to it than they are now, and you seem to be approaching it in a smart way, so I hope you get funding!

  I’m especially excited about the impact of this research on general concern for invertebrate sentience (e.g. establishing norms that there are at least some smart humans are actively working on insect welfare policy) and on helping humans better consider artificial sentience when important tech policy decisions are made (e.g. on AI ethics).

• JacySep 19, 2018, 8:00 AM

  9 points

  0 ∶ 0

  on: 2018 list of half-baked vol­un­teer re­search ideas

  [1] Cochrane mass media health articles (and similar):

  • Targeted mass media interventions promoting healthy behaviours to reduce risk of non-communicable diseases in adult, ethnic minorities

  • Mass media interventions for smoking cessation in adults

  • Mass media interventions for preventing smoking in young people.

  • Mass media interventions for promoting HIV testing

  • Smoking cessation media campaigns and their effectiveness among socioeconomically advantaged and disadvantaged populations

  • Population tobacco control interventions and their effects on social inequalities in smoking: systematic review

  • Are physical activity interventions equally effective in adolescents of low and high socioeconomic status (SES): results from the European Teenage project

  • The effectiveness of nutrition interventions on dietary outcomes by relative social disadvantage: a systematic review

  • Use of folic acid supplements, particularly by low-income and young women: a series of systematic reviews to inform public health policy in the UK

  • Use of mass media campaigns to change health behaviour

  • The role of the media in promoting and reducing tobacco use

  • Getting to the Truth: Evaluating National Tobacco Countermarketing Campaigns

  • Effect of televised, tobacco company-funded smoking prevention advertising on youth smoking-related beliefs, intentions, and behavior

  • Do mass media campaigns improve physical activity? a systematic review and meta-analysis

2018 list of half-baked vol­un­teer re­search ideas

• JacySep 15, 2018, 9:37 AM

  7 points

  0 ∶ 0

  in reply to: Ben Pace’s comment on: Which piece got you more in­volved in EA?

  I can’t think of anything that isn’t available in a better form now, but it might be interesting to read for historical perspective, such as what it looks like to have key EA ideas half-formed. This post on career advice is a classic. Or this post on promoting Buddhism as diluted utilitarianism, which is similar to the reasoning a lot of utilitarians had for building/​promoting EA.