I am Issa Rice. https://issarice.com/
riceissa(Issa Rice)
Karma: 1,070
[Question] Why “cause area” as the unit of analysis?
In the April 2020 payout report, Oliver Habryka wrote:
I’ve also decided to reduce my time investment in the Long-Term Future Fund since I’ve become less excited about the value that the fund can provide at the margin (for a variety of reasons, which I also hope to have time to expand on at some point).
I’m curious to hear more about this (either from Oliver or any of the other fund managers).
It seems to me that this post has introduced a new definition of cause X that is weaker (i.e. easier to satisfy) than the one used by CEA.
This post defines cause X as:
The concept behind a “cause X” is that there could be a cause neglected by the EA community but that is as important, or more important, to work on than the four currently established EA cause areas.
But from Will MacAskill’s talk:
What are the sorts of major moral problems that in several hundred years we’ll look back and think, “Wow, we were barbarians!”? What are the major issues that we haven’t even conceptualized today?
I will refer to this as Cause X.
See also the first paragraph of Emanuele Ascani’s answer here.
From the “New causes one could consider” list in this post, I think only Invertebrates and Moral circle expansion would qualify as a potential cause X under CEA’s definition (the others already have researchers/organizations working on them full-time, or wouldn’t sound crazy to the average person).
I think it would be good to have a separate term specifically for the cause areas that seem especially crazy or unconceptualized, since searching for causes in this stricter class likely requires different strategies, more open-mindedness, etc.
Related: Guarded definition.
Back in July, you held an in-person Q&A at REACH and said “There are a bunch of things about AI alignment which I think are pretty important but which aren’t written up online very well. One thing I hope to do at this Q&A is try saying these things to people and see whether people think they make sense.” Could you say more about what these important things are, and what was discussed at the Q&A?
In November 2018 you said “we want to hire as many people as engineers as possible; this would be dozens if we could, but it’s hard to hire, so we’ll more likely end up hiring more like ten over the next year”. As far as I can tell, MIRI has hired 2 engineers (Edward Kmett and James Payor) since you wrote that comment. Can you comment on the discrepancy? Did hiring turn out to be much more difficult than expected? Are there not enough good engineers looking to be hired? Are there a bunch of engineers who aren’t on the team page/haven’t been announced yet?
A trend I’ve noticed in the AI safety independent research grants for the past two rounds (April and August) is that most of the grantees have little to no online presence as far as I know (they could be using pseudonyms I am unaware of); I believe Alex Turner and David Manheim are the only exceptions. However, when I think about “who am I most excited to give individual research grants to, if I had that kind of money?”, the names I come up with are people who leave interesting comments and posts on LessWrong about AI safety. (This isn’t surprising because I mostly interact with the AI safety community publicly online, so I don’t have much access to private info.) To give an idea of the kind of people I am thinking of, I would name John Wentworth, Steve Byrnes, Ofer G., Morgan Sinclaire, and Evan Hubinger as examples.
This has me wondering what’s going on. Some possibilities I can think of:
the people who contribute on LW aren’t applying for grants
the private people are higher quality than the online people
the private people have more credentials than the online people (e.g. Hertz Fellowship, math contests experience)
fund managers are more receptive offline than online and it’s easier to network offline
fund managers don’t follow online discussions closely
I would appreciate if the fund managers could weigh in on this so I have a better sense of why my own thinking seems to diverge so much from the actual grant recommendations.
I read the paper (skipping almost all the math) and Philip Trammell’s blog post. I’m not sure I understood the paper, and in any case I’m pretty confused about the topic of how growth influences x-risk, so I want to ask you a bunch of questions:
-
Why do the time axes in many of the graphs span hundreds of years? In discussions about AI x-risk, I mostly see something like 20-100 years as the relevant timescale in which to act (i.e. by the end of that period, we will either go extinct or else build an aligned AGI and reach a technological singularity). Looking at Figure 7, if we only look ahead 100 years, it seems like the risk of extinction actually goes up in the accelerated growth scenario.
-
What do you think of Wei Dai’s argument that safe AGI is harder to build than unsafe AGI and we are currently putting less effort into the former, so slower growth gives us more time to do something about AI x-risk (i.e. slower growth is better)?
-
What do you think of Eliezer Yudkowsky’s argument that work for building an unsafe AGI parallelizes better than work for building a safe AGI, and that unsafe AGI benefits more in expectation from having more computing power than safe AGI, both of which imply that slower growth is better from an AI x-risk viewpoint?
-
What do you think of Nick Bostrom’s urn analogy for technological developments? It seems like in the analogy, faster growth just means pulling out the balls at a faster rate without affecting the probability of pulling out a black ball. In other words, we hit the same amount of risk but everything just happens sooner (i.e. growth is neutral).
-
Looking at Figure 7, my “story” for why faster growth lowers the probability of extinction is this: The richer people are, the less they value marginal consumption, so the more they value safety (relative to consumption). Faster growth gets us sooner to the point where people are rich and value safety. So faster growth effectively gives society less time in which to mess things up (however, I’m confused about why this happens; see the next point). Does this sound right? If not, I’m wondering if you could give a similar intuitive story.
-
I am confused why the height of the hazard rate in Figure 7 does not increase in the accelerated growth case. I think equation (7) for might be the cause of this, but I’m not sure. My own intuition says accelerated growth not only condenses along the time axis, but also stretches along the vertical axis (so that the area under the curve is mostly unaffected).
As an extreme case, suppose growth halted for 1000 years. It seems like in your model, the graph for hazard rate would be constant at some fixed level, accumulating extinction probability during that time. But my intuition says the hazard rate would first drop near zero and then stay constant, because there are no new dangerous technologies being invented. At the opposite extreme, suppose we suddenly get a huge boost in growth and effectively reach “the end of growth” (near period 1800 in Figure 7) in an instant. Your model seems to say that the graph would compress so much that we almost certainly never go extinct, but my intuition says we do experience a lot of risk for extinction. Is my interpretation of your model correct, and if so, could you explain why the height of the hazard rate graph does not increase?
This reminds me of the question of whether it is better to walk or run in the rain (keeping distance traveled constant). We can imagine a modification where the raindrops are motionless in the air.
-
The timing of this AMA is pretty awkward, since many people will presumably not have access to the book or will not have finished reading the book. For comparison, Stuart Russell’s new book was published in October, and the AMA was in December, which seems like a much more comfortable length of time for people to process the book. Personally, I will probably have a lot of questions once I read the book, and I also don’t want to waste Toby’s time by asking questions that will be answered in the book. Is there any way to delay the AMA or hold a second one at a later date?
How is Nonlinear currently funded, and how does it plan to get funding for the RFPs?
The 2017 MIRI fundraiser post says “We plan to say more in the future about the criteria for strategically adequate projects in 7a” and also “A number of the points above require further explanation and motivation, and we’ll be providing more details on our view of the strategic landscape in the near future”. As far as I can tell, MIRI hasn’t published any further explanation of this strategic plan. Is MIRI still planning to say more about its strategic plan in the near future, and if so, is there a concrete timeframe (e.g. “in a few months”, “in a year”, “in two years”) for publishing such an explanation?
(Note: I asked this question a while ago on LessWrong.)
Animal Charity Evaluators previously had a page on Wikipedia, but was deleted after discussion. You can see a copy of what the page looked like, which can also be used in case someone wants to write the page.
My guess (based on intuition/experience and without spending any time digging up sources) is that almost all of these do not meet Wikipedia’s general notability guideline, so it is somewhat uncertain as to whether they would survive for long (if someone were to write the page). In other words, they might be deleted like the ACE article.
The Chivers book will likely meet the notability criteria for books (if it hasn’t already).
Donations List Website: tutorial and request for feedback
Do you have any thoughts on Qualia Research Institute?
I have two questions:
Is there a full list of grantees and respective grant amounts for the referral-based round?
Is there some sort of evaluation process for funded projects that have concluded? I am curious especially about the outcomes of the projects that were funded in the 2017 round (that have now had the money for about a year). This question was asked about a year ago, but the details seemed uncertain at the time so I am re-asking the question.
I am worried that exposing oneself to extreme amounts of suffering without also exposing oneself to extreme amounts of pleasure, happiness, tranquility, truth, etc., will predictably lead one to care a lot more about reducing suffering compared to doing something about other common human values, which seems to have happened here. And the fact that certain experiences like pain are a lot easier to induce (at extreme intensities) than other experiences creates a bias in which values people care the most about.
Carl Shulman made a similar point in this post: “This is important to remember since our intuitions and experience may mislead us about the intensity of pain and pleasure which are possible. In humans, the pleasure of orgasm may be less than the pain of deadly injury, since death is a much larger loss of reproductive success than a single sex act is a gain. But there is nothing problematic about the idea of much more intense pleasures, such that their combination with great pains would be satisfying on balance.”
Personally speaking, as someone who has been depressed and anxious most of my life and sometimes have (unintentionally) experienced extreme amounts of suffering, I don’t currently find myself caring more about pleasure/happiness compared to pain/suffering (I would say I care about them roughly the same). There’s also this thing I’ve noticed where sometimes when I’m suffering a lot, the suffering starts to “feel good” and I don’t mind it as much, and symmetrically, when I’ve been happy the happiness has started to “feel fake” somehow so overall I feel pretty confused about what terminal values I am even optimizing for (but thankfully it seems like on the current strategic landscape I don’t need to figure this out immediately).
On the SSC roadtrip post, you say “After our trip, I’ll write up a post-mortem for other people who might be interested in doing things like this in the future”. Are you still planning to write this, and if so, when do you expect to publish it?
The inconsistency is itself a little concerning.
I am one of the contributors to the Donations List Website (DLW), the site you link to. DLW is not affiliated with the EA Hotel in anyway (although Vipul, the maintainer of DLW, made a donation to the EA Hotel). Some reasons for the discrepancy in this case:
As stated in bold letters at the top of the page, “Current data is preliminary and has not been completely vetted and normalized”. I don’t think this is the main reason in this case.
Pulling data into DLW is not automatic, so there is a lag between when the donations are made and when they appear on DLW.
DLW only tracks public donations.
Funding chains in the x-risk/AI safety ecosystem
For the past five years I have been doing contract work for a bunch of individuals and organizations, often overlapping with the EA movement’s interests. For a list of things I’ve done, you can see here or here. I can say more about how I got started and what it’s like to do this kind of work if there is interest.
I am wondering how the fund managers are thinking more long-term about encouraging more independent researchers and projects to come into existence and stay in existence. So far as I can tell, there hasn’t been much renewed granting to independent individuals and projects (i.e. granting for a second or third time to grantees who have previously already received an LTFF grant). Do most grantees have a solid plan for securing funding after their LTFF grant money runs out, and if so what do they tend to do?
I think LTFF is doing something valuable by giving people the freedom to not “sell out” to more traditional or mass-appeal funding sources (e.g. academia, established orgs, Patreon). I’m worried about a situation where receiving a grant from LTFF isn’t enough to be sustainable, so that people go back to doing more “safe” things like working in academia or at an established org.
Any thoughts on this topic?