Ajeya

• Ajeya 18 Aug 2016 0:17 UTC

  6 points

  0 ∶ 0

  in reply to: Kerry_Vaughan’s comment on: Effec­tive Altru­ists re­ally love EA: Ev­i­dence from EA Global

  Some kind of anonymous survey mechanism that managed to capture people who had interacted with EA in a low-to-medium-intensity way (eg, through the Facebook group or one of the websites, through playing a giving game at a campus group, attending one meet up of a campus group, etc) and tracked a) whether they interacted at higher-intensity (eg, applying to EAG) later, and b) whether they internally felt it was welcoming.

• Ajeya 4 Dec 2016 22:47 UTC

  2 points

  0 ∶ 0

  in reply to: AlyssaVance’s comment on: Con­tra the Giv­ing What We Can pledge

  It seems like “deeply committed” is doing a lot of work there. In the last EA survey, it seemed like the median donation from a person who identified as “EA”, listed “earning to give” as their career, was not a student, and believed they should give now rather than give later was $1933. At typical starting software engineer salaries (which I would guess is a typical career for a median “earning to give” EA), this represents a 1-5% donation. This suggests the pledge would increase the donations of over 50% of EAs who list their primary career path as earning to give (so the argument that the mental effort needed to keep the pledge would distract from their careers doesn’t apply). Link to analysis here: https://​​www.facebook.com/​​bshlgrs/​​posts/​​10208520740630756?match=YnVjayBzaGxlZ2VyaXMsc2hsZWdlcmlzLHN1cnZleSxidWNr

  Edit: Speaking for myself only, not my employer.

• Ajeya 5 Dec 2016 19:16 UTC

  0 points

  0 ∶ 0

  in reply to: Dan_Keys’s comment on: Con­tra the Giv­ing What We Can pledge

  Thanks Dan! I didn’t know this, I’ll look more closely at the data when I get the chance.

• Ajeya 13 Dec 2016 23:25 UTC

  18 points

  0 ∶ 0

  on: Are You Sure You Want To Donate To The Against Malaria Foun­da­tion?

  Hi Michael! Thanks for engaging with GiveWell research, we appreciate it. As others in the comments have pointed out, many of the critiques in your post would have applied to our earlier November 2015 cost-effectiveness analysis (CEA). Our 2016 CEA has changed a lot, in part because more staff engaged deeply with the kind of population ethics considerations you point out. Because of this, I think most people will not have to make substantial discounts to our 2016 cost-effectiveness estimate of the Against Malaria Foundation to account for their values. I wrote a longer response on the GiveWell blog here: http://​​blog.givewell.org/​​2016/​​12/​​12/​​amf-population-ethics/​​

• Ajeya 21 Dec 2016 20:24 UTC

  2 points

  0 ∶ 0

  in reply to: Benjamin_Todd’s comment on: Thoughts on the “Meta Trap”

  Re: Leisure time. I think I would have probably either taken another class, gotten a part-time paying job as a TA, or done technical research with a professor if I weren’t leading EAB (which took ~10 hours of my time each week). I’m not positive how representative this is across the board, but I think this is likely true of at least some other chapter leaders, and more likely to be true of the most dedicated (who probably produce a disproportionate amount of the value of student groups).

• Ajeya 17 Jul 2017 4:11 UTC

  8 points

  0 ∶ 0

  on: An ar­gu­ment for broad and in­clu­sive “mind­set-fo­cused EA”

  Views my own, not my employers.

  Thanks for writing this up! I agree that it could be a big win if general EA ideas besides cause prioritization (or the idea of scope-limited cause prioritization) spread to the point of being as widely accepted as environmentalism. Some alternatives to this proposal though:

  1. It might be better to spread rationality and numeracy concepts like expected value, opportunity costs, comparative advantage, cognitive biases, etc completely unconnected to altruism than to try to explicitly spread narrow or cause-specific EA. People on average care much more about being productive, making money, having good relationships, finding meaning, etc than about their preferred altruistic causes. And it really would be a big win if they succeeded—less ambiguously so than with narrow EA I think (see Carl’s comment below). The biggest objection to this is probably crowdedness/​lack of obvious low-hanging fruit.

  2. Another alternative might be to focus on spreading the prerequisites/​correlates of cause-neutral, intense EA: e.g. math education, high levels of caring/​empathy, cosmopolitanism, motivation to think systematically about ethics, etc. I’m unsure how difficult this would be.

  Both of these alternatives seem to have what is (to me) an advantage: they don’t involve the brand and terminology of EA. I think it would be easier to push on the frontiers of cause-neutral/​broad EA if the label were a good signal of a large set of pretty unusual beliefs and attitudes, so that people can have high trust collaboration relatively quickly.

  FWIW, I think I would be much more excited to evangelize broad low-level EA memes if there were some strong alternative channel to distinguish cause-neutral, super intense/​obsessive EAs. Science has a very explicit distinction between science fans and scientists, and a very explicit funnel from one to the other (several years of formal education). EA doesn’t have that yet, and may never. My instinct is that we should work on building a really really great “product”, then build high and publicly-recognized walls around “practitioners” and “consumers” (a practical division of labor rather than a moral high ground thing), and then market the product hard to consumers.

New Re­port on the Welfare Differ­ences Between Cage and Cage-Free Housing

Draft re­port on AI timelines

AMA: Ajeya Co­tra, re­searcher at Open Phil

• Ajeya 1 Feb 2021 18:50 UTC

  28 points

  0 ∶ 0

  in reply to: kokotajlod’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  Thanks so much, that’s great to hear! I’ll answer your first question in this comment and leave a separate reply for your Murphyjitsu question.
  
  First of all, I definitely agree that the difference between 2050 and 2032 is a big deal and worth getting to the bottom of; it would make a difference to Open Phil’s prioritization (and internally we’re trying to do projects that could convince us of timelines significantly shorter than in my report). You may be right that it could have a counterintuitively small impact on many individual people’s career choices, for the reasons you say, but I think many others (especially early career people) would and should change their actions substantially.
  
  I think there are roughly three types of reasons why Bob might disagree with Alice about a bottom line conclusion like TAI timelines, which correspond to three types of research or discourse contributions Bob could make in this space:
  
  1. Disagreements can come from Bob knowing more facts than Alice about a key parameter, which can allow Bob to make “straightforward corrections” to Alice’s proposed value for that parameter. E.g., “You didn’t think much about hardware, but I did a solid research project into hardware and I think experts would agree that because of optical computing progress will be faster than you assumed; changing to the better values makes timelines shorter.” If Bob does a good enough job with this empirical investigation, Alice will often just say “Great, thanks!” and adopt Bob’s number.
  2. Disagreements can come from Bob modeling out a part of the world in more mechanistic detail that Alice fudged or simplified, which can allow Bob to propose a better structure than Alice’s model. E.g., “You agree that earlier AI systems can generate revenue which can be reinvested into AI research but you didn’t explicitly model that and just made a guess about spending trajectory; I’ll show that accounting for this properly would make timelines shorter.” Alice may feel some hesitance adopting Bob’s model wholesale here, because Alice’s model may fudge/​elide one thing in an overly-conservative direction which she feels is counterbalanced by fudging/​eliding another thing in an overly-aggressive direction, but it will often be tractable to argue that the new model is better and Alice will often be happy to adopt it (perhaps changing some other fudged parameters a little to preserve intuitions that seemed important to her).
  3. Finally, disagreements can come from differences in intuition about the subjective weight different considerations should get when coming up with values for the more debatable parameters (such as the different biological anchor hypotheses). It’s more difficult for Bob to make a contribution toward changing Alice’s bottom line here, because a lot of the action is in hard-to-access mental alchemy going on in Alice and Bob’s minds when they make difficult judgment calls. Bob can try to reframe things, offer intuition pumps, trace disagreements about one topic back to a deeper disagreement about another topic and argue about that, and so on, but he should expect it to be slow going and expect Alice to be pretty hard to move.
  
  In my experience, most large and persistent disagreements between people about big-picture questions like TAI timelines or the magnitude of risk from AI are mostly the third kind of disagreement, and these disagreements can be entangled with dozens of other differences in background assumptions /​ outlook /​ worldview. My sense is that your most major disagreements with me fall into the third category: you think that I’m overweighting the hypothesis that we’d need to do meta-learning in which the “inner loop” takes a long subjective time; you may also think that I’m underweighting the possibility of sudden takeoff or overweighting the efficiency of markets in a certain way, which leads me to lend too much credence to considerations like “Well if the low end of the compute range is actually right, we should probably be seeing more economic impact from the slightly-smaller AI systems right now.” If you were to change my mind on this, it might not even be from doing “timelines research”: maybe you do “takeoff speeds research” that convinces me to take sudden takeoff more seriously, which in turn causes me to take shorter timelines (which would imply more sudden takeoff) more seriously.
  
  I’d say tackling category 3 disagreements is high risk and effort but has the possibility of high reward, and tackling category 1 disagreements is lower risk and effort with more moderate reward. My subjective impression is that EAs tend to under-invest in tackling categories 1 and 2 because they perceive category 3 as where the real action is—in some sense they’re right about that, but they may underestimate how hard it’ll be to change people’s minds there. For example, changing someone’s minds about a category 3 disagreement often greatly benefits from having a lot of face time with them, which isn’t very scalable, and arguments may be more particular to individuals: what finally convinces Alice may not be moving to Charlie.
  
  I think one potential way to get at a category 3 disagreement about a long-term forecast is by proposing bets about nearer-term forecasts, although I think this is often a lot harder than it sounds, because people are sensitive to the possibility of “losing on a technicality”: they were right about the big picture but wrong about how that big picture actually translates to a near-term prediction. Even making short-term bets often benefits from having a lot of face time to hash out the terms.

  What links here?

  • Fun with +12 OOMs of Compute by Daniel Kokotajlo (LessWrong; 1 Mar 2021 13:30 UTC; 224 points)
  • EA Fo­rum Prize: Win­ners for Jan­uary 2021 by Aaron Gertler (2 Apr 2021 2:58 UTC; 43 points)
  • Sce­nar­ios and Warn­ing Signs for Ajeya’s Ag­gres­sive, Con­ser­va­tive, and Best Guess AI Timelines by Kevin Liu (LessWrong; 29 Mar 2021 1:38 UTC; 25 points)

• Ajeya 1 Feb 2021 19:03 UTC

  8 points

  0 ∶ 0

  in reply to: kokotajlod’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  Murphyjitsu: Conditional on TAI being built in 2025, what happened? (i.e. how was it built, what parts of your model were wrong, what do the next 5 years look like, what do the 5 years after 2025 look like?)

  On the object level, I think it would probably turn out to be the case that a) I was wrong about horizon length and something more like ~1 token was sufficient, b) I was wrong about model size and something more like ~10T parameter was sufficient. On a deeper level, it would mean I was wrong about the plausibility of ultra-sudden takeoff and shouldn’t have placed as much weight as I did on the observation that AI isn’t generating a lot of annual revenue right now and its value-added seems to have been increasing relatively smoothly so far.
  
  I would guess that the model looks like a scaled-up predictive model (natural language and/​or code), perhaps combined with simple planning or search. Maybe a coding model rapidly trains more-powerful successors in a pretty classically Bostromian /​ Yudkowskian way.
  
  Since this is a pretty Bostromian scenario, and I haven’t thought deeply about those scenarios, I would default to guessing that the world after looks fairly Bostromian, with risks involving the AI forcibly taking control of most of the world’s resources, and the positive scenario involving cooperatively using the AI to prevent other x-risks (including risks from other AI projects).

• Ajeya 1 Feb 2021 19:10 UTC

  9 points

  0 ∶ 0

  in reply to: richard_ngo’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  I generally spend most of my energy looking for inside-view considerations that might be wrong, because they are more likely to suggest a particular directional update (although I’m not focused only on inside view arguments specifically from ML researchers, and place a lot of weight on inside view arguments from generalists too).
  
  It’s often hard to incorporate the most outside-view considerations into bottom line estimates, because it’s not clear what their implication should be. For example, the outside-view argument “it’s difficult to forecast the future and you should be very uncertain” may imply spreading probability out more widely, but that would involve assigning higher probabilities to TAI very soon, which is in tension with another outside view argument along the lines of “Predicting something extraordinary will happen very soon has a bad track record.”

• Ajeya 1 Feb 2021 19:23 UTC

  6 points

  0 ∶ 0

  in reply to: Linch’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  I think the inclusion of “in principle” makes the answer kind of boring—when we’re not thinking about practicality at all, I think I’d definitely prefer to know more facts (about e.g. the future of AI or what would happen in the world if we pursued strategy A vs strategy B) than to have better reasoning skills, but that’s not a very interesting answer.
  
  In practice, I’m usually investing a lot more in general reasoning, because I’m operating in a domain (AI forecasting and futurism more generally) where it’s pretty expensive to collect new knowledge/​facts, it’s pretty difficult to figure out how to connect facts about the present to beliefs about the distant future, and facts you could gather in 2021 are fairy likely to be obsoleted by new developments in 2022. So I would say most of my importance-weighted errors are going to be in the general reasoning domain. I think it’s fairy similar for most people at Open Phil, and most EAs trying to do global priorities research or cause prioritization, especially within long-termism. I think the more object-level your work is, the more likely it is that your big mistakes will involve being unaware of empirical details.
  
  However, investing in general reasoning doesn’t often look like “explicitly practicing general reasoning” (e.g. doing calibration training, studying probability theory or analytic philosophy, etc). It’s usually incidental improvement that’s happening over the course of a particular project (which will often involve developing plenty of content knowledge too).

• Ajeya 1 Feb 2021 19:24 UTC

  8 points

  0 ∶ 0

  in reply to: Linda Linsefors’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  I primarily do research rather than grantmaking, but I can give my speculations about what grant opportunities people on the grantmaking side of the organization would be excited about. In general, I think it’s exciting when there is an opportunity to fund a relatively senior person with a strong track record who can manage or mentor a number of earlier-career people, because that provides an opportunity for exponential growth in the pool of people who are working on these issues. For example, this could look like funding a new professor who is aligned with our priorities in a sub-area and wants to mentor students to work on problems we are excited about in that sub-area.
  
  In terms of why more people and projects don’t get funded: at least at Open Phil, grantmakers generally try not to evaluate large numbers of applications or inquiries from earlier-career people individually, because each evaluation can be fairly intensive but the grant size is often relatively small; grantmakers at Open Phil prefer to focus on investigations that could lead to larger grants. Open Phil does offer some scholarships for early career researchers (e.g. here and here), but in general we prefer that this sort of grantmaking be handled by organizations like EA Funds.

• Ajeya 1 Feb 2021 19:28 UTC

  5 points

  0 ∶ 0

  in reply to: Aryeh Englander’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  That’s fair, and I do try to think about this sort of thing when choosing e.g. how wide to make my probability distributions and where to center them; I often make them wider than feels reasonable to me. I didn’t mean to imply that I explicitly avoid incorporating such outside view considerations, just that returns to further thinking about them are often lower by their nature (since they’re often about unkown-unkowns).

• Ajeya 1 Feb 2021 19:39 UTC

  10 points

  0 ∶ 0

  in reply to: abergal’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  Thanks! :)
  
  The first time I really thought about TAI timelines was in 2016, when I read Holden’s blog post. That got me to take the possibility of TAI soonish seriously for the first time (I hadn’t been explicitly convinced of long timelines earlier or anything, I just hadn’t thought about it).
  
  Then I talked more with Holden and technical advisors over the next few years, and formed the impression that there was a relatively simple argument that many technical advisors believed that if a brain-sized model could be transformative, then there’s a relatively tight argument implying it would take X FLOP to train it, which would become affordable in the next couple decades. That meant that if we had a moderate probability on the first premise, we should have a moderate probability on TAI in the next couple decades. This made me take short timelines even more seriously because I found the biological analogy intuitively appealing, and I didn’t think that people who confidently disagreed had strong arguments against it.
  
  Then I started digging into those arguments in mid-2019 for the project that ultimately became the report, and I started to be more skeptical again because it seemed that even conditional on assuming a brain-sized model would constitute TAI, there are many different hypotheses you could have about how much computation it would take to train it (what eventually became the biological anchors), and different technical advisors believed in different versions of this. In particular, it felt like the notion of a horizon length made sense and incorporating it into the argument(s) made timelines seem longer.

  Then after writing up an earlier draft of the report, it felt like a number of people (including those who had longish timelines) felt that I was underweighting short and medium horizon lengths, which caused me to upweight those views some.

• Ajeya 1 Feb 2021 19:47 UTC

  6 points

  0 ∶ 0

  in reply to: MichaelA’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  In my reply to Linch, I said that most of my errors were probably in some sense “general reasoning” errors, and a lot of what I’m improving over the course of doing my job is general reasoning. But at the same time, I don’t think that most EAs should spend a large fraction of their time doing things that look like explicitly practicing general reasoning in an isolated or artificial way (for example, re-reading the Sequences, studying probability theory, doing calibration training, etc). I think it’s good to be spending most of your time trying to accomplish something straightforwardly valuable, which will often incidentally require building up some content expertise. It’s just that a lot of the benefit of those things will probably come through improving your general skills.

  What links here?

  • MichaelA's comment on AMA: Ajeya Co­tra, re­searcher at Open Phil by Ajeya (2 Feb 2021 2:19 UTC; 2 points)

• Ajeya 1 Feb 2021 20:01 UTC

  5 points

  0 ∶ 0

  in reply to: Alex HT’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  Hm, I think I’d say progress at this stage largely looks like being better able to cash out disagreements about big-picture and long-term questions in terms of disagreements about more narrow, empirical, or near-term questions, and then trying to further break down and ultimately answer these sub-questions to try to figure out which big picture view(s) are most correct. I think given the relatively small amount of effort put into it so far and the intrinsic difficulty of this project, returns have been pretty good on that front—it feels like people are having somewhat narrower and more tractable arguments as time goes on.

  I’m not sure about what exact skillsets the field most needs. I think the field right now is still in a very early stage and could use a lot of disentanglement research, and it’s often pretty chaotic and contingent what “qualifies” someone for this kind of work. Deep familiarity with the existing discourse and previous arguments/​attempts at disentanglement is often useful, and some sort of quantitative background (e.g. economics or computer science or math) or mindset is often useful, and subject matter expertise (in this case machine learning and AI more broadly) is often useful, but none of these things are obviously necessary or sufficient. Often it’s just that someone happens to strike upon an approach to the question that has some purchase, they write it up on the EA Forum or LessWrong, and it strikes a chord with others and results in more progress along those lines.

  What links here?

  • EA Fo­rum Prize: Win­ners for Jan­uary 2021 by Aaron Gertler (2 Apr 2021 2:58 UTC; 43 points)

• Ajeya 1 Feb 2021 20:06 UTC

  16 points

  0 ∶ 0

  in reply to: tamgent’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  I’m afraid I don’t have crisp enough models of the simulation hypothesis and related sub-questions to have a top n list. My biggest question is something more like “This seems like a pretty fishy argument, and I find myself not fully getting or buying it despite not being able to write down a simple flaw. What’s up with that? Can somebody explain away my intuition that it’s fishy in a more satisfying way and convince me to buy it more wholeheartedly, or else can someone pinpoint the fishiness more precisely?” My second biggest question is something like “Does this actually have any actionable implications for altruists/​philanthropists? What are they, and can you justify them in a way that feels more robust and concrete and satisfying than earlier attempts, like Robin Hanson’s How to Live in a Simulation?”

• Ajeya 1 Feb 2021 20:10 UTC

  6 points

  0 ∶ 0

  in reply to: Ajeya’s comment on: AMA: Ajeya Co­tra, re­searcher at Open Phil

  It occurred to me that another way to try to move someone on complicated category 3 disagreements might be to put together a well-constructed survey of a population that the person is inclined to defer to. This approach is definitely still tricky: you’d have to convince the person that the relevant population was provided with the strongest arguments for that person’s view in addition to your counterarguments, and that the individuals surveyed were thinking about it reasonably hard. But if done well, it could be pretty powerful.