Ben Snodin

• Ben SnodinSep 7, 2021, 6:02 AM

  3 points

  0 ∶ 0

  in reply to: Linch’s comment on: When pool­ing fore­casts, use the ge­o­met­ric mean of odds

  Thanks both (and Owen too), I now feel more confident that geometric mean of odds is better!

  (Edit: at 1:4 odds I don’t feel great about a blanket recommendation, but I guess the odds at which you’re indifferent to taking the bet are more heavily stacked against us changing our mind. And Owen’s <1% is obviously way lower)

In­ves­ti­gat­ing how tech­nol­ogy-fo­cused aca­demic fields be­come self-sustaining

• Ben SnodinSep 4, 2021, 6:47 AM

  11 points

  0 ∶ 1

  on: When pool­ing fore­casts, use the ge­o­met­ric mean of odds

  (don’t feel extremely confident about the below but seemed worth sharing)

  I think it’s really great to flag this! But as I mentioned to you elsewhere I’m not sure we’re certain enough to make a blanket recommendation to the EA community.

  I think we have some evidence that geometric mean of odds is better, but not that much evidence. Although I haven’t looked into the evidence that Simon_M shared from Metaculus.

  I guess I can potentially see us changing our minds in a year’s time and deciding that arithmetic mean of probabilities is better after all, or that some other method is better than both of these.

  Then maybe people will have made a costly change to a new method (learning what odds are, what a geometric mean is, learning how to apply it in practice, maybe understanding the argument for using the new method) that turns out not to have been worth it.

• Ben SnodinAug 26, 2021, 5:06 PM

  3 points

  0 ∶ 0

  in reply to: kokotajlod’s comment on: Ta­boo “Out­side View”

  Nice, thanks for this!

  I mean, depending on what you mean by “an okay approach sometimes… especially when you want to do something quick and dirty” I may agree with you! What I said was:

  This is not Tetlock’s advice, nor is it the lesson from the forecasting tournaments, especially if we use the nebulous modern definition of “outside view” instead of the original definition.

  I guess I was reacting to the part just after the bit you quoted

  For an entire book written by Yudkowsky on why the aforementioned forecasting method is bogus

  Which I took to imply “Daniel thinks that the aforementioned forecasting method is bogus”. Maybe my interpretation was incorrect. Anyway, seems very possible we in fact roughly agree here.

  Re your 1, 2, 3, 4: It seems cool to try doing 4, and I can believe it’s better (I don’t have a strong view). Fwiw re 1 vs 2, my initial reaction is that partitioning by outside/​inside view lets you decide how much weight you give to each, and maybe we think that for non-experts it’s better to mostly give weight to the outside view, so the partitioning performed a useful service. I guess this is kind of what you were trying to argue against and unfortunately you didn’t convince me to repent :).

• Ben SnodinAug 21, 2021, 11:15 AM

  9 points

  0 ∶ 0

  on: Ben_Sn­odin’s Shortform

  Here are some forecasts for near-term progress /​ impacts of AI on research. They are the results of some small-ish number of hours of reading + thinking, and shouldn’t be taken at all seriously. I’m sharing in case it’s interesting for people and especially to get feedback on my bottom line probabilities and thought processes. I’m pretty sure there are some things I’m very wrong about in the below and I’d love for those to be corrected.

  1. Deepmind will announce excellent performance from Alphafold2 (AF2) or some successor /​ relative for multi-domain proteins by end of 2023; or some other group will announced this using some AI scheme: 80% probability

  2. Deepmind will announce excellent performance from AF2 or some successor /​ relative for protein complexes by end of 2023; or some other group will announced this using some AI scheme: 70% probability

  3. Widespread adoption of a system like OpenAI Codex for data analysis will happen by end of 2023: 20% probability

  I realise that “excellent performance” etc is vague, I choose to live with that rather than putting in the time to make everything precise (or not doing the exercise at all).

  If you don’t know what multi-domain proteins and protein complexes are, I found this Mohammed Al Quraishi blog very useful (maybe try ctrl-f for those terms), although maybe you need to start with some relevant background knowledge. I don’t have a great sense for how big a deal this would be for various areas of biological science, but my impression is that they’re both roughly the same order of magnitude of usefulness as getting excellent performance on single-domain proteins was (i.e. what AF2 has already achieved).

  As for why:

  80% chance that excellent AI performance on multi-domain proteins is announced by end of 2023

  • Top reasons for event happening

    • Deepmind apparently wants to tackle protein complexes (“Based on this and DeepMind’s repeated assertions…”) and multi-domain proteins seem like a stepping stone for that

    • Mohammed Al Quraishi already thought AF2 dealt with multi-domain proteins “just fine” in Dec 2020

    • It’s an extremely similar problem to the one they already cracked (single domains) and it seems tractable

  • Top reasons against

    • Maybe they won’t announce it, because it’s not newsworthy enough; or they’ll bundle it with some bigger announcement with lots of cool results (resulting in delayed announcement)

  • Other reasons against

    • In particular, the results from the next CASP competition will presumably be announced in December 2022; if they haven’t cracked it by then, maybe we won’t hear about it by end of 2023

      • They’d need to get there by April 2022 (I think that is the submission deadline for CASP)

    • Maybe it will turn out to be way less tractable than expected

    • Maybe Deepmind will have other, even more pressing priorities, or some key people will unexpectedly leave, or they’ll lose funding, or something else unexpected happens

  • Key uncertainties

    • Are rival protein folding schemes targeting this?

  70% chance that excellent AI performance on protein complexes is announced by end of 2023

  • Top reasons for event happening

    • Deepmind apparently wants to tackle protein complexes (“Based on this and DeepMind’s repeated assertions…”)

    • Mohammed Al Quraishi already thought it dealt with multi-domain proteins “just fine” in Dec 2020, and these are a stepping stone to protein complexes

    • There is quite a lot of data (though 10x less than for single proteins)

      • Unsure whether transfer learning (I maybe using the wrong term) is relevant here?

  • Top reasons against

    • Maybe even if it’s done by say mid 2023 it won’t be announced until after 2023 because of Deepmind’s media strategy

      • In particular, targeting a CASP would seem to require the high performance to be achieved by mid 2022; maybe this is the most likely scenario in worlds where Deepmind announces this before end of 2023

      • (although if Deepmind doesn’t get there by CASP15, it seems like another group might announce something in say 2023)

    • Protein complexes are (maybe?) qualitatively different to single proteins

  • Other reasons against

    • Maybe the lack of data will be decisive

    • Maybe Deepmind’s priorities will change, etc, as in noted above in the multi-domain case

  • Key uncertainties

    • Are rival schemes targeting this?

  20% chance of widespread adoption of a system like OpenAI Codex for data analysis by end of 2023

  (NB this is just about data analysis /​ “data science” rather than about usage of Codex in general)

  • My “best guess” scenario

    • OpenAI releases an API for data science that is cheap but not free. In its current iteration, the software is “handy” but not more than that. A later iteration, released in 2023, is significantly more powerful and useful. But by the end of 2023 it is still not yet “widely used”.

  • Some reasons against event happening

    • Maybe Codex is currently not that useful in practice for data analysis

    • I think OpenAI won’t release it for free so it won’t become part of the “standard toolkit” in the same way that e.g. RStudio has

    • Things like RStudio take a long time to diffuse /​ become adopted

      • E.g. my guess is ~5 years to get 25% uptake of for ipython notebook or rstudio by data scientists, or something like that

  • Key uncertainties

    • How much are OpenAI going to push this on people?

      • How much are they pushing the data science aspect particularly?

    • Will this be ~free to use or will it be licensed?

    • How quickly will it improve? How often does OpenAI release improved versions of things?

    • How fast did ipython notebook/​rstudio get adopted?

    • How much has the GPT-3 API been used so far?

  • Some useful links

    • From the OpenAI website: “During the initial period, OpenAI Codex will be offered for free”

    • This guy at VentureBeat thinks that Microsoft will make the profits from Codex (but that doesn’t seem to rule out it being widely used to assist with data analysis; just implies that the data analysis software will be owned by microsoft, I guess?)

    • The Verge says “although Codex is initially being released as free API, OpenAI will start charging for access at some point in the future.”

    • The impression I get from Appendix H of OpenAI’s paper on Codex is that they expect at least fairly wide adoption of Codex or something similar

• Ben SnodinAug 20, 2021, 2:11 PM

  4 points

  0 ∶ 0

  on: Ta­boo “Out­side View”

  Separately, various people seem to think that the appropriate way to make forecasts is to (1) use some outside-view methods, (2) use some inside-view methods, but only if you feel like you are an expert in the subject, and then (3) do a weighted sum of them all using your intuition to pick the weights. This is not Tetlock’s advice, nor is it the lesson from the forecasting tournaments, especially if we use the nebulous modern definition of “outside view” instead of the original definition. (For my understanding of his advice and those lessons, see this post, part 5. For an entire book written by Yudkowsky on why the aforementioned forecasting method is bogus, see Inadequate Equilibria, especially this chapter. Also, I wish to emphasize that I myself was one of these people, at least sometimes, up until recently when I noticed what I was doing!)

  This is a bit tangential to the main point of your post, but I thought I’d give some thoughts on this, partly because I basically did exactly this procedure a few months ago in an attempt to come to a personal all-things-considered view about AI timelines (although I did “use some inside-view methods” even though I don’t at all feel like I’m an expert in the subject!).

  I liked your AI Impacts post, thanks for linking to it! Maybe a good summary of the recommended procedure is the part at the very end. I do feel like it was useful for me to read it.

  Tetlock describes how superforecasters go about making their predictions.⁵⁶ Here is an attempt at a summary:

  1. Sometimes a question can be answered more rigorously if it is first “Fermi-ized,” i.e. broken down into sub-questions for which more rigorous methods can be applied.

  2. Next, use the outside view on the sub-questions (and/​or the main question, if possible). You may then adjust your estimates using other considerations (‘the inside view’), but do this cautiously.

  3. Seek out other perspectives, both on the sub-questions and on how to Fermi-ize the main question. You can also generate other perspectives yourself.

  4. Repeat steps 1 – 3 until you hit diminishing returns.

  5. Your final prediction should be based on an aggregation of various models, reference classes, other experts, etc.

  I’m less sure about the direct relevance of Inadequate Equilibria for this, apart from it making the more general point that ~”people should be less scared of relying on their own intuition /​ arguments /​ inside view”. Maybe I haven’t scrutinised it closely enough.

  To be clear, I don’t think “weighted sum of ‘inside views’ and ‘outside views’” is the gold standard or something. I just think it’s an okay approach sometimes (maybe especially when you want to do something “quick and dirty”).

  If you strongly disagree (which I think you do), I’d love for you to change my mind! :)

• Ben SnodinAug 19, 2021, 5:52 PM

  1 point

  0 ∶ 0

  in reply to: Ben Snodin’s comment on: Ben_Sn­odin’s Shortform

  This from Paul Christiano in 2014 is also very relevant (part of it makes similar points to a lot of the recent stuff from Open Philanthropy, but the arguments are very brief; it’s interesting to see how things have evolved over the years): Three impacts of machine intelligence

• Ben SnodinAug 14, 2021, 10:55 AM

  1 point

  0 ∶ 0

  in reply to: Ben Snodin’s comment on: What EA pro­jects could grow to be­come megapro­jects, even­tu­ally spend­ing $100m per year?

  I realise re-reading this that I’m not sure whether these projects are supposed to cost $100million per year or e.g. $100million over their lifetime or something. Maybe something in between?

• Ben SnodinAug 14, 2021, 10:54 AM

  1 point

  0 ∶ 0

  on: What EA pro­jects could grow to be­come megapro­jects, even­tu­ally spend­ing $100m per year?

  (idea probably stolen from somewhere else) create an organisation employing an army of superforecasters to gather facts and/​or forecasts about the world that are vitally important from an EA perspective.

  Maybe it’s hard to get to $100million? E.g. 400 employees each costing $250k would get you there, which (very naively) seems on the high end of what’s likely to work well. Also e.g. other comments in this post have said that CSET was set up for $55m/​5 years.

• Ben SnodinAug 14, 2021, 10:45 AM

  4 points

  0 ∶ 0

  on: What EA pro­jects could grow to be­come megapro­jects, even­tu­ally spend­ing $100m per year?

  (extremely speculative)

  Promote global cooperation and moral circle expansion by paying people (/​ incentivising them in some smarter way) to have regular video calls with a random other person somewhere on the planet.

• Ben SnodinAug 9, 2021, 3:56 PM

  2 points

  0 ∶ 0

  on: Ben_Sn­odin’s Shortform

  Here are some thoughts after reading a book called “The Inner Game of Tennis” by Timothy Gallwey. I think it’s quite a famous book and maybe a lot of people know it well already. I consider it to be mainly about how to prevent your system 2/​conscious mind/​analytical mind from interfering with the performance of your system 1/​subconscious mind/​intuitive mind. This is explained in the context of tennis, but it seems applicable to many other contexts, as the author himself argues. If that sounds interesting, I recommend checking the book out, it’s short and quite readable.

  My interest in the book comes mainly from thinking about the best way to go about doing research, at a day-to-day level. Although the arguments of the book seem most directly applicable to learning a physical skill/​activity and (to some extent) to performing well at key moments, I still think there are lessons for mental activities performed routinely, i.e. for activities like research.

  I think reading the book has generally pushed me a bit more in favour of “trusting my system 1/​intuitive mind” while doing research, e.g. trusting that my brain is doing some important processing when I feel inclined to just stare into space and not make any apparent progress to whatever it is I’m trying to achieve at that moment. This feels pretty important.

  I think Owen Cotton Barratt says some interesting things about trusting his intuition for prioritisation in this interview with Lynette Bye, which feels kind of related.

  The book predates by many decades Kahneman’s Thinking Fast and Slow, which (I think) popularised the concept of system 1 mind and system 2 mind. The book instead refers to “self 1” and “self 2″ which seem to have roughly similar meanings, although unfortunately reversed: Gallwey’s self 1 and Kahneman’s system 2 refer to the conscious/​analytical mind, while Gallwey’s self 2 and Kahneman’s system 1 refer to the subconscious/​intuitive mind.

  Here are some disorganised notes on bits that seemed worth highlighting (page numbers refer to 2015 edition published by Pan Books):

  • p13 mastering the mental side of tennis:

    • picture desired outcomes as clearly as possible

    • allow self 2 to perform and learn from successes and failures

    • learn to see non-judgementally: see what is happening rather than (just) seeing how well or badly it’s happening

    • all subsidiary to the master skill: relaxed concentration

  • p38 “Remember that you are not your tennis game. You are not your body. Trust the body to learn and to play, as you would trust another person to do a job, and in a short time it will perform beyond your expectations. Let the flower grow.”

  • p41 communicating with self 2

    • Gallwey exhorts the reader to trust their self 2 (system 1 /​ intuitive mind). But how can we be sure that self 2 will be optimising for the thing “we” (self 1) thinks is important? Gallwey gives 3 ways to convey to self 2 what the goal is, in the context of tennis:

      • Asking for results: visualise the exact path of the ball. Hold that image in your mind for several seconds

      • Asking for form: observe some particular aspect of your form (e.g. the flatness of your racket while it moves through the ball). Don’t make an effort to make the change. Just visualise the change you want

      • Asking for qualities: imagine you are playing the role of a top tennis player on the court for a film

        • There are particular benefits of playing the role of someone very different to you

    • I’m not sure how to turn this into policies for doing research well. Things that seem interesting to explore: visualising the output you want at the start of the day; reflecting each day on how what you did links to your ultimate goals; picturing yourself as playing the role of a researcher you admire.

  • p80 on the “ego satisfaction” from a self-1-controlled success

    • Gallwey talks a lot about the ego satisfaction from self-1-controlled success.

    • In the context of research, this doesn’t seem to ring true for me from my experience (maybe it’s obviously true for tennis or similar activities for people who have experience there, I don’t know).

  • p82 “Fighting the mind does not work. What works best is learning to focus it”

    • Gallwey talks about focussing on the seams of the ball and other techniques to focus the (self 1) mind on something kind of irrelevant while playing tennis so that the body and self 2 can perform without interference.

  • p87 on what focus is: “Focus is not achieved by staring hard at something. It is not trying to force focus, nor does it mean thinking hard about something. Natural focus occurs when the mind is interested. When this occurs, the mind is drawn irresistibly toward the object (or subject) of interest. It is effortless and relaxed, not tense and overly controlled.”

    • Re research, this seems like good advice for tackling a difficult problem or making progress on some task. One related thing is that I find it much easier to “effortlessly focus” on what I think is important if I’m free of distractions.

  • p127 On managing stress. Pressures come at us from all corners: demands from partners, bosses, coaches, society, etc. These external demands can end up being internalised by self 1 and feeling as if they’re things you really want, but this is an illusion.

    • (kind of reminds me of the message from another book I liked called Essentialism)

• Ben SnodinJul 30, 2021, 10:11 AM

  13 points

  0 ∶ 0

  on: Ben_Sn­odin’s Shortform

  Takeaways from some reading about economic effects of human-level AI

  I spent some time reading things that you might categorise as “EA articles on the impact of human-level AI on economic growth”. Here are some takeaways from reading these (apologies for not always providing a lot of context /​ for not defining terms; hopefully clicking the links will provide decent context).

  • Economic growth models that (essentially) take growth to be driven by population seem to fit /​ explain historic growth data fairly well.

    • (in at least most cases, I think) these predict that “AI robots” will lead to explosive growth (30%+ GWP growth for 10 years).

    • Things other than AI robots that could cause (effective) population to dramatically increase include: whole brain emulation, artificial wombs, genetic engineering, cultural evolution (naturally favouring cultures that tend to have lots of children), behavioural changes driven by something other than cultural evolution.

  • Ben Garfinkel cast some doubt on some of the data (I think especially pre-1500 CE) that is used to fit e.g. the Roodman model, but Tom Davidson doesn’t think this is a decisive blow for these kinds of models.

  • Tom Davidson thinks that “explosive growth would require AI that substantially accelerates the automation of a very wide range of tasks in production, R&D, and the implementation of new technologies.”

  • Tom Davidson gives ~60% as his probability of explosive growth this century given early enough “virtual AI” (though he says this is extremely rough!).

    • He thinks the main ways virtual AI doesn’t lead to explosive growth are bottlenecks like getting raw materials or conducting physical experiments, or that the economic models that imply explosive growth from AI robots are wrong

  • Tom Davidson’s (toy) mechanism for AI replacing human labour is

    • AI becomes good enough to do any task a top quality human worker could do working remotely

    • This allows us to quickly develop the level of robotics for AI robots capable of explosive growth by replacing human workers at ~almost all tasks

  • Energy payback times for solar panels are currently ~1 year at best (financial payback times are longer). If historic experience curves continue to hold, this could fall to months if solar panels supply a large fraction of current demand.

  • Terrestrial solar power could provide up to 1000x current civilizational energy usage, so there’s room for a new “AI economy” to grow its energy usage significantly (depending on future pre-AI-economy energy usage, of course).

  • Combining experience curves, evidence regarding feasibility of technological capabilities from nature, and availability of solar energy suggests that leaving earth will be feasible, which will in turn make settling other stars in the galaxy feasible, which will in turn (maybe) make settling other galaxies feasible.

  • Roodman’s model is arguably not useful for forecasting the arrival of AI, because it basically assumes powerful AI (it assumes that population is output bottlenecked).

  If you’re interested in more on this topic, I’d highlight Holden Karnofsky’s recent blog series and Tom Davidson’s recent Open Phil report as good places to start.

  In case it’s useful for other people, here’s the main stuff I (at least partially) read /​ listened to:

  • Experience curves, large populations, and the long run (from Carl Shulman’s blog)

  • Terrestrial solar energy could eventually support extremely large economies and populations (from Carl Shulman’s blog)

  • Roadmap for the “most important century” series (Holden Karnofsky’s Cold Takes blog post series)

  • Phil Trammell on Economic Growth under Transformative AI (Hear This Idea podcast)

  • Does Economic History Point Toward a Singularity? (Ben Garfinkel EA Forum post)

  • Could Advanced AI Drive Explosive Economic Growth? (report by Tom Davidson of Open Phil; see also accompanying blog post)

• Ben SnodinJul 30, 2021, 7:51 AM

  3 points

  0 ∶ 0

  on: Re­port on Whether AI Could Drive Ex­plo­sive Eco­nomic Growth

  Thanks for this, I think it’s really brilliant, I really appreciate how clearly the details are laid out in the blog and report. It’s really cool to be able to see external reviewer comments too.

  I found it kind of surprising that there isn’t any mention of civilizational collapse etc when thinking about growth outcomes for the 21st century (e.g. in Appendix G, but also apparently in your bottom line probabilities in e.g. Section 4.6 “Conclusion”—or maybe it’s there and I missed it /​ it’s not explicit).

  I guess your probabilities for various growth outcomes in Appendix G are conditional on ~no civilizational collapse (from any cause) and ~no AI-triggered fundamental reshaping of society that unexpectedly prevents growth? Or should I read them more as “conditional on ~no civilizational collapse etc other than due to AI”, with the probability mass for AI-triggered collapse etc being incorporated into your “AI robots don’t have a tendency to drive explosive growth because none of our theories are well-suited for this situation” and/​or “an unanticipated bottleneck prevents explosive growth”?

• Ben SnodinJun 14, 2021, 5:48 PM

  1 point

  0 ∶ 0

  in reply to: Linch’s comment on: Ben_Sn­odin’s Shortform

  Thanks that’s interesting, I’ve heard of it but I haven’t looked into it.

• Ben SnodinJun 8, 2021, 8:23 PM

  7 points

  0 ∶ 0

  on: Ben_Sn­odin’s Shortform

  Causal vs evidential decision theory

  I wrote this last Autumn as a private “blog post” shared only with a few colleagues. I’m posting it publicly now (after mild editing) because I have some vague idea that it can be good to make things like this public. Decision theories are a pretty well-worn topic in EA circles and I’m definitely not adding new insights here. These are just some fairly naive thoughts-out-loud about how CDT and EDT handle various scenarios. If you’ve already thought a lot about decision theory you probably won’t learn anything from this.

  The last two weeks of the Decision Theory seminars I’ve been attending have focussed on contrasting causal decision theory (CDT) and evidential decision theory (EDT). This seems to be a pretty active area of discussion in the literature—one of the papers we looked at was published this year, and another is yet to be published.

  In terms of the history of the field, it seems that Newcomb’s problem prompted a move towards CDT (e.g. in Lewis 1981). I find that pretty surprising because to me Newcomb’s problem provides quite a bit of motivation for EDT, and without weird scenarios like Newcomb’s problem I think I might have taken something like CDT to be the default, obviously correct theory. But it seems like you didn’t need to worry about having a “causal aspect” to decision theories until Newcomb’s problem and other similar problems brought out a divergence in recommendations from (what became known as) CDT and EDT.

  I guess this is a very well-worn area (especially in places like Lesswrong) but anyway I can’t resist giving my fairly naive take even though I’m sure I’m just repeating what others have said. When I first heard about things like Newcomb’s problem a few years ago I think I was a pretty ardent CDTer, whereas nowadays I am much more sympathetic to EDT.

  In Newcomb’s problem, it seems pretty clear to me that one-boxing is the best option, because I’d rather have $1,000,000 than $1000. Seems like a win for EDT.

  Dicing With Death is designed to give CDT problems, and in my opinion it does this very effectively. In Dicing With Death, you have to choose between going to Aleppo or Damascus, and you know that whichever you choose, death will have predicted your choice and be waiting for you (a very bad outcome for you). Luckily, a merchant offers you a magical coin which you can toss to decide where to go, in which case death won’t be able to predict where you go, giving you a 50% chance of avoiding death. The merchant will charge a small fee for this. However CDT gets into some strange contortions and as a result recommends against paying for the magical coin, even though the outcome if you pay for the magical coin seems clearly better. EDT recommends paying for the coin, another win for EDT.

  To me, The Smoking Lesion is a somewhat problematic scenario for EDT. Still, I feel like it’s possible for EDT to do fine here if you think carefully enough.

  You could make the following simple model for what happens in The Smoking Lesion: in year 1, no-one knows why some people get cancer and some don’t. In year 2, it’s discovered that everyone who smokes develops cancer, and furthermore there’s a common cause (a lesion) that causes both of these things. Everyone smokes iff they have the lesion, and everyone gets cancer iff they have the lesion. In year 3, following the publication of these results, some people who have the lesion try not to smoke. Either (i) none of them can avoid smoking because the power of the lesion is too strong; or (ii) some of them do avoid smoking, but (since they still have the lesion) they still develop cancer. In case (i), the findings from year 2 remain valid even after everyone knows about them. In case (ii), the findings from year 2 are no longer valid: they just tell you about how the world would have been if the correlation between smoking and cancer wasn’t known.

  The cases where you use the knowledge about the year 2 finding to decide not to smoke are exactly the cases where the year 2 finding doesn’t apply. So there’s no point in using the knowledge about the year 2 finding to not smoke: either your not smoking (through extreme self-control etc) is pointless because you still have the lesion and this is a case where the year 2 finding doesn’t apply, or it’s pointless because you don’t have the lesion.

  So it seems to me like the right answer is to smoke if you want to, and I think EDT can recommend this by incorporating the fact that if you choose not to smoke purely because of the year 2 finding, this doesn’t give you any evidence about whether you have the lesion (though this is pretty vague and I wouldn’t be that surprised if making it more precise made me realise it doesn’t work).

  In general it seems like these issues arise from treating the agent’s decision making process as being removed from the physical world—a very useful abstraction which causes issues in weird edge cases like the ones considered above.

• Ben SnodinJun 2, 2021, 9:56 AM

  8 points

  0 ∶ 0

  on: Ben_Sn­odin’s Shortform

  Changing your working to fit the answer

  I wrote this last Autumn as a private “blog post” shared only with a few colleagues. I’m posting it publicly now (after mild editing) because I have some vague idea that it can be good to make things like this public. It is quite rambling and doesn’t really have a clear point (but I think it’s at least an interesting topic).

  Say you want to come up with a model for AI timelines, i.e. the probability of transformative AI being developed by year X for various values of X. You put in your assumptions (beliefs about the world), come up with a framework for combining them, and get an answer out. But then you’re not happy with the answer—your framework must have been flawed, or maybe on reflection one of your assumptions needs a bit of revision. So you fiddle with one or two things and get another answer—now it looks much better, close enough to your prior belief that it seems plausible, but not so close that it seems suspicious.

  Is this kind of procedure valid? Here’s one case where the answer seems to be yes: if your conclusions are logically impossible, you know that either there’s a flaw in your framework or you need to revise your assumptions (or both).

  A closely related case is where the conclusion is logically possible, but extremely unlikely. It seems like there’s a lot of pressure to revise something then too.

  But in the right context revising your model in this way can look pretty dodgy. It seems like you’re “doing things the wrong way round”—what was the point of building the model if you were going to fiddle with the assumptions until you got the answer you expected anyway?

  I think this is connected to a lot of related issues /​ concepts:

  • Model building

    • Option pricing models in finance: you start (both historically and conceptually) with the nice clean Black-Scholes model, which fails to explain actually observed option prices. Due to this, various assumptions are relaxed or modified, adding (arguably, somewhat ad hoc) complexity until, for the right set of parameters, the model gets all (sufficiently important) observed option prices right.

    • Regularisation /​ overfitting in ML: you might think of overfitting as “placing too much weight on getting the answer you expect”.

  • Arguments

    • “One person’s modus ponens is another’s modus tollens”: if we’re presented with a logical argument, usually the person presenting it wants us to accept the premises and agree that the argument is valid, in which case we must accept the conclusion. If we don’t like the conclusion, we often focus on showing that the argument is invalid. But if you think the conclusion is very unlikely, you also have the option of acknowledging the argument as valid, but rejecting one of the premises. There are lots of fun examples of this from science and philosophy on Gwern’s page on the subject.

    • “Begging the question”: a related accusation in philosophy that seems to mean roughly “your conclusion follows trivially from your premises but I reject one of your premises (and by the way it should have been obvious that I’d reject one of your premises so it was a waste of both my time and yours that you made this argument)”

    • Reductio ad absurdum: disprove something by using it as an assumption that leads to an implausible (or maybe logically impossible) conclusion

    • “Proving too much”: an accusation in philosophy that is supposed to count against the argument doing the “proving”.

    • (Not) updating your beliefs from an argument that appears convincing on the face of it: if the conclusions are implausible enough, you might not update your beliefs too much the first time you encounter the argument, even if it appears watertight.

  • Research methods

    • Sanity checking your answer: check that the results of a complex calculation or experiment roughly match the result you get from a quick and crude approach.

  Presumably, you could put this question of whether and how much to modify your model into some kind of formal Bayesian framework where on learning a new argument you update all your beliefs based on your prior beliefs in the premises, conclusion, and validity of the argument. I’m not sure whether there’s a literature on this, or whether e.g. highly skilled forecasters actually think like this.

  In general though, it seems (to me) that there’s something important about “following where the assumptions /​ model takes you”. Maybe, given all the ways we fall short of being perfectly rational, we should (and I think that in fact we do) put more emphasis on this than a perfectly rational Bayesian agent would. Avoiding having a very strong prior on the conclusion seems helpful here.

• Ben SnodinJun 2, 2021, 9:44 AM

  8 points

  0 ∶ 0

  in reply to: MichaelA🔸’s comment on: In­ter­ven­tion op­tions for im­prov­ing the EA-al­igned re­search pipeline

  One (maybe?) low-effort thing that could be nice would be saying “these are my top 5” or “these are listed in order of how promising I think they are” or something (you may well have done that already and I missed it).

• Ben SnodinJun 1, 2021, 9:58 AM

  3 points

  0 ∶ 0

  on: In­ter­ven­tion op­tions for im­prov­ing the EA-al­igned re­search pipeline

  Thanks, I think this is a great topic and this seems like a useful list (although I do find reading through 19 different types of options without much structure a bit overwhelming!).

  I’ll just ~repost a private comment I made before.

  Encouraging and facilitating aspiring/​junior researchers and more experienced researchers to connect in similar ways

  This feels like an especially promising area to me. I’d guess there are lots of cases where this would be very beneficial for the junior researcher and at least a bit beneficial for the experienced researcher. It just needs facilitation (or something else, e.g. a culture change where people try harder to make this happen themselves, some strong public encouragement to juniors to make this happen, …).
  
  This isn’t based on really strong evidence, maybe mostly my own (limited) experience + assuming at least some experienced researchers are similar to me. And that there are lots of excellent junior researcher candidates out there (again from first hand impressions).

  Improving the vetting of (potential) researchers, and/​or better “sharing” that vetting

  This also seems like a big deal and an area where maybe you could improve things significantly with a relatively small amount of effort. I don’t have great context here though.

[Question] How much will pre-trans­for­ma­tive AI speed up R&D?

• Ben SnodinMay 26, 2021, 6:58 AM

  8 points

  0 ∶ 0

  on: Ben_Sn­odin’s Shortform

  Two papers I read on imprecise probabilities

  I wrote this last Autumn as a private “blog post” shared only with a few colleagues. I’m posting it publicly now (after mild editing) because I have some vague idea that it can be good to make things like this public.

  In this post I’m going to discuss two papers regarding imprecise probability that I read this week for a Decision Theory seminar. The first paper seeks to show that imprecise probabilities don’t adequately constrain the actions of a rational agent. The second paper seeks to refute that claim.

  Just a note on how seriously to take what I’ve written here: I think I’ve got the gist of what’s in these papers, but I feel I could spend a lot more time making sure I’ve understood them and thinking about which arguments I find persuasive. It’s very possible I’ve misunderstood or misrepresented the points the papers were trying to make, and I can easily see myself changing my mind about things if I thought and read more.

  Also, a note on terminology: it seems like “sharpness/​unsharpness” and “precision/​imprecision” are used interchangeably in these papers, as are “probability” and “credence”. There might well be subtle distinctions that I’m missing, but I’ll try to consistently use “imprecise probabilities” here.

  Imprecise probabilities

  I imagine there are (at least) several different ways of formulating imprecise probabilities. One way is the following: your belief state is represented by a set of probability functions, and your degree of belief in a particular proposition is represented by the set of values assigned to it by the set of probability functions. You also then have an imprecise expectation: each of your probability functions has an associated expected utility. Sometimes, all of your probability functions will agree on the action that has the highest expected value. In that case, you are rationally required to take that action. But if there’s no clear winner, that means there’s more than one permissible action you could take.

  Subjective Probabilities should be Sharp

  The first paper, Subjective Probabilities should be Sharp, was written in 2010 by Elga. The central claim is that there’s no plausible account of how imprecise probabilities constrain which choices are reasonable for a perfectly rational agent.

  The argument centers around a particular betting scenario: someone tells you “I’m going to offer you bet A and then bet B, regarding a hypothesis H”:

  Bet A: win $15 if H, else lose $10

  Bet B: lose $10 if H, else win $15

  You’re free to choose whether to take bet B independently of whether you choose bet A.

  Depending on what you believe about H, it could well be that you prefer just one of the bets to both bets. But it seems like you really shouldn’t reject both bets. Taking both bets guarantees you’ll win exactly $5, which is strictly better than the $0 you’ll win if you reject both bets.

  But under imprecise probabilities, it’s rationally permissible to have some range of probabilities for H, which implies that it’s permissible to reject both bet A and bet B. So imprecise probabilities permit something which seems like it ought to be impermissible.

  Elga considers various rules that might be added to the initial imprecise probabilities-based decision theory, and argues that none of them are very appealing. I guess this isn’t as good as proving that there are no good rules or other modifications, but I found it fairly compelling on the face of it.

  The rules that seemed most likely to work to me were Plan and Sequence. Both rules more or less entail that you should accept bet B if you already accepted bet A, in which case rejecting both bets is impermissible and it looks like the theory is saved.

  Elga tries to show that these don’t work by inviting us to imagine the case where a particular agent called Sally faces the decision problem. Sally has imprecise probabilities, maximises expected utility and has a utility function that is linear in dollars.

  Elga argues that in this scenario it just doesn’t make sense for Sally to accept bet B only if she already accepted bet A—the decision to accept bet B shouldn’t depend on anything that came before. It might do if Sally had some risk averse decision theory, or had a utility function that was concave in dollars—but by assumption, she doesn’t. So Plan and Sequence, which had seemed like the best candidates for rescuing imprecise probabilities, aren’t plausible rules for a rational agent like Sally.

  Should Subjective Probabilities be Sharp?

  The 2014 paper by Bradley and Steele, Should Subjective Probabilities be Sharp? is, as the name suggests, a response to Elga’s paper. The core of their argument is that the assumptions for rationality implied by Elga’s argument are too strong and that it’s perfectly possible to have rational choice with imprecise probabilities provided that you don’t make these too-strong assumptions.

  I’ll highlight two objections and give my view.

  Objection 1:

  • Bradley and Steele give the label Retrospective Rationality to the idea that an agent’s sequence of decisions should not be dominated by another sequence the agent could have made. They seem to reject Retrospective Rationality as a constraint on rational decision making because “[it] is useless to an agent who is wondering what to do… [the agent] should be concerned to make the best decision possible at [the time of the decision]”.

  • My view: I don’t find this a very compelling argument, at least in the current context—it seems to me that the agent should avoid foreseeably violating Retrospective Rationality, and in Elga’s betting scenario the irrationality of the “reject both bets” sequence of decisions seems perfectly foreseeable.

  Objection 2:

  • Their second objection is that Elga is wrong to think that your current decision about whether to accept bet B should be unaffected by whether you previously accepted or rejected bet A (they make a similar point regarding the decision to take bet A with vs without the knowledge that you’re about to be offered bet B).

  • My view: it’s true that, because in Elga’s betting scenario the outcomes of the bets are correlated, knowing whether or not you previously accepted bet A might well change your inclination to accept bet B, e.g. because of risk aversion or a non-linear utility function. But to me it seems right that for an agent whose decision theory doesn’t include these features, it would be irrational to change their inclination to accept bet B based on what came before—and Elga was considering such an agent. So I think I side with Elga here.

  Summary and some thoughts

  In summary, in Subjective Probabilities should be Sharp, Elga illustrates how imprecise probabilities appear to permit a risk-neutral agent with linear utility to make irrational choices. In addition, Elga argues that there aren’t any ways to rescue things while keeping imprecise probabilities. In Should Subjective Probabilities be Sharp?, Bradley and Steele argue that Elga makes some implausibly strong assumptions about what it takes to be rational. I didn’t find these arguments very convincing, although I might well have just failed to appreciate the points they were trying to make.

  I think it basically comes down to this: for an agent with decision theory features like Sally’s, i.e. no risk aversion and linear utility, the only way to avoid passing up opportunities like making a risk-free $5 by taking bet A and bet B is if you’re always willing to take one side of any particular bet. The problem with imprecise probabilities is that they permit you to refrain from taking either side, which implies that you’re permitted to decline the risk-free $5.

  The fan of imprecise probabilities can wriggle out of this by saying that you should be allowed to do things like taking bet B only if you just took bet A—but I agree with Elga that this just doesn’t make sense for an agent like Sally. I think the reason this might look overly demanding on the face of it is that we’re not like Sally—we’re risk averse and have concave utility. But agents who are risk averse or have concave utility are allowed both to sometimes decline bets and to take risk-free sequences of bets, even according to Elga’s rationality requirements, so I don’t think this intuition pushes against Elga’s rationality requirements.

  It feels kind of useful to have read these papers, because

  • I’ve been kind of aware of imprecise probabilities and had a feeling I should think about them, and this has given me a bit of a feel for what they’re about.

  • It makes further reading in this area easier.

  • It’s good to get an idea of what sort of considerations people think about when deciding whether a decision theory is a good one. Similarly to when I dug more into moral philosophy, I now have more of a feeling along the lines of “there’s a lot of room for disagreement about what makes a good decision theory”.

  • Relatedly, it’s good to get a bit of a feeling of “there’s nothing really revolutionary or groundbreaking here and I should to some extent feel free to do what I want”.

  What links here?

  • Fore­cast­ing Newslet­ter: May 2021 by NunoSempere (Jun 1, 2021, 3:51 PM; 23 points)
  • Fore­cast­ing Newslet­ter: May 2021 by NunoSempere (LessWrong; Jun 1, 2021, 3:51 PM; 11 points)