Senior Research Scientist at NTT Research, Physics & Informatics Lab. jessriedel.com , jessriedel[at]gmail[dot]com
Jess_Riedel
Karma: 343
TAI Safety Bibliographic Database
Were there a lot of new unknown or underappreciated facts in this book? From the summary, it sounds mostly like a reinterpretation of the standard history, which hinges on questions of historical determinism.
I am on the whole positive about this idea. Obviously, specialization is good, and creating dedicated fund managers to make donation decisions can be very beneficial. And it makes sense that the boundaries between these funds arise from normative differences between donors, while putting fund managers in charge of sorting out empirical questions about efficiency. This is just the natural extension, of the original GiveWell concept, to account for normative differences, and also to utilize some of the extra trust that some EAs will have for other people in the community that isn’t shared by a lot of GiveWell’s audience.
That said, I’m worried about principle-agent problems and transparency, and about CEA becoming an organization receiving monthly direct debits from the bank accounts of ten thousand people. Even if we assume that current CEA employees are incorruptible superhuman angels, giving CEA direct control of a firehose of cash makes it an attractive target for usurpers (in a way that it is not when it’s merely making recommendations and doing outreach). These sorts of worries apply much less to GiveWell when it’s donating to developing-world health charities than to CEA when it’s donating to EA start-ups who are good friends with the staff.
Will EA Fund managers be committed to producing the sorts of detailed explanations and justifications we see from GiveWell and Open Phil, at least after adjusting for donation size? How will the conflicts of interest be managed and documented with such a tightly interlinked community?
What sorts of additional precautions will be taken to manage these risks, especially for the long term?
I’m still fuzzy on the relationship between the EA Facebook group and the EA forum. Are we supposed to move most or all the discussion that was going on in the FB group here? Will the FB be shut down, and if not what will is be used for?
I think the format of the forum will present a higher barrier to low-key discussion than the FB group, e.g. I’d guess people are much less likely to post an EA related new article if they don’t have too much to add to it. This is primarily because the forum looks like a blog. Is FB style posting encouraged?
If this has all been described somewhere. Could someone point me toward it?
Also, what’s the relationship between the EA forum and the EA hub? http://effectivealtruismhub.com/
I was curious about the origins of this concept in the EA community since I think it’s correct, insightful, and I personally had first noticed it in conversation among people at Open Phil. On Twitter, @alter_ego_42 pointed out the existence of the Credal Resilience page in the “EA concepts” section of this website. That page cites
Skyrms, Brian. 1977. Resiliency, propensities, and causal necessity. The journal of philosophy 74(11): 704-713. [PDF]
which is the earliest thorough academic reference to this idea that I know of. With apologies to Greg, this seems like the appropriate place to post a couple comments on that paper so others don’t have to trudge through it.
I didn’t find Skyrms’s critique of frequentism at the beginning, or his pseudo-formalization of resilency on page 705 (see for instance the criticism “Some Remarks on the Concept of Resiliency” by Patrick Suppes in the very next article, pages 713-714), to be very insightful, so I recommend the time-pressed reader concentrate on
The bottom of p. 705 (“The concept of probabilistic resiliency is nicely illustrated...”) to the top of p. 708 (”… well confirmed to its degree of instantial resiliency, as specified above..”).
The middle of p. 712 (“The concept of resiliency has connections with...”) to p. 713 (the end).
Skyrms quotes Savage (1954) as musing about the possibility of introducing “second-order probabilities”. This is grounded in a relative-frequency intuition: when I say that there is a (first-order) probability p of X occurring but that I am uncertain, what I really mean is something like that there is some objective physical process that generates X with (second-order) probability q, but I am uncertain about the details of that process (i.e., about what q is), so my value of p is obtained by integrating over some pdf f (q).
There is, naturally, a Bayesian version of the same idea: We shouldn’t concern ourselves with a hypothetical giant (second-order) ensemble of models, each of which generates a hypothetical (first-order) ensemble of individual trials. Resilience about probabilities is best measured by our bets on how future evidence would change those probabilities, just as probabilities is best measured by our bets on future outcomes.
(Unfortunately, and unlike the case for standard credences, there seems to be multiple possible formulations depending on which sorts of evidence we are supposing: what I expect to learn in the actual future, what I could learn if I thought about it hard, what a superforecaster would say in my shoes, etc.)
Thanks Jacob. That last link is broken for me, but I think you mean this?
You sort of acknowledge this already, but one bias in this list is that it’s very tilted towards large organizations like DeepMind, CHAI, etc.
Well, it’s biased toward safety organizations, not large organizations. (Indeed, it seems to be biased toward small safety organizations over larges ones since they tend to reply to our emails!) We get good coverage of small orgs like Ought, but you’re right we don’t have a way to easily track individual unaffiliated safety researchers and it’s not fair.
I look forward to a glorious future where this database is so well known that all safety authors naturally send us a link to their work when its released, but for now the best way we have of finding papers is (1) asking safety organizations for what they’ve produced and (2) taking references from review articles. If you can suggest another option for getting more comprehensive coverage per hour of work we’d be very interested to hear it (seriously!).
For what it’s worth, the papers by Hendrycks are very borderline based on our inclusion criteria, and in fact I think if I were classifying it today I think I would not include it. (Not because it’s not high quality work, but just because I think it still happens in a world where no research is motivated by the safety of transformative AI; maybe that’s wrong?) For now I’ve added the papers you mention by Hendrycks, Wong, and Cohen to the database, but my guess is they get dropped for being too near-term-motivated when they get reviewed next year.
More generally, let me mention that we do want to recognize great work, but our higher priority is to (1) recognize work that is particularly relevant to TAI safety and (2) help donors assess safety organizations.
Thanks again! I’m adding your 2019 review to the list.
for instance if you think Wong and Cohen should be dropped then about half of the DeepMind papers should be too since they’re on almost identical topics and some are even follow-ups to the Wong paper).
Yea, I’m saying I would drop most of those too.
I think focusing on motivation rather than results can also lead to problems, and perhaps contributes to organization bias (by relying on branding to asses motivation).
I agree this can contribute to organizational bias.
I do agree that counterfactual impact is a good metric, i.e. you should be less excited about a paper that was likely to soon happen anyways; maybe that’s what you’re saying? But that doesn’t have much to do with motivation.
Just to be clear: I’m using “motivation” here in the technical sense of “What distinguishes this topic for further examination out of the space of all possible topics?”, i.e., is the topic unusually likely to lead to TAI safety results down the line?” (It’s not anything to do with the author’s altruism or whatever.)
I think what would best advance this conversation would be for you to propose alternative practical inclusion criteria which could be contrasted the ones we’ve given.
Here’s how is how I arrived at ours. The initial desiderata are:
-
Criteria are not based on the importance/quality of the paper. (Too hard for us to assess.)
-
Papers that are explicitly about TAI safety are included.
-
Papers are not automatically included merely for being relevant to TAI safety. (There are way too many.)
-
Criteria don’t exclude papers merely for failure to mention TAI safety explicitly. (We want to find and support researchers working in institutions where that would be considered too weird.)
(The only desiderata that we could potentially drop are #2 or #4. #1 and #3 are absolutely crucial for keeping the workload manageable.)
So besides papers explicitly about TAI safety, what else can we include given the fact that we can’t include everything relevant to safety? Papers that TAI safety researchers are unusually likely (relative to other researchers) to want to read, and papers that TAI safety donors will want to fund. To me, that means the papers that are building toward TAI safety results more than most papers are. That’s what I’m trying to get across by “motivated”.
Perhaps that is still too vague. I’m very in your alternative suggestions!
-
I like this post a lot but I will disobey Rapoport’s rules and dive straight into criticism.
Historically, many AI researchers believed that creating general AI would be more about coming up with the right theories of intelligence, but over and over again, researchers eventually found that impressive results only came after the price of computing fell far enough that simple, “blind” techniques began working (Sutton 2019).
I think this is a poor way to describe a reasonable underlying point. Heavier-than-air flying machines were pursued for centuries, but airplanes appeared almost instantly (on a historic scale) after the development of engines with sufficient power density. Nonetheless, it would be confusing to say “flying is more about engine power than the right theories of flight”. Both are required. Indeed, although the Wright brothers were enabled by the arrival of powerful engines, they beat out other would-be inventors (Ader, Maxim, and Langley) who emphasized engine power over flight theory. So a better version of your claim has to be something like “compute quantity drives algorithmic ability; if we independently vary compute (e.g., imagine an exogenous shock) then algorithms follow along”, which (I think) is what you arguing further in the post.
But this also doesn’t seem right. As you observe, algorithmic progress has been comparable to compute progress (both within and outside of AI). You list three “main explanations” for where algorithmic progress ultimately comes from and observe that only two of them explain the similar rates of progress in algorithms and compute. But both of these draw a causal path from compute to algorithms without considering the (to-me-very-natural) explanation that some third thing is driving them both at a similar rate. There are a lot of options for this third thing! Researcher-to-researcher communication timescales, the growth rate of the economy, the individual learning rate of humans, new tech adoption speed, etc. It’s plausible to me that compute and algorithms are currently improving more or less as fast as they can, given their human intermediaries through one or all of these mechanisms.
The causal structure is key here, because the whole idea is to try and figure out when economic growth rates change, and the distinction I’m trying to draw becomes important exactly around the time that you are interested in: when the AI itself is substantially contributing to its own improvement. Because then those contributions could be flowing through at least three broad intermediaries: algorithms (the AI is writing its own code better), compute (the AI improves silicon lithography), or the wider economy (the AI creates useful products that generate money which can be poured into more compute and human researchers).
Of course, even if AI performance is, in principle, predictable as a function of scale, we lack data on how AIs are currently improving on the vast majority of tasks in the economy, hindering our ability to predict when AI will be widely deployed. While we hope this data will eventually become available, for now, if we want to predict important AI capabilities, we are forced to think about this problem from a more theoretical point of view.
Humans have been automating mechanical task for many centuries, and information-processing tasks for many decades. Moore’s law, the growth rate of the thing (compute) that you ague drives everything else, has been stated explicitly for almost 58 years (and presumably applicable for at least a few decade before that). Why are you drawing a distinction between all the information processing that happened in the past and “AI”, which you seem to be taking as a basket of things that have mostly not had a chance to be applied yet (so no data)?
If compute is the central driving force behind AI, and transformative AI (TAI) comes out of something looking like our current paradigm of deep learning, there appear to be a small set of natural parameters that can be used to estimate the arrival of TAI. These parameters are:
The total training compute required to train TAI
The average rate of growth in spending on the largest training runs, which plausibly hits a maximum value at some significant fraction of GWP
The average rate of increase in price-performance for computing hardware
The average rate of growth in algorithmic progress
This list is missing the crucial parameters that would translate the others into what we agree is most notable: economic growth. I think needs to be discussed much more in section 4 for it to be a useful summary/invitation to the models you mention.
Paul Graham writes that Noora Health is doing something like this.
https://twitter.com/Jess_Riedel/status/1389599895502278659
https://opensea.io/assets/0x495f947276749ce646f68ac8c248420045cb7b5e/96773753706640817147890456629920587151705670001482122310561805592519359070209