Do you have any recommendations for doing threat modeling well? In particular, resources that seem applicable to many different risks—nuclear, AI, bio.
PatrickL
Karma: 446
Good find, thanks! I’m not very keen on instructing teams to run bug bounties and not other mechanisms, so am not particularly enthusiastic about this.
It looks like this would focus on infosecurity of the AI systems being used (i.e. can this weapon’s AI be hacked?) rather than testing for potential vulnerabilities from the AI systems themself.
How unusual does he think the current policy interest in AI safety is? Will this be a temporary window or an ever-increasing level of interest?
Yeah good find, I also think that passes the bar. Although I do think people have generally overestimated GPT’s essay-writing ability compared to humans, and think I might be falling for that here.
I’m not planning to change the doc because Bing’s AI wasn’t released by Feb 23, but if you think it should be included (which would be reasonable given OpenAI pretty obviously made this before Feb 23), it would mean:Experts expected 9 milestones to be met vs actually 11 milestones
The calibration curve looks four percentage points worse at the 10% mark
Bulls’ Brier score: 0.29
Experts’ Brier score: 0.24
Bears’ Brier score: 0.29
I’ve added it to this tracker of milestones (feel free to request edit access).
The experts thought beating humans at Angry Birds would be relatively easy—they put a 90% chance of it being feasible by now. The surprise is that it has not been done—although I think this is mostly explained by no labs seriously trying it.
Hmm I disagree on the numbers—have I got something wrong in the below?
If you assigned 0% or 100% by coin flip, you would get a Brier score of 0.5 (half the time you would get 0, half the time you would get 1), and if you assigned a random probability between 0% and 100% for every question, you would get a Brier score of 0.33. If you put 50% on everything you would indeed get 0.25.
As the experts had to give 10%, 50%, and 90% forecasts, if they had done this at random they would have ended up with a score of 0.36 [1].
So I think they—including the bullish and bearish groups—still did a fair bit better than random, which would be 0.36 in this context. And all simulated groups did better than the ‘randomized’ group which got a Brier score of 0.31 in my randomization. This does seem like worthwhile context to add though.- ^
[(1-0.1)^2 + (0-0.1)^2 + (1-0.5)^2 + (0-0.5)^2 + (1-0.9)^2 + (1-0.1)^2] / 6
- ^
I was also looking at https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Data (the dataset itself; the summary doesn’t include the milestones). This new version seems like total garbage. The experts continue to predict several of the milestones are five years out, including milestones that were achieved by ChatGPT (ie a few months after the survey) and at least one milestone that had already clearly been achieved by the time the survey was released!
I’ve only given the data a quick look and found it hard to analyse—but yeah, many of the forecasts look bad. But some of the medians (I think- from eyeballing data) seem not terrible—the ‘create top forty song’ shifted from 10 year median to ~5 year median. The ‘answer open-ended questions’ shifted from 10 year median to ~3 years.
But like you say, for many of the milestones I resolved as being met before this survey went out, they still have medians >0 years from now so—if I’m right in my judgements—the experts seem pretty poorly clued up on recent developments across the field.
This is great—thanks for this comment! I’ve gone through each to explain my reasoning. Your comments/sources changed my opinion on Starcraft and Explain—I’ve updated the post and scores to reflect this, and think the conclusion is now the same but slightly weaker, because the experts’ Brier score is 0.2 points worse, but the comparative Brier scores are also worse to a similar amount. There’s also my reasoning for other milestones in the appendix (and I’ve copy-pasted some of them below).
Zach Stein-Perlman from AI Impacts said that he thought “efficiently sort very large lists” and “write good Python code” were false, because the questions said it had to be done in a certain way by a certain type of neural net, and that wasn’t how it was done.
On “efficiently sort very large lists”, I think Zach ended up deferring to me on this. But no one has red-teamed my thinking on this. Here’s my source:
“We compared this algorithm against common sorting approaches and measured its performance for up to 1 billion normally-distributed double-precision keys. The results show that our approach yields an average 3.38x performance improvement over C++ STL sort, which is an optimized Quicksort hybrid, 1.49x improvement over sequential Radix Sort, and 5.54x improvement over a C++ implementation of Timsort, which is the default sorting function for Java and Python.”
On “write good Python code”, Zach and I (at least initially) disagreed. The criteria specified
Suppose the system is given only:A specification of what counts as a sorted list
Several examples of lists undergoing sorting by quicksort
The concern was that LLMs like Codex are trained on open source code, including examples of quicksort algorithms. So they are given more than just ‘a specification of what counts’ and ‘examples of lists undergoing sorting’. I’m unsure, but I think ‘given’ in the requirement refers to the goal specification post-training. Otherwise, the wording should be ‘suppose the system is trained only on...’ Therefore, Codex does count as only being given the stated requirements (or less).
For example, the ‘Starcraft’ milestone uses the wording ‘Beat the best human Starcraft 2 players at least 50% of the time, given a video of the screen’. For this milestone, I think the AI can be trained on more than the video of the screen, it just can’t use more than this while playing.
But could definitely be convinced otherwise on this, I might be making a conceptual misunderstanding. And I haven’t given Zach a chance to review my argument above.I was planning to count “transcribe as well as humans” as false, based on https://interactiveaimag.org/columns/is-ai-at-human-parity-yet-a-case-study-on-speech-recognition/ . Maybe the top labs could achieve this with a year of work, but I think the question specifies they need to do as well as the best human transcriptionists, and right now they don’t seem close.
The question specifies the AI needs to do “as well as the typical human”—I agree AI is not quite as good as the best human transcriptionists, but think it is better than the typical human.
OpenAI says Whisper “approaches human level robustness and accuracy on English speech recognition.” More detail is given in page 10 of the OpenAI report, which concludes “Whisper’s English ASR performance is not perfect but very close to human-level accuracy”.
However, this is when compared to “professional human transcribers”. Given Whisper sometimes performs better than professional human transcribers, and always performs to a similar level it seems very likely that Whisper regularly does “as well as a typical human”. I think Lorenzo’s analysis below seems reasonable too.
I counted “translate as well as bilingual humans” as true based on a few quick tests of ChatGPT; I’m curious if you have some specific source for why it’s false.
I have questionable anecdata! I asked two friends who are fluent in two languages but not skilled in translation, and one who has done a university module in translation, and all three thought they were better than AI at translating web pages (I think all three seem modest in general), although AI does a fine job for most uses.
This article notes the worse performance from AI when interpreting context and cultural nuances and says that “Le Monde explains that “the team (of its English language edition) is composed of eight journalists (…) responsible for selecting articles for translation, editing the translated versions and managing the English home page of the site. The translation is done by international agencies with the help of an artificial intelligence tool. Native English-speaking journalists do the selection and editing of the articles”.”Not-skilled-in-translation humans are still paid to translate writing between English and Chinese (I know of one organisation that does this and the CEO chooses to use AI to assist but not replace human translators, where the human translators are bilingual students not trained in translation). It does seem possible that this is just business lagging behind though.
I don’t think AI has won at Starcraft. The last word I’ve heard from this was https://www.extremetech.com/extreme/301325-deepminds-starcraft-ii-ai-can-now-defeat-99-8-percent-of-human-players , where AlphaStar could beat 99.8% of humans but not the absolute champions. I haven’t seen any further progress on this since 2019. Again, it’s possible that a year of concerted effort could change this, but that seems speculative. See also https://www.reddit.com/r/starcraft/comments/uakohx/why_cant_we_make_a_perfect_ai_for_starcraft/
I’m convinced by your argument/sources here, and this reddit thread. It does seem open to interpretation if “the best human players” refers to pros (which it did beat >50% of the time), or the literal very best player—but I think this should probably be interpreted as the very best. Given it reached a top 20 score but only beat Serral 2⁄3 times when heavy restrictions were placed on Serral, I’ve updated this to false.
I’m surprised you judged “high marks for a high school essay” as false; this seems like a central use case for ChatGPT and Bing/GPT4.
I was also surprised by this. Others disagreed with me and we discussed a bit more in this LW comment thread. Here’s my reasoning:
I couldn’t find this done and think, by now, someone would have submitted a fully ChatGPT-generated high school essay and talked about it publicly if it had gotten high marks. I’ve seen some evidence of cherry-picking paragraphs leading to a mid/low-level, e.g. this article describes someone who got a passing mark (53) on a university social policy essay.This high school teacher judged two ChatGPT-generated history essays as “below average, scoring a 9⁄20 or lower”. This Guardian article says, uncited, that ‘academics have generated responses to exam queries that they say would result in full marks if submitted by an undergraduate’. I think, if this claim were true, there would be more evidence.
I was planning to judge “concisely explain game play” as true, based on https://www.forbes.com/sites/carlieporterfield/2022/11/22/metas-ai-gamer-beat-humans-in-diplomacy-using-strategy-and-negotiation/, which is testing basically this skill. Also, I was able to play a partial game of chess with ChatGPT where it explained all its moves—before it started hallucinating and making moves which were impossible. Still, it seemed to have the “explanation” skill down pat! I imagine if you asked it to explain why a chess engine made a given move, it would give a pretty plausible answer.
Good point! This is an update for me—I wouldn’t have guessed ChatGPT could explain chess moves. The question specifies “For any computer game that can be played well by a machine”. I just played Scrabble with ChatGPT and it explained its moves well. And I asked ChatGPT to imagine it was watching Agent57 play Montezuma’s revenge for 10 seconds and it gave a very clear summary of what it imagined Agent57 was doing and why.
So I’m just about convinced enough that it could do this for all games—and have changed to True.
Thanks Adam :)
I have a rough (i.e. considered for <15 minutes) take: if top labs one year ago had attempted these particular milestones, and had the same policies on disclosing capabilities as they currently seem to, then there’s a 40-50% chance they would have achieved 2 of Angry Birds, Atari fifty , Laundry and Go low by now. But I don’t put much weight on my prediction, whereas I put a lot more weight on my analysis of what has happened (though this is also somewhat subjective!).
I agree though that checking what has actually happened ends up being more conservative than the original “Ignore the question of whether they would choose to” , which makes the expert forecasts worse than they seem to be here. This is a weakness of this analysis! And of the resolvability of the original survey.
Do you have an estimate of how many of the tasks would have been achieved by now if labs tried a year ago?
I’d be interested in seeing this too! Although I’m not planning to spend the time to do this soon, I’d be up for having a quick chat if someone else was up for it. The constant risk model seems to me like a better-than-nothing model, so could be a slight update if it gives different results. Or could just use a linear increase of probability from 0 years to 10 years.
FWIW—from this survey alone, I’m not convinced that timelines are systematically longer with a fixed-year framing. I sampled ten forecasts where probabilities were given in both framings on a 10 year timescale, and five of them (Subtitles, Transcribe, Top forty, Random game, Explain) gave later forecasts when asked with ‘probability in N years’ rather than ‘year that the probability is M’, three of them (Video scene, Read aloud, Atari) gave the same forecasts, and two of them (Rosetta, Taylor) gave an earlier forecast. So from this sample, it seems they’re commonly longer.
I feel like I want to comment, for people that know me, that this wasn’t me (same first name, the auhtor’s description could probably be me, it’s the type of thing I’d say (this isn’t me necessarily endorsing the post)).
Thanks for posting this. I thought it was well-considered with lots of good brainstroming of options for valuable policy/system change, so am happy you’re joining the community! I think 3 is a particularly interesting idea. I think the EA community is far too large and independent to get to 80% agreement of EAs endorsing something, but there may be some way of surveying EAs in relevant fields for each of your policy proposals. It seems probably valuable to me to get politically-interested EAs to have in-depth knowledge on certain areas, although I think this could be broader than your suggestion of specific policies, and could be more like ‘lean into this area because it seems possible to have extra-large positive impact to others’ rather than going in with the aim of advocating for a specific policy. In my opinion, I’d probably endorse the main areas being those listed by 80k here—but seems like there’s room for these to be challenged by new particularly policy-relevant areas.
I recommend this post about US policy careers and this post about longterm sustainability and growth in developing countries which argue some similar things and some different things to you!
I would probably add animal welfare too to your list of issues most commonly championed in EA (but note there are plenty more causes that various people within EA are passionate about).
I noticed Ali Abdaal referencing some of the examples from this intro in his summary of WWOTF :)
07:50 and 10:45
I would reframe this as ‘community builders don’t spend enough time on other EA things’ rather than ‘community builders spend too much time community building’. I know the thought is that less time marketing → more time on other stuff, but I think it’s worth setting a different tone. I worry there’s too much of a meme going round like ‘community building is not that good a thing to do’, where it should be ‘we are finding ways community building could be done better—exciting!’
To me, this is a great post for suggesting how community building could be done better through more direct experience—exciting!
I would be very happy to talk to people! I have been to EAG London three times.
Once, I didn’t know anyone and was too shy to reach out to anyone so floated round different talks. Once, I volunteered on reception and also got in some good conversations and workshops. Once, I spent most of the weekend in individual conversations—coming out inspired and exhausted!
Very happy to talk to anyone if you message me :) I also used to work in the UK’s foreign office and department for international development and am now studying AI. I think I’m nice and approachable!
I liked this review—useful tips I will consider if facilitating again!
I’ve found the connections/friendships formed the most valuable part—which would mean a strong +1 for your suggestion of games/debates. In my version of the usefulness list, the personal connections would probably go number one—particularly those nearby that you can eventually meet in person.
Also, I get the impression from this you’d be a great facilitator!
I listened and come away with the same feeling as I commented above- IMO Rory is being a good ambassador for GiveDirectly here!
Also, I was excited about this because I thought Rory Stewart was the new Comms Director at No. 10, which I’ve just realised was an April Fools prank...
Thanks! Yes this was just my impression from reading, not listening. I’ll hopefully get round to listening later and see if that updates my impression.
In 2015, it seems to be ~2% (£200m/£12bn). This was general support for cash transfer schemes which included other features though, like nutritional support. Seems very high though still! Can’t see anything more recent—my naive guess would be its less than this now.
Link on UK spend 2011-15 on cash transfers.
neglectedness of the latter outweighing increased tractability of the former