But the point stands that if you have more âmedium-sizedâ probabilities on those capabilities being available (as opposed to very high or very low ones), then a sane response to AGI should explicitly grapple with that, not pretend the probability is negligible because itâs scary.
I do think debates between the ârisk is extremely highâ camp and the ârisk is medium-sizedâ camp are important. But the importance mostly stems from âthis suggests we have different background models, and should try to draw those out so they can be discussed explicitlyâ, not âwe should only take action about extreme risks once weâre 95+% sure of themâ.
Nate Silver said there was a 29% chance Trump would win. Most people interpreted that as âTrump probably wonât winâ and got shocked when he did. What was the percent attached to your âcoronavirus probably wonât be a disasterâ prediction? Was it also 29%? 20%? 10%? Are you sure you want to go lower than 10%? Wuhan was already under total lockdown, they didnât even have space to bury all the bodies, and youâre saying that there was less than 10% odds that it would be a problem anywhere else? I hear people say thereâs a 12 â 15% chance that future civilizations will resurrect your frozen brain, surely the risk of coronavirus was higher than that?
And if the risk was 10%, shouldnât that have been the headline. âTEN PERCENT CHANCE THAT THERE IS ABOUT TO BE A PANDEMIC THAT DEVASTATES THE GLOBAL ECONOMY, KILLS HUNDREDS OF THOUSANDS OF PEOPLE, AND PREVENTS YOU FROM LEAVING YOUR HOUSE FOR MONTHSâ? Isnât that a better headline than Coronavirus panic sells as alarmist information spreads on social media? But thatâs the headline you could have written if your odds were ten percent!
Nice post!
In other words, a failure, but not of prediction: