Research analyst at Redwood Research. All opinions are my own.
Lukas Finnveden
Karma: 1,803
AGI and Lock-In
We used the geometric mean of the samples with the minimum and maximum removed to better deal with extreme outliers, as described in our previous post
I don’t see how that’s consistent with:
What is the probability that Russia will use a nuclear weapon in Ukraine in the next MONTH?
Aggregate probability: 0.0859 (8.6%)
All probabilities: 0.27, 0.04, 0.02, 0.001, 0.09, 0.08, 0.07
What is the probability that Russia will use a nuclear weapon in Ukraine in the next YEAR?
Aggregate probability: 0.2294 (23%)
All probabilities: 0.38, 0.11, 0.11, 0.005, 0.42, 0.2, 0.11
I get that the first of those should be 0.053. Haven’t run the numbers on the latter, but pretty sure the geometric mean should be smaller than 23% from eyeballing it. (I also haven’t run the numbers on other aggregated numbers in this post.)
On the other hand, the critic updated me towards higher numbers on p(nuke london|any nuke). Though I assume Samotsvety have already read it, so not sure how to take that into account. But given that uncertainty, given that that number only comes into play in confusing worlds where everyone’s models are broken, and given Samotsvety’s 5x higher unconditional number, I will update at least a bit in that direction.
Thanks for the links! (Fyi the first two points to the same page.)
The critic’s 0.3 assumes that you’ll stay until there’s nuclear exchanges between Russia and NATO. Zvi was at 75% if you leave as soon as a conventional war between NATO and Russia starts.
I’m not sure how to compare that situation with the current situation, where it seems more likely that the next escalatory step will be a nuke on a non-NATO target than conventional NATO-Russia warfare. But if you’re happy to leave as soon as either a nuke is dropped anywhere or conventional NATO/Russia warfare breaks out, I’m inclined to aggregate those numbers to something closer to 75% than 50%.
Thanks for doing this!
In this squiggle you use “ableToEscapeBefore = 0.5”. Does that assume that you’re following the policy “escape if you see any tactical nuclear weapons being used in Ukraine”? (Which someone who’s currently on the fence about escaping London would presumably do.)
If yes, I would have expected it to be higher than 50%. Do you think very rapid escalation is likely, or am I missing something else?
I think this particular example requires an assumption of logarithmically diminishing returns, but is right with that.
(I think the point about roughly quadratic value of information applies more broadly than just for logarithmically diminishing returns. And I hadn’t realised it before. Seems important + underappreciated!)
One quirk to note: If a funder (who I want to be well-informed) is 50⁄50 on S vs L, but my all-things-considered belief is 60⁄40, then I would value the first 1% they shift towards my position much more than they do (maybe 10x more?) and will put comparatively little value on shifting them all the way (ie the last percent from 59% to 60% is much less important). You can get this from a pretty similar argument as in the above example.
(In fact, the funder’s own much greater valuation of shifting 10% than 1% can be seen as a two-step process where (i) they shift to 60⁄40 beliefs, and then (ii) they first get a lot of value from shifting their allocation from 50 to 51, then slightly less from shifting from 51 to 52, etc...)
I think that’s right other than that weak upvotes never become worth 3 points anymore (although this doesn’t matter on the EA forum, given that no one has 25,000 karma), based on this lesswrong github file linked from the LW FAQ.
Nitpicking:
A property of making directional claims like this is that MacAskill always has 50% confidence in the claim I’m making, since I’m claiming that his best-guess estimate is too high/low.
This isn’t quite right. Conservation of expected evidence means that MacAskill’s current probabilities should match his expectation of the ideal reasoning process. But for probabilities close to 0, this would typically imply that he assigns higher probability to being too high than to being too low. For example: a 3% probability is compatible with 90% probability that the ideal reasoning process would assign probability ~0% and a 10% probability that it would assign 30%. (Related.)
This is especially relevant when the ideal reasoning process is something as competent as 100 people for 1000 years. Those people could make a lot of progress on the important questions (including e.g. themselves working on the relevant research agendas just to predict whether they’ll succeed), so it would be unsurprising for them to end up much closer to 0% or 100% than is justifiable today.
The term “most important century” pretty directly suggests that this century is unique, and I assume that includes its unusually large amount of x-risk (given that Holden seems to think that the development of TAI is both the biggest source of x-risk this century and the reason for why this might be the most important century).
Holden also talks specifically about lock-in, which is one way the time of perils could end.
See e.g. here:
It’s possible, for reasons outlined here, that whatever the main force in world events is (perhaps digital people, misaligned AI, or something else) will create highly stable civilizations with “locked in” values, which populate our entire galaxy for billions of years to come.
If enough of that “locking in” happens this century, that could make it the most important century of all time for all intelligent life in our galaxy.
I want to roughly say that if something like PASTA is developed this century, it has at least a 25% chance of being the “most important century” in the above sense.
The page for the Century Fellowship outlines some things that fellows could do, which are much broader than just university group organizing:
When assessing applications, we will primarily be evaluating the candidate rather than their planned activities, but we imagine a hypothetical Century Fellow may want to:
Lead or support student groups relevant to improving the long-term future at top universities
Develop a research agenda aimed at solving difficult technical problems in advanced deep learning models
Start an organization that teaches critical thinking skills to talented young people
Run an international contest for tools that let us trace where synthetic biological agents were first engineered
Conduct research on questions that could help us understand how to to make the future go better
Establish a publishing company that makes it easier for authors to print and distribute books on important topics
Partly this comment exists just to give readers a better impression of the range of things that the century fellowship could be used for. For example, as far as I can tell, the fellowship is currently one of very few options for people who want to pursue fairly independent longtermist research and who want help with getting work authorization in the UK or US.
But I’m also curious if you have any comments on the extent to which you expect the century fellowship to take on community organizers vs researchers vs ~entrepeneurs. (Is the focus on community organizing in this post indicative, or just a consequence of the century fellowship being mentioned in a post that’s otherwise about community organizing?)
I’m not saying it’s infinite, just that (even assuming it’s finite) I assign non-0 probability to different possible finite numbers in a fashion such that the expected value is infinite. (Just like the expected value of an infinite st petersburg challenge is infinite, although every outcome has finite size.)
The topic under discussion is whether pascalian scenarios are a problem for utilitarianism, so we do need to take pascalian scenarios seriously, in this discussion.
I simply don’t believe that infinities exist, and even though 0 isn’t a probability, I reject the probabilistic argument that any possibility of infinity allows them to dominate all EV calculations.
Problems with infinity doesn’t go away just because you assume that actual infinities don’t exist. Even with just finite numbers, you can face gambles that have infinite expected value, if increasingly good possibilities have insufficiently rapidly diminishing probabilities. And this still causes a lot of problems.
(I also don’t think that’s an esoteric possibility. I think that’s the epistemic situation we’re currently in, e.g. with respect to the amount of possible lives that could be created in the future.)
Also, as far as I know (which isn’t a super strong guarantee) every nice theorem that shows that it’s good to maximize expected value assumes that possible utility is bounded in both directions (for outcomes with probability >0). So there’s no really strong reason to think that it would make sense to maximize expected welfare in an unbounded way, in the first place.
See also: www.lesswrong.com/posts/hbmsW2k9DxED5Z4eJ/impossibility-results-for-unbounded-utilities
10^12 might be too low. Making up some numbers: If future civilizations can create 10^50 lives, and we think there’s an 0.1% chance that 0.01% of that will be spent on ancestor simulations, then that’s 10^43 expected lives in ancestor simulations. If each such simulation uses 10^12 lives worth of compute, that’s a 10^31 multiplier on short-term helping.
A proper treatment of this should take into account that short-term helping also might have positive effects in lots of simulations to a much greater extent than long-term helping. https://longtermrisk.org/how-the-simulation-argument-dampens-future-fanaticism
I agree. Anecdotally, among people I know, I’ve found aphantasia to be more common among those who are very mathematically skilled.
(Maybe you could have some hypothesis that aphantasia tracks something slightly different than other variance in visual reasoning. But regardless, it sure seems similar enough that it’s a bad idea to emphasize the importance of “shape rotating”. Because that will turn off some excellent fits.)
But note the hidden costs. Climbing the social ladder can trade of against building things. Learning all the Berkeley vibes can trade of against, eg., learning the math actually useful for understanding agency.
This feels like a surprisingly generic counterargument, after the (interesting) point about ladder climbing. “This could have opportunity costs” could be written under every piece of advice for how to spend time.
In fact, it applies less to this posts than to most advice on how to spend time, since the OP claimed that the environment caused them to work harder.
(A hidden cost that’s more tied to ladder climbing is Chana’s point that some of this can be at least somewhat zero-sum.)
By the way, as an aside, the final chapter here is that Protect our Future PAC went negative in May—perhaps a direct counter to BoldPAC’s spending. (Are folks here proud of that? Is misleading negative campaigning compatible with EA values?)
I wanted to see exactly how misleading these were. I found this example of an attack ad, which (after some searching) I think cites this, this, this, and this. As far as I can tell:
The first source says that Salinas “worked for the chemical manufacturers’ trade association for a year”, in the 90s.
The second source says that she was a “lobbyist for powerful public employee unions SEIU Local 503 and AFSCME Council 75 and other left-leaning groups” around 2013-2014. The video uses this as a citation for the slide “Andrea Salinas — Drug Company Lobbyist”.
The third source says that insurers’ drug costs rose by 23% between 2013-2014. (Doesn’t mention Salinas.)
The fourth source is just the total list of contributors to Salina’s campaigns, and the video doesn’t say what company she supposedly lobbied for that gave her money. The best I can find is that this page says she lobbied for Express Scripts in 2014, who is listed as giving her $250.
So my impression is that the situation boils down to: Salinas worked for a year for the chemical manufacturers’ trade association in the 90s, had Express Scripts as 1 out of 11 clients in 2014 (although the video doesn’t say they mean Express Scripts, or provide any citation for the claim that Salinas was a drug lobbyist in 2013/2014), and Express Scripts gave her $250 in 2018. (And presumably enough other donors can be categorised as pharmaceutical to add up to $18k.)
So yeah, very misleading.
(Also, what’s up with companies giving and campaigns accepting such tiny amounts as $250? Surely that’s net-negative for campaigns by enabling accusations like this.)
(1) maybe doom should be disambiguated between “the short-lived simulation that I am in is turned of”-doom (which I can’t really observe) and “the basement reality Earth I am in is turned into paperclips by an unaligned AGI”-type doom.
Yup, I agree the disambiguation is good. In aliens-context, it’s even useful to disambiguate those types of doom from “Intelligence never leaves the basement reality Earth I am on”-doom. Since paperclippers probably would become grabby.
And tags / wiki entries.