Ross Rheingans-Yoođ¸
The nice thing about the quadratic voting /â quadratic funding formula (and the reason that so many people are huge nerds about it) is that the optional diversification is really easy to state:
You should donate in a $X : $1 ratio of org A to org B if you believe that org A is X times as effective as org B (in their marginal use of funding).
One explanation for this is, if youâre donating $X to A and $1 to B, then adding one more cent to A increases Aâs total match by 1/âX the amount that B would get if you gave it to B. So the point where your marginal next-dollar is equal is the point where your funding /â votes are proportional to impact.
(I have a comment nephew to this one that argues against âpolitics doesnât belong hereâ, but I also wanted to provide a cautionary suggestion...)
I have found it pretty difficult to think in a balanced way about...any election in the last three cycles...but I want to propose that, on the outside view, âcandidate seems Obviously Badâ and âcandidate would have a negative counterfactual effect on [classical EA cause area]â is nowhere near as correlated as it intuitively feels it should be.
The example that I will keep pointing to, probably forever, is that George W. Bush was almost certainly the best president in the modern era for both Global Health and Welfare[1] and for GCBRs[2], based on programs that were very far from what I (still) think of as his major policy positions.
I think that Bushâs interest in HIV response in Africa was in theory knowable at the time[3], but figuring it out would have required digging into some pretty unlikely topics on a candidate that my would-have-been intellectual circles[4] was pretty strongly convinced was the worse one. (Iâm not sure how knowable his proactive interest in pandemic prep was.)
I donât want to claim that itâs correct to equivocate this cycleâs Republican candidate and W. Bush here, and I donât have any concrete reason to believe that the Republican candidate is good on particular cause areas. I just mean to say, I wouldnât have believed it of W. Bush, either. And in this cycle, Iâm not aware of anyone who has really done the reasearch that would convince me one way or another in terms of the shut-up-and-multiply expected counterfactual utility.
So, while I donât oppose making decisions on other-than-consequentialist and/âor commonsense grounds here (which is likely whatâs going to actually sway my ballot as a citizen), I want to argue for a stance of relatively deep epistemic uncertainty on the consequentialist dimension, until I see more focused argument from someone who really has done the homework.
- âŠď¸
In a word, PEPFAR.
- âŠď¸
The National Pharmaceutical Stockpile was founded under Clinton with $51mln of initial funding, but Bush increased the budget tenfold between the Project BioShield Act and PAHPA; expansion of the program since then has been small by comparison. Plus I think that the effect of PEPFAR on the âbiosecurity waterlineâ of the world is under-appreciated.
- âŠď¸
Wikipedia: âAccording to [Bushâs] 2010 memoir, Decision Points, [George W. and Laura Bush] developed a serious interest in improving the fate of the people of Africa after reading Alex Haleyâs Roots, and visiting The Gambia in 1990. In 1998, while pondering a run for the U.S. presidency, he discussed Africa with Condoleezza Rice, his future secretary of state; she said that, if elected, working more closely with countries on that continent should be a significant part of his foreign policy.â
- âŠď¸
I was too young to have âintellectual circlesâ during the GWB presidency; Iâm approximating myself by my parents here, though itâs conflated by EA, LessWrong, et al. not existing at the time.
- âŠď¸
I think that your argument is a fair one to make, but I think itâs easier to argue for than against, so I want to argue against to avoid an information cascade towards consensus.
I generally support the outside-view argument for ideological diversification, and if that diversification means anything it has to mean supporting things that wouldnât âget thereâ on their own (especially as a small /â exploratory fraction of donation portfolio, as OP indicates here).
EA need not be totalizing, and I think the world is better-off if EAs discuss how to bring a mindset towards effectiveness to other endeavors in their lives.
I generally think that weâve swung too far towards consensus and OpenPhil deference as a community (though thereâs been some swing back), and am actively happier to see things that arenât obviously bunk but swing us towards a more pluralistic and diverse set of approaches.
I think in particular that EA donors have historically under-considered opportunities in politics, and am happy to see increased engagement in considering opportunities there (even if I might disagree with the choice of a particular political race as the most effective, like I might disagree with the choice of an approach within a cause area).
Whatâs more, democratic capitalism + effective altruism will direct effort and resources to effective uses even if only a few capital-havers are unselfishly motivated in this way.
If socialism means the command-economy things, then democratic socialism + effective altruism doesnât reliably direct resources to causes that only a small minority are motivated to support.
Makes total sense not to invest in the charitable sideâIâm generally off a similar mind.[1] The reason Iâm curious is that âconsider it as two separate accountsâ is the most-compelling argument Iâve seen against tithing investment gains. (The argument is basically, that if both accounts were fully-invested, then tithing gains from the personal account to the charity account leads to a total 4:1 ratio between them as withdrawal_time â â, not a 9:1 ratio.[2] Then, why does distribution out of the charity account affect the ârightâ additional amount to give out of the personal account?)
Another way to count it is, if you believe that the returns on effective charity are greater than private investments returns and so always make donations asap, then tithing at the start and after years is worse for both accounts than just giving say up-front (and giving of the further investment gains).
Probably this is most relevant to startup employees, who might receive â$100,000 in equityâ that they only can sell when it later exits for, say, 10x that. Should a 10% pledge mean $10,000 up-front and $90,000 of the exit (10% when paid + 10% of gains), or just $100,000 of the exit (10% went to the charity account, then exited)?[3]
(Sorry, donât mean to jump on your personal post with this tangentâam happy to chat if you find this interesting to think about, but also can write my own post about it on my own time if not
- ^
The one case where I do think investment can sense is where I want to direct the funding to accelerating the program of a for-profit company, eg in biotech, and the right way to do so is via direct investment. I do think there are such cases that can be on the frontier of most-effective in EV terms (and for them I only count it as effective giving if I precommit to re-giving any proceeds, without re-counting it as a donation for pledge purposes).
- ^
Consider receiving $1,000 in salary, splitting it $100 : $900 between the accounts, investing each so they grow 10x and become $1,000 : $9,000, then realizing the personal investment gains and tithing $800 on them. Now the accounts are $1,800 : $8,200, which seems a lot more like âgiving 18%â than âgiving 10%â!
- ^
If the correct baseline is â10% of the exitâ, should this be any different from the case of a salary worker who makes the $100,000 in cash and puts it in an index fund until it [10x]s? Or what about a professional trader who ârealizes gainsâ frequently with daily trading, but doesnât take any of the money out until after many iterations?
- ^
-
Thought-provoking post; thanks for sharing!
-
A bit of a tangential point, but Iâm curious, because itâs something Iâve also considered:
putting 10% of my paycheck directly in a second account which was exclusively for charity
What do you do with investment income? Itâs pretty intuitive that if youâre âinvesting to giveâ and you have $9,000 of personal savings and $1,000 of donation-investments and they both go up 10% over a year, that you should have $9,900 of personal savings and $1,100 of donation-investments. But what would you (or do you) do differently if you put the money into the accounts, donated half of the charity account, and then ended up with $9,900 in personal savings (a $900 annual gain) and $550 in savings-for-giving (a $50 annual gain)?
I have heard at least three different suggestions for how to do this sort of accounting, but am curious what you go with, since the rest of your perspective self seems fairly intentional and considered!
-
Iâd argue that you need to use a point estimate to decide what bets to make, and that you should make that point estimate by (1) geomean-pooling raw estimates of parameters, (2) reasoning over distributions of all parameters, then (3) taking arithmean of the resulting distribution-over-probabilities and (4) acting according to that mean probability.
I think âact according to that mean probabilityâ is wrong for many important decisions you might want to takeâanalogous to buying a lot of trousers with 1.97 legs in my example in the essay. No additional comment if that is what you meant though and were just using shorthand for that position.
Clarifying, I do agree that there are some situations where you need something other than a subjective p(risk) to compare EV(value|action A) with EV(value|action B). I donât actually know how to construct a clear analogy from the 1.97-legged trousers example if the variable weâre meaning is probabilities (though I agree that there are non-analogous examples; VOI for example).
Iâll go further, though, and claim that what really matters is what worlds the risk is distributed over, and that expanding the point-estimate probability to a distribution of probabilities, by itself, doesnât add any real value. If it is to be a valuable exercise, you have to be careful what youâre expanding and what youâre refusing to expand.
More concretely, you want to be expanding over things your intervention wonât control, and then asking about your interventionâs effect at each point in things-you-wonât-control-space, then integrating back together. If you expand over any axis of uncertainty, then not only is there a multiplicity of valid expansions, but the natural interpretation will be misleading.
For example, say we have a 10% chance of drawing a dangerous ball from a series of urns, and 90% chance of drawing a safe one. If we describe it as (1) â50% chance of 9.9% risk, 50% chance of 10.1% riskâ or (2) â50% chance of 19% risk, 50% chance of 1% riskâ or (3) â10% chance of 99.1% risk, 90% chance of 0.1% riskâ, what does it change our opinion of <intervention A>? (You can, of course, construct a two-step ball-drawing procedure that produces any of these distributions-over-probabilities.)
I think the natural intuition is that interventions are best in (2), because most probabilities of risk are middle-ish, and worst in (3), because probability of risk is near-determined. And this, I think, is analogous to the argument of the post that anti-AI-risk interventions are less valuable than the point-estimate probability would indicate.
But that argument assumes (and requires) that our interventions can only chance the second ball-drawing step, and not the first. So using that argument requires that, in the first place, we sliced the distribution up over things we couldnât control. (If that is the thing we can control with our intervention, then interventions are best in the world of (3).)
Back to the argument of the original post: Youâre deriving a distribution over several p(X|Y) parameters from expert surveys, and so the bottom-line distribution over total probabilities reflects the uncertainty in expertsâ opinions on those conditional probabilities. Is it right to model our potential interventions as influencing the resolution of particular p(X|Y) rolls, or as influencing the distribution of p(X|Y) at a particular stage?
I claim itâs possible to argue either side.
Maybe a question like âp(much harder to build aligned than misaligned AGI | strong incentives to build AGI systems)â (the second survey question) is split between a quarter of the experts saying ~0% and three-quarters of the experts saying ~100%. (This extremizes the example, to sharpen the hypothetical analysis.) We interpret this as saying thereâs a one-quarter chance weâre ~perfectly safe and a three-quarters chance that itâs hopeless to develop and aligned AGI instead of a misaligned one.
If we interpret that as if God will roll a die and put us in the âmuch harderâ world with three-quarters probability and the ânot much harderâ world with one-quarters probability, then maybe our work to increase the we get an aligned AGI is low-value, because itâs unlikely to move either the ~0% or ~100% much lower (and we canât change the die). If this was the only stage, then maybe all of working on AGI risk is worthless.
But âthree-quarter chance itâs hopelessâ is also consistent with a scenario where thereâs a three-quarters chance that AGI development will be available to anyone, and many low-resourced actors will not have alignment teams and find it ~impossible to develop with alignment, but a one-quarter chance that AGI development will be available only to well-resourced actors, who will find it trivial to add on an alignment team and develop alignment. But then working on AGI risk might not be worthless, since we can work on increasing the chance that AGI development is only available to actors with alignment teams.
I claim that it isnât clear, from the survey results, whether the distribution of expertsâ probabilities for each step reflect something more like the God-rolls-a-die model, or different opinions about the default path of a thing we can intervene on. And if thatâs not clear, then itâs not clear what to do with the distribution-over-probabilities from the main results. Probably theyâre a step forward in our collective understanding, but I donât think you can conclude from the high chances of low risk that thereâs a low value to working on risk mitigation.
I agree that geomean-of-odds performs better than geomean-of-probs!
I still think it has issues for converting your beliefs to actions, but I collected that discussion under a cousin comment here: https://ââforum.effectivealtruism.org/ââposts/ââZ7r83zrSXcis6ymKo/ââdissolving-ai-risk-parameter-uncertainty-in-ai-future?commentId=9LxG3WDa4QkLhT36r
An explicit case where I think itâs important to arithmean over your subjective distribution of beliefs:
coin A is fair
coin B is either 2% heads or 98% heads, you donât know
you lose if either comes up tails.
So your p(win) is âeither 1% or 49%â.
I claim the FF should push the button that pays us $80 if win, -$20 if lose, and in general make action decisions consistent with a point estimate of 25%. (Iâm ignoring here the opportunity to seek value of information, which could be significant!).
Itâs important not to use geomean-of-odds to produce your actions in this scenario; that gives you ~9.85%, and would imply you should avoid the +$80;-$20 button, which I claim is the wrong choice.
Thanks for clarifying âgeomean of probabilitiesâ versus âgeomean of odds elsethread. I agree that that resolves some (but not all) of my concerns with geomeaning.
I think the way in which I actually disagree with the Future Fund is more radical than simple means vs geometric mean of oddsâI think they ought to stop putting so much emphasis on summary statistics altogether.
I agree with your pro-distribution position here, but I think you will be pleasantly surprised by how much reasoning over distributions goes into cost-benefit estimates at the Future Fund. This claim is based on nonpublic information, though, as those estimates have not yet been put up for public discussion. I will suggest, though, that itâs not an accident that Leopold Aschenbrenner is talking with QURI about improvements to Squiggle: https://ââgithub.com/ââquantified-uncertainty/ââsquiggle/ââdiscussions
So my subjective take is that if the true issue is âyou should reason over distributions of core parametersâ, then in fact thereâs little disagreement between you and the FF judges (which is good!), but it all adds up to normality (which is bad for the claim âmoving to reasoning over distributions should move your subjective probabilitiesâ).
If weâre focusing on the Worldview Prize question as posed (âshould these probability estimates change?â), then I think the geo-vs-arith difference is totally cruxyânote that the arithmetic summary of your results (9.65%) is in line with the product of the baseline subjective probabilities for the prize (something like a 3% for loss-of-control x-risk before 2043; something like 9% before 2100).
I do think itâs reasonable to critique the fact that those point probabilities are presented without any indication that the path of reasoning goes through reasoning over distributions, though. So I personally am happy with this post calling attention to distributional reasoning, since itâs unclear in this case whether that is an update. I just donât expect it to win the prizes for changing estimates.
Because I do think distributional reasoning is important, though, I do want to zoom in on the arith-vs-geo question (which I think, on reflection, is subtler than the position I took in my top-level comment). Rather than being a minor detail, I think this is important because it influences whether greater uncertainty tends to raise or lower our âfair betting oddsâ (which, at the end of the day, are the numbers that matter for how the FF decides to spend money).
I agree with Jamie and you and Linch that when pooling forecasts, itâs reasonable (maybe optimal? maybe not?) to use geomeans. So if youâre pooling expert forecasts of {1:1000, 1:100, 1:10}, you might have a subjective belief of something like â1:100, but with a âstandard deviationâ of 6.5x to either sideâ. This is lower than the arithmean-pooled summary stats, and I think thatâs directionally right.
I think this is an importantly different question from âhow should you act when your subjective belief is a distribution like that. I think that if you have a subjective belief like â1%, but with a âstandard deviationâ of 6.5x to either sideâ, you should push a button that gives you $98.8 if youâre right and loses $1.2 if youâre wrong. In particular, I think you should take the arithmean over your subjective distribution of beliefs (here, ~1.4%) and take bets that are good relative to that number. This will lead to decision-relevant effective probabilities that are higher than geomean-pooled point estimates (for small probabilities).
If youâre combining multiple case parameters multiplicatively, then the arith>geo effect compounds as you introduce uncertainty in more placesâif the quantity of interest is x*y, where x and y each had expert estimates of {1:1000, 1:100, 1:10} that we assume independent, then arithmean(x*y) is about twice geomean(x*y). Hereâs a quick Squiggle showing what I mean: https://ââwww.squiggle-language.com/ââplayground/ââ#code=eNqrVirOyC8PLs3NTSyqVLIqKSpN1QELuaZkluQXwUQy8zJLMhNzggtLM9PTc1KDS4oy89KVrJQqFGwVcvLT8%2FKLchNzNIAsDQM9A0NNHQ0jfWPNOAM9U82YvJi8SqJUVQFVVShoKVQCsaGBQUyeUi0A3tIyEg%3D%3D
For this use-case (eg, âwhat bets should we make with our moneyâ), Iâd argue that you need to use a point estimate to decide what bets to make, and that you should make that point estimate by (1) geomean-pooling raw estimates of parameters, (2) reasoning over distributions of all parameters, then (3) taking arithmean of the resulting distribution-over-probabilities and (4) acting according to that mean probability.
In the case of the Worldview Prize, my interpretation is that the prize is described and judged in terms of (3), because that is the most directly valuable thing in terms of producing better (4)s.
It sounds like the headline claim is that (A) we are 33.2% to live in a world where the risk of loss-of-control catastrophe is <1%, and 7.6% to live in a world where the risk is >35%, and a whole distribution of values between, and (B) that it follows from A that the correct subjective probability of loss-of-control catastrophe is given by the geometric mean of the risk, over possible worlds.
The âheadlineâ result from this analysis is that the geometric mean of all synthetic forecasts of the future is that the Communityâs current best guess for the risk of AI catastrophe due to an out-of-control AGI is around 1.6%. You could argue the toss about whether this means that the most reliable âfair betting oddsâ are 1.6% or not (Future Fund are slightly unclear about whether theyâd bet on simple mean, median etc and both of these figures are higher than the geometric mean[9]).
I want to argue that the geometric mean is not an appropriate way of aggregating probabilities across different âworlds we might live inâ into a subjective probability (as requested by the prize). This argument doesnât touch on the essayâs main argument in favor of considering distributions, but may move the headline subjective probability that it suggests to 9.65%, effectively outside the range of opinion-change prizes, so I thought it worth clarifying in case I misunderstand.
Consider an experiment where you flip a fair coin A. If A is heads you flip a 99%heads coin B; if A is tails you flip a 1%heads coin B. Weâre interested in forming a subjective probability that B is heads.
The answer I find intuitive for p(B=heads) is 50%, which is achieved by taking the arithmetic average over worlds. The geometric average over worlds gives 9.9% instead, which doesnât seem like âfair betting oddsâ for B being heads under any natural interpretation of those words. Whatâs worse, the geometric-mean methodology suggests a 9.9% subjective probability of tails, and then p(H)+p(T) does not add to 1.
(If youâre willing to accept probabilities that are 0 and 1, then an even starker experiment is given by a 1% chance to end up in a world with 0% risk and a 99% chance to end up in a world with 100% riskâthe geometric mean is 0.)
Footnote 9 of the post suggests that the operative meaning of âfair betting oddsâ is sufficiently undefined by the prize announcement that perhaps it refers to a Brier-score bet, but I believe that it is clear from the prize announcement that a X bet is the kind under consideration. The prize announcementâs footnote 1 says âWe will pose many of these beliefs in terms of <u>subjective probabilities, which represent betting odds</âu> that we consider fair in the sense that weâd be roughly indifferent between betting in favor of the relevant propositions <u>at those odds</âu> or betting against them.â
I donât know of a natural meaning of âbet in favor of P at 97:3 oddsâ other than âbet to win $97N if P and lose $3N if not Pâ, which the bettor should be indifferent about if . Is there some other bet that you believe âbet in favor of P at odds of X:Yâ could mean? In particular, is there a meaning which would support forming odds (and subjective probability) according to a geometric mean over worlds?
(I work at the FTX Foundation, but have no connection to the prizes or their judging, and my question-asking here is as a EA Forum user, not in any capacity connected to the prizes.)
To qualify, please please publish your work (or publish a post linking to it) on the Effective Altruism Forum, the AI Alignment Forum, or LessWrong with a âFuture Fund worldview prizeâ tag. You can also participate in the contest by publishing your submission somewhere else (e.g. arXiv or your blog) and filling out this submission form. We will then linkpost/âcrosspost to your submission on the EA Forum.
I think it would be nicer if you say your P(Doom|AGI in 2070) instead of P(Doom|AGI by 2070), because the second one implicitly takes into account your timelines.
I disagree. (At least, if defining ânicerâ as âmore useful to the stated goals for the prizesâ.)
As an interested observer, I think itâs an advantage to take timelines into account. Specifically, I think the most compelling way to argue for a particular P(Catastrophe|AGI by 20__) to the FF prize evaluators will be:
states and argues for a timelines distribution in terms of P(AGI in 20__) for a continuous range of 20__s
states and argues for a conditional-catastrophe function in terms of P(Catastrophe|AGI in 20__) over the range
integrates the product over the range to get a P(Catastrophe|AGI by 20__)
argues that the final number isnât excessively sensitive to small shifts in the timelines distribution or the catastrophe-conditional-on-year function.
An argument which does all of this successfully is significantly more useful to informing the FFâs actions than an argument which only defends a single P(Catastrophe|20__).
I do agree that it would be nice to have the years line up, but as above I do expect a winning argument for P(Catastrophe|AGI by 2070) to more-or-less explicitly inform a P(Catastrophe|AGI by 2043), so I donât expect a huge loss.
(Not speaking for the prizes organizers/âevaluators, just for myself.)
are there candidate interventions that only require mobile equipment and not (semi-)permanent changes to buildings?
Fortunately, yes. Within-room UVC (upper-room 254nm and lower-room 222nm) can be provided by mobile lights on tripod stands.
This is what the JHU Center for Health Security did for their IAQ conference last month. (Pictures at https://ââtwitter.com/ââDrNikkiTeran/ââstatus/ââ1567864920087138304 )
(Speaking for myself and not my employer.)
US tax law requires that US citizens pay income tax and capital gains tax, regardless of their physical/âlegal residency. Some limited deductions apply, but donât change the basic story.
Are you proposing to bite the bullet on the $100/âhr card charge scenario by the $50/âhr staffer (paid â$47.5/âhr plus perksâ at the EA org)?
âMarket rateâ of $50/âhr for labor netting $500/âhr of value seems well within the distribution Iâd expect (not to mention that EA orgs might value that work even more than any org in industry ever will, perhaps because weâre counting the consumer surplus and un-capturable externalities and the industry employer wonât).
(Iâm a trader at a NY-based quant firm, and work on education and training for new traders, among other things.)
Iâm nearly certain that your hiring manager (or anyone involved in hiring you) would be happy to receive literally this question from you, and would have advice specifically tailored to the firm youâre joining.
The firm has very a strong interest in your success (likely more so than anyone youâve interacted with in college), and theyâve already already committed to spending substantial resources to helping you prepare for a successful career as a trader. Answering questions like this one (even before youâve âofficiallyâ started) is literally (part of) someoneâs job.
(Iâm declining to answer the actual question not to be unfriendly, but because I think the folks at your future employer will have more accurate answers than I can give.)
Finally, I expect that my earmarking of grant funds will be partially funged within the GFI organization, and I think this is inevitable, basically fine, and in fact weakly good.
I received a private request (from an early reviewer of this post) to expand on my thoughts here, so a few more words:
When making decisions under collective uncertainty, aggregating information is a hard problem (citation not required). I think that my relative opinions here push the world towards a more efficient allocation, but I recognize that my opinions about GFI are inevitably incomplete. So if I overstated my certainty too much when translating my opinions into effects-on-the-world, I expect I would be making the allocation of resources less efficient overall. If I insisted on absolutely no counterfactual funging, I would be overstating my confidence.
On the other hand, if I trust GFI to take my grants in the spirit that theyâre intended, then I expect theyâll take them as information given in good faith, trust that I was trying to communicate something that I thought was not already known to them, consider what things they know that (they think) were not known to me, and decide what the net effect of my additional opinion should be. (This should remind you of Aumannâs agreement theorem, if youâre familiar with that concept from the rationality literature.)
(I think itâs also plausible in general that earmarking $X in a vote of confidence in a particular program prompts the receiving organization to update their beliefs and direct more non-earmarked funding than they would have otherwise, causing the opposite of funging.)
Do I actually believe that GFIâs principals are as good at playing this Aumann-esque information-aggregation game as the professional colleagues Iâm used to working with? Probably not, no. But this is the way I think cooperative allocation of resources should play out, and I think that the EA community only gets better at it if we start discussing ideas like this and playing âcooperateâ in the epistemic prisonersâ dilemma. And my instinct is actually that if some of my funding ends up being funged towards initiatives that GFI principals think are highest-value, itâs probably net good for the overall work.
The quadratic-proportional lemma works in the setting where thereâs an unbounded total pool; if one projectâs finding necessarily pulls from another, then I agree it doesnât work to the extent that that tradeoff is in play.
In this case, Iâm modeling each cause as small relative to the total pool, in which case the error should be correspondingly small.