Ross RheingansYoođ¸
The nice thing about the quadratic voting /â quadratic funding formula (and the reason that so many people are huge nerds about it) is that the optional diversification is really easy to state:
You should donate in a $X : $1 ratio of org A to org B if you believe that org A is X times as effective as org B (in their marginal use of funding).
One explanation for this is, if youâre donating $X to A and $1 to B, then adding one more cent to A increases Aâs total match by 1/âX the amount that B would get if you gave it to B. So the point where your marginal nextdollar is equal is the point where your funding /â votes are proportional to impact.
(I have a comment nephew to this one that argues against âpolitics doesnât belong hereâ, but I also wanted to provide a cautionary suggestion...)
I have found it pretty difficult to think in a balanced way about...any election in the last three cycles...but I want to propose that, on the outside view, âcandidate seems Obviously Badâ and âcandidate would have a negative counterfactual effect on [classical EA cause area]â is nowhere near as correlated as it intuitively feels it should be.
The example that I will keep pointing to, probably forever, is that George W. Bush was almost certainly the best president in the modern era for both Global Health and Welfare^{[1]} and for GCBRs^{[2]}, based on programs that were very far from what I (still) think of as his major policy positions.
I think that Bushâs interest in HIV response in Africa was in theory knowable at the time^{[3]}, but figuring it out would have required digging into some pretty unlikely topics on a candidate that my wouldhavebeen intellectual circles^{[4]} was pretty strongly convinced was the worse one. (Iâm not sure how knowable his proactive interest in pandemic prep was.)
I donât want to claim that itâs correct to equivocate this cycleâs Republican candidate and W. Bush here, and I donât have any concrete reason to believe that the Republican candidate is good on particular cause areas. I just mean to say, I wouldnât have believed it of W. Bush, either. And in this cycle, Iâm not aware of anyone who has really done the reasearch that would convince me one way or another in terms of the shutupandmultiply expected counterfactual utility.
So, while I donât oppose making decisions on otherthanconsequentialist and/âor commonsense grounds here (which is likely whatâs going to actually sway my ballot as a citizen), I want to argue for a stance of relatively deep epistemic uncertainty on the consequentialist dimension, until I see more focused argument from someone who really has done the homework.
 âŠď¸
In a word, PEPFAR.
 âŠď¸
The National Pharmaceutical Stockpile was founded under Clinton with $51mln of initial funding, but Bush increased the budget tenfold between the Project BioShield Act and PAHPA; expansion of the program since then has been small by comparison. Plus I think that the effect of PEPFAR on the âbiosecurity waterlineâ of the world is underappreciated.
 âŠď¸
Wikipedia: âAccording to [Bushâs] 2010 memoir, Decision Points, [George W. and Laura Bush] developed a serious interest in improving the fate of the people of Africa after reading Alex Haleyâs Roots, and visiting The Gambia in 1990. In 1998, while pondering a run for the U.S. presidency, he discussed Africa with Condoleezza Rice, his future secretary of state; she said that, if elected, working more closely with countries on that continent should be a significant part of his foreign policy.â
 âŠď¸
I was too young to have âintellectual circlesâ during the GWB presidency; Iâm approximating myself by my parents here, though itâs conflated by EA, LessWrong, et al. not existing at the time.
 âŠď¸
I think that your argument is a fair one to make, but I think itâs easier to argue for than against, so I want to argue against to avoid an information cascade towards consensus.
I generally support the outsideview argument for ideological diversification, and if that diversification means anything it has to mean supporting things that wouldnât âget thereâ on their own (especially as a small /â exploratory fraction of donation portfolio, as OP indicates here).
EA need not be totalizing, and I think the world is betteroff if EAs discuss how to bring a mindset towards effectiveness to other endeavors in their lives.
I generally think that weâve swung too far towards consensus and OpenPhil deference as a community (though thereâs been some swing back), and am actively happier to see things that arenât obviously bunk but swing us towards a more pluralistic and diverse set of approaches.
I think in particular that EA donors have historically underconsidered opportunities in politics, and am happy to see increased engagement in considering opportunities there (even if I might disagree with the choice of a particular political race as the most effective, like I might disagree with the choice of an approach within a cause area).
Whatâs more, democratic capitalism + effective altruism will direct effort and resources to effective uses even if only a few capitalhavers are unselfishly motivated in this way.
If socialism means the commandeconomy things, then democratic socialism + effective altruism doesnât reliably direct resources to causes that only a small minority are motivated to support.
Makes total sense not to invest in the charitable sideâIâm generally off a similar mind.^{[1]} The reason Iâm curious is that âconsider it as two separate accountsâ is the mostcompelling argument Iâve seen against tithing investment gains. (The argument is basically, that if both accounts were fullyinvested, then tithing gains from the personal account to the charity account leads to a total 4:1 ratio between them as withdrawal_time â â, not a 9:1 ratio.^{[2]} Then, why does distribution out of the charity account affect the ârightâ additional amount to give out of the personal account?)
Another way to count it is, if you believe that the returns on effective charity are greater than private investments returns and so always make donations asap, then tithing at the start and after years is worse for both accounts than just giving say upfront (and giving of the further investment gains).
Probably this is most relevant to startup employees, who might receive â$100,000 in equityâ that they only can sell when it later exits for, say, 10x that. Should a 10% pledge mean $10,000 upfront and $90,000 of the exit (10% when paid + 10% of gains), or just $100,000 of the exit (10% went to the charity account, then exited)?^{[3]}
(Sorry, donât mean to jump on your personal post with this tangentâam happy to chat if you find this interesting to think about, but also can write my own post about it on my own time if not
 ^
The one case where I do think investment can sense is where I want to direct the funding to accelerating the program of a forprofit company, eg in biotech, and the right way to do so is via direct investment. I do think there are such cases that can be on the frontier of mosteffective in EV terms (and for them I only count it as effective giving if I precommit to regiving any proceeds, without recounting it as a donation for pledge purposes).
 ^
Consider receiving $1,000 in salary, splitting it $100 : $900 between the accounts, investing each so they grow 10x and become $1,000 : $9,000, then realizing the personal investment gains and tithing $800 on them. Now the accounts are $1,800 : $8,200, which seems a lot more like âgiving 18%â than âgiving 10%â!
 ^
If the correct baseline is â10% of the exitâ, should this be any different from the case of a salary worker who makes the $100,000 in cash and puts it in an index fund until it [10x]s? Or what about a professional trader who ârealizes gainsâ frequently with daily trading, but doesnât take any of the money out until after many iterations?
 ^

Thoughtprovoking post; thanks for sharing!

A bit of a tangential point, but Iâm curious, because itâs something Iâve also considered:
putting 10% of my paycheck directly in a second account which was exclusively for charity
What do you do with investment income? Itâs pretty intuitive that if youâre âinvesting to giveâ and you have $9,000 of personal savings and $1,000 of donationinvestments and they both go up 10% over a year, that you should have $9,900 of personal savings and $1,100 of donationinvestments. But what would you (or do you) do differently if you put the money into the accounts, donated half of the charity account, and then ended up with $9,900 in personal savings (a $900 annual gain) and $550 in savingsforgiving (a $50 annual gain)?
I have heard at least three different suggestions for how to do this sort of accounting, but am curious what you go with, since the rest of your perspective self seems fairly intentional and considered!

Iâd argue that you need to use a point estimate to decide what bets to make, and that you should make that point estimate by (1) geomeanpooling raw estimates of parameters, (2) reasoning over distributions of all parameters, then (3) taking arithmean of the resulting distributionoverprobabilities and (4) acting according to that mean probability.
I think âact according to that mean probabilityâ is wrong for many important decisions you might want to takeâanalogous to buying a lot of trousers with 1.97 legs in my example in the essay. No additional comment if that is what you meant though and were just using shorthand for that position.
Clarifying, I do agree that there are some situations where you need something other than a subjective p(risk) to compare EV(valueaction A) with EV(valueaction B). I donât actually know how to construct a clear analogy from the 1.97legged trousers example if the variable weâre meaning is probabilities (though I agree that there are nonanalogous examples; VOI for example).
Iâll go further, though, and claim that what really matters is what worlds the risk is distributed over, and that expanding the pointestimate probability to a distribution of probabilities, by itself, doesnât add any real value. If it is to be a valuable exercise, you have to be careful what youâre expanding and what youâre refusing to expand.
More concretely, you want to be expanding over things your intervention wonât control, and then asking about your interventionâs effect at each point in thingsyouwonâtcontrolspace, then integrating back together. If you expand over any axis of uncertainty, then not only is there a multiplicity of valid expansions, but the natural interpretation will be misleading.
For example, say we have a 10% chance of drawing a dangerous ball from a series of urns, and 90% chance of drawing a safe one. If we describe it as (1) â50% chance of 9.9% risk, 50% chance of 10.1% riskâ or (2) â50% chance of 19% risk, 50% chance of 1% riskâ or (3) â10% chance of 99.1% risk, 90% chance of 0.1% riskâ, what does it change our opinion of <intervention A>? (You can, of course, construct a twostep balldrawing procedure that produces any of these distributionsoverprobabilities.)
I think the natural intuition is that interventions are best in (2), because most probabilities of risk are middleish, and worst in (3), because probability of risk is neardetermined. And this, I think, is analogous to the argument of the post that antiAIrisk interventions are less valuable than the pointestimate probability would indicate.
But that argument assumes (and requires) that our interventions can only chance the second balldrawing step, and not the first. So using that argument requires that, in the first place, we sliced the distribution up over things we couldnât control. (If that is the thing we can control with our intervention, then interventions are best in the world of (3).)
Back to the argument of the original post: Youâre deriving a distribution over several p(XY) parameters from expert surveys, and so the bottomline distribution over total probabilities reflects the uncertainty in expertsâ opinions on those conditional probabilities. Is it right to model our potential interventions as influencing the resolution of particular p(XY) rolls, or as influencing the distribution of p(XY) at a particular stage?
I claim itâs possible to argue either side.
Maybe a question like âp(much harder to build aligned than misaligned AGI  strong incentives to build AGI systems)â (the second survey question) is split between a quarter of the experts saying ~0% and threequarters of the experts saying ~100%. (This extremizes the example, to sharpen the hypothetical analysis.) We interpret this as saying thereâs a onequarter chance weâre ~perfectly safe and a threequarters chance that itâs hopeless to develop and aligned AGI instead of a misaligned one.
If we interpret that as if God will roll a die and put us in the âmuch harderâ world with threequarters probability and the ânot much harderâ world with onequarters probability, then maybe our work to increase the we get an aligned AGI is lowvalue, because itâs unlikely to move either the ~0% or ~100% much lower (and we canât change the die). If this was the only stage, then maybe all of working on AGI risk is worthless.
But âthreequarter chance itâs hopelessâ is also consistent with a scenario where thereâs a threequarters chance that AGI development will be available to anyone, and many lowresourced actors will not have alignment teams and find it ~impossible to develop with alignment, but a onequarter chance that AGI development will be available only to wellresourced actors, who will find it trivial to add on an alignment team and develop alignment. But then working on AGI risk might not be worthless, since we can work on increasing the chance that AGI development is only available to actors with alignment teams.
I claim that it isnât clear, from the survey results, whether the distribution of expertsâ probabilities for each step reflect something more like the Godrollsadie model, or different opinions about the default path of a thing we can intervene on. And if thatâs not clear, then itâs not clear what to do with the distributionoverprobabilities from the main results. Probably theyâre a step forward in our collective understanding, but I donât think you can conclude from the high chances of low risk that thereâs a low value to working on risk mitigation.
I agree that geomeanofodds performs better than geomeanofprobs!
I still think it has issues for converting your beliefs to actions, but I collected that discussion under a cousin comment here: https://ââforum.effectivealtruism.org/ââposts/ââZ7r83zrSXcis6ymKo/ââdissolvingairiskparameteruncertaintyinaifuture?commentId=9LxG3WDa4QkLhT36r
An explicit case where I think itâs important to arithmean over your subjective distribution of beliefs:
coin A is fair
coin B is either 2% heads or 98% heads, you donât know
you lose if either comes up tails.
So your p(win) is âeither 1% or 49%â.
I claim the FF should push the button that pays us $80 if win, $20 if lose, and in general make action decisions consistent with a point estimate of 25%. (Iâm ignoring here the opportunity to seek value of information, which could be significant!).
Itâs important not to use geomeanofodds to produce your actions in this scenario; that gives you ~9.85%, and would imply you should avoid the +$80;$20 button, which I claim is the wrong choice.
 22 Oct 2022 4:07 UTC; 1 point) 's comment on âDisÂsolvÂingâ AI Risk â PaÂramÂeÂter UncerÂtainty in AI FuÂture Forecasting by (
Thanks for clarifying âgeomean of probabilitiesâ versus âgeomean of odds elsethread. I agree that that resolves some (but not all) of my concerns with geomeaning.
I think the way in which I actually disagree with the Future Fund is more radical than simple means vs geometric mean of oddsâI think they ought to stop putting so much emphasis on summary statistics altogether.
I agree with your prodistribution position here, but I think you will be pleasantly surprised by how much reasoning over distributions goes into costbenefit estimates at the Future Fund. This claim is based on nonpublic information, though, as those estimates have not yet been put up for public discussion. I will suggest, though, that itâs not an accident that Leopold Aschenbrenner is talking with QURI about improvements to Squiggle: https://ââgithub.com/ââquantifieduncertainty/ââsquiggle/ââdiscussions
So my subjective take is that if the true issue is âyou should reason over distributions of core parametersâ, then in fact thereâs little disagreement between you and the FF judges (which is good!), but it all adds up to normality (which is bad for the claim âmoving to reasoning over distributions should move your subjective probabilitiesâ).
If weâre focusing on the Worldview Prize question as posed (âshould these probability estimates change?â), then I think the geovsarith difference is totally cruxyânote that the arithmetic summary of your results (9.65%) is in line with the product of the baseline subjective probabilities for the prize (something like a 3% for lossofcontrol xrisk before 2043; something like 9% before 2100).
I do think itâs reasonable to critique the fact that those point probabilities are presented without any indication that the path of reasoning goes through reasoning over distributions, though. So I personally am happy with this post calling attention to distributional reasoning, since itâs unclear in this case whether that is an update. I just donât expect it to win the prizes for changing estimates.
Because I do think distributional reasoning is important, though, I do want to zoom in on the arithvsgeo question (which I think, on reflection, is subtler than the position I took in my toplevel comment). Rather than being a minor detail, I think this is important because it influences whether greater uncertainty tends to raise or lower our âfair betting oddsâ (which, at the end of the day, are the numbers that matter for how the FF decides to spend money).
I agree with Jamie and you and Linch that when pooling forecasts, itâs reasonable (maybe optimal? maybe not?) to use geomeans. So if youâre pooling expert forecasts of {1:1000, 1:100, 1:10}, you might have a subjective belief of something like â1:100, but with a âstandard deviationâ of 6.5x to either sideâ. This is lower than the arithmeanpooled summary stats, and I think thatâs directionally right.
I think this is an importantly different question from âhow should you act when your subjective belief is a distribution like that. I think that if you have a subjective belief like â1%, but with a âstandard deviationâ of 6.5x to either sideâ, you should push a button that gives you $98.8 if youâre right and loses $1.2 if youâre wrong. In particular, I think you should take the arithmean over your subjective distribution of beliefs (here, ~1.4%) and take bets that are good relative to that number. This will lead to decisionrelevant effective probabilities that are higher than geomeanpooled point estimates (for small probabilities).
If youâre combining multiple case parameters multiplicatively, then the arith>geo effect compounds as you introduce uncertainty in more placesâif the quantity of interest is x*y, where x and y each had expert estimates of {1:1000, 1:100, 1:10} that we assume independent, then arithmean(x*y) is about twice geomean(x*y). Hereâs a quick Squiggle showing what I mean: https://ââwww.squigglelanguage.com/ââplayground/ââ#code=eNqrVirOyC8PLs3NTSyqVLIqKSpN1QELuaZkluQXwUQy8zJLMhNzggtLM9PTc1KDS4oy89KVrJQqFGwVcvLT8%2FKLchNzNIAsDQM9A0NNHQ0jfWPNOAM9U82YvJi8SqJUVQFVVShoKVQCsaGBQUyeUi0A3tIyEg%3D%3D
For this usecase (eg, âwhat bets should we make with our moneyâ), Iâd argue that you need to use a point estimate to decide what bets to make, and that you should make that point estimate by (1) geomeanpooling raw estimates of parameters, (2) reasoning over distributions of all parameters, then (3) taking arithmean of the resulting distributionoverprobabilities and (4) acting according to that mean probability.
In the case of the Worldview Prize, my interpretation is that the prize is described and judged in terms of (3), because that is the most directly valuable thing in terms of producing better (4)s.
It sounds like the headline claim is that (A) we are 33.2% to live in a world where the risk of lossofcontrol catastrophe is <1%, and 7.6% to live in a world where the risk is >35%, and a whole distribution of values between, and (B) that it follows from A that the correct subjective probability of lossofcontrol catastrophe is given by the geometric mean of the risk, over possible worlds.
The âheadlineâ result from this analysis is that the geometric mean of all synthetic forecasts of the future is that the Communityâs current best guess for the risk of AI catastrophe due to an outofcontrol AGI is around 1.6%. You could argue the toss about whether this means that the most reliable âfair betting oddsâ are 1.6% or not (Future Fund are slightly unclear about whether theyâd bet on simple mean, median etc and both of these figures are higher than the geometric mean[9]).
I want to argue that the geometric mean is not an appropriate way of aggregating probabilities across different âworlds we might live inâ into a subjective probability (as requested by the prize). This argument doesnât touch on the essayâs main argument in favor of considering distributions, but may move the headline subjective probability that it suggests to 9.65%, effectively outside the range of opinionchange prizes, so I thought it worth clarifying in case I misunderstand.
Consider an experiment where you flip a fair coin A. If A is heads you flip a 99%heads coin B; if A is tails you flip a 1%heads coin B. Weâre interested in forming a subjective probability that B is heads.
The answer I find intuitive for p(B=heads) is 50%, which is achieved by taking the arithmetic average over worlds. The geometric average over worlds gives 9.9% instead, which doesnât seem like âfair betting oddsâ for B being heads under any natural interpretation of those words. Whatâs worse, the geometricmean methodology suggests a 9.9% subjective probability of tails, and then p(H)+p(T) does not add to 1.
(If youâre willing to accept probabilities that are 0 and 1, then an even starker experiment is given by a 1% chance to end up in a world with 0% risk and a 99% chance to end up in a world with 100% riskâthe geometric mean is 0.)
Footnote 9 of the post suggests that the operative meaning of âfair betting oddsâ is sufficiently undefined by the prize announcement that perhaps it refers to a Brierscore bet, but I believe that it is clear from the prize announcement that a X bet is the kind under consideration. The prize announcementâs footnote 1 says âWe will pose many of these beliefs in terms of <u>subjective probabilities, which represent betting odds</âu> that we consider fair in the sense that weâd be roughly indifferent between betting in favor of the relevant propositions <u>at those odds</âu> or betting against them.â
I donât know of a natural meaning of âbet in favor of P at 97:3 oddsâ other than âbet to win $97N if P and lose $3N if not Pâ, which the bettor should be indifferent about if . Is there some other bet that you believe âbet in favor of P at odds of X:Yâ could mean? In particular, is there a meaning which would support forming odds (and subjective probability) according to a geometric mean over worlds?
(I work at the FTX Foundation, but have no connection to the prizes or their judging, and my questionasking here is as a EA Forum user, not in any capacity connected to the prizes.)
To qualify, please please publish your work (or publish a post linking to it) on the Effective Altruism Forum, the AI Alignment Forum, or LessWrong with a âFuture Fund worldview prizeâ tag. You can also participate in the contest by publishing your submission somewhere else (e.g. arXiv or your blog) and filling out this submission form. We will then linkpost/âcrosspost to your submission on the EA Forum.
I think it would be nicer if you say your P(DoomAGI in 2070) instead of P(DoomAGI by 2070), because the second one implicitly takes into account your timelines.
I disagree. (At least, if defining ânicerâ as âmore useful to the stated goals for the prizesâ.)
As an interested observer, I think itâs an advantage to take timelines into account. Specifically, I think the most compelling way to argue for a particular P(CatastropheAGI by 20__) to the FF prize evaluators will be:
states and argues for a timelines distribution in terms of P(AGI in 20__) for a continuous range of 20__s
states and argues for a conditionalcatastrophe function in terms of P(CatastropheAGI in 20__) over the range
integrates the product over the range to get a P(CatastropheAGI by 20__)
argues that the final number isnât excessively sensitive to small shifts in the timelines distribution or the catastropheconditionalonyear function.
An argument which does all of this successfully is significantly more useful to informing the FFâs actions than an argument which only defends a single P(Catastrophe20__).
I do agree that it would be nice to have the years line up, but as above I do expect a winning argument for P(CatastropheAGI by 2070) to moreorless explicitly inform a P(CatastropheAGI by 2043), so I donât expect a huge loss.
(Not speaking for the prizes organizers/âevaluators, just for myself.)
are there candidate interventions that only require mobile equipment and not (semi)permanent changes to buildings?
Fortunately, yes. Withinroom UVC (upperroom 254nm and lowerroom 222nm) can be provided by mobile lights on tripod stands.
This is what the JHU Center for Health Security did for their IAQ conference last month. (Pictures at https://ââtwitter.com/ââDrNikkiTeran/ââstatus/ââ1567864920087138304 )
(Speaking for myself and not my employer.)
US tax law requires that US citizens pay income tax and capital gains tax, regardless of their physical/âlegal residency. Some limited deductions apply, but donât change the basic story.
Are you proposing to bite the bullet on the $100/âhr card charge scenario by the $50/âhr staffer (paid â$47.5/âhr plus perksâ at the EA org)?
âMarket rateâ of $50/âhr for labor netting $500/âhr of value seems well within the distribution Iâd expect (not to mention that EA orgs might value that work even more than any org in industry ever will, perhaps because weâre counting the consumer surplus and uncapturable externalities and the industry employer wonât).
(Iâm a trader at a NYbased quant firm, and work on education and training for new traders, among other things.)
Iâm nearly certain that your hiring manager (or anyone involved in hiring you) would be happy to receive literally this question from you, and would have advice specifically tailored to the firm youâre joining.
The firm has very a strong interest in your success (likely more so than anyone youâve interacted with in college), and theyâve already already committed to spending substantial resources to helping you prepare for a successful career as a trader. Answering questions like this one (even before youâve âofficiallyâ started) is literally (part of) someoneâs job.
(Iâm declining to answer the actual question not to be unfriendly, but because I think the folks at your future employer will have more accurate answers than I can give.)
The quadraticproportional lemma works in the setting where thereâs an unbounded total pool; if one projectâs finding necessarily pulls from another, then I agree it doesnât work to the extent that that tradeoff is in play.
In this case, Iâm modeling each cause as small relative to the total pool, in which case the error should be correspondingly small.