# AllAmericanBreakfast comments on Biases in our estimates of Scale, Neglectedness and Solvability?

• The end of this post will be be­yond my math til next year, so I’m glad you wrote it :) Have you given thought to the pre-ex­ist­ing cri­tiques of the ITN frame­work? I’ll link to my re­view of them later.

In gen­eral, ITN should be used as a rough, non-math­e­mat­i­cal heuris­tic. I’m not sure the the­ory of cause pri­ori­ti­za­tion is de­vel­oped enough to per­mit so much math­e­mat­i­cal re­fine­ment.

In fact, I fear that it gives a sheen of pre­ci­sion to what is truly a rough-hewn com­mu­ni­ca­tion de­vice. Can you give an ex­am­ple of how an EA or­ga­ni­za­tion presently us­ing ITN could im­prove their anal­y­sis by im­ple­ment­ing some of the changes and con­sid­er­a­tions you’re point­ing out?

• The end of this post will be be­yond my math til next year, so I’m glad you wrote it :)

Thanks!

Have you given thought to the pre-ex­ist­ing cri­tiques of the ITN frame­work? I’ll link to my re­view of them later.

I think I’ve looked at a few briefly.

I think the frame­work is mostly fine the­o­ret­i­cally, based on the for­mal defi­ni­tions of the (lin­ear scale) terms and their product as a cost-effec­tive­ness es­ti­mate. I’d imag­ine the con­cerns are more with the ac­tual in­ter­pre­ta­tion and ap­pli­ca­tion. For ex­am­ple, Ne­glect­ed­ness is calcu­lated based on cur­rent re­souces, not also pro­jected re­sources, so an is­sue might not be re­ally ne­glected, be­cause you ex­pect re­sources to in­crease in the fu­ture with­out your in­ter­ven­tion. This ac­counts par­tially for “ur­gency”.

Can you give an ex­am­ple of how an EA or­ga­ni­za­tion presently us­ing ITN could im­prove their anal­y­sis by im­ple­ment­ing some of the changes and con­sid­er­a­tions you’re point­ing out?

I think EA orgs are mostly not mak­ing the mis­takes I de­scribe in each of sec­tions 1, 2 and 3, but the solu­tion is pretty straight­for­ward: con­sider the pos­si­bil­ity of nega­tive out­comes, and take ex­pec­ta­tions be­fore tak­ing log­a­r­ithms.

For the Bonus sec­tion, my sug­ges­tion would be to give (in­de­pen­dent) dis­tri­bu­tions for each of the fac­tors, and check the bounds I de­scribe in “Bound­ing the er­ror.” to check how sen­si­tive the anal­y­sis could be to de­pen­den­cies, and if it’s not sen­si­tive enough to change your pri­ori­ties, then pro­ceed as usual. If you find that it could be sen­si­tive enough and you think there may be de­pen­den­cies, model de­pen­den­cies in the dis­tri­bu­tions and ac­tu­ally calcu­late/​es­ti­mate the ex­pected value of the product (us­ing Guessti­mate, for ex­am­ple, but keep­ing in mind that ex­tremely un­likely out­comes might not get sam­pled, so if most of the value is in those, your es­ti­mate will usu­ally be way off).

Or you can just rely on cost-effec­tive­ness analy­ses for spe­cific in­ter­ven­tions when they’re available, but they aren’t always available.

• Here is that re­view I men­tioned. I’ll try and add this post to that sum­mary when I get a chance, though I can’t do jus­tice to all the math­e­mat­i­cal de­tails.

If you do give it a glance, I’d be cu­ri­ous to hear your thoughts on the cri­tiques re­gard­ing the shape and size of the marginal re­turns graph. It’s these con­cerns that I found most com­pel­ling as fun­da­men­tal cri­tiques of us­ing ITN as more than a rough first-pass heuris­tic.

• I’ve added a sum­mary at the start of the Bonus sec­tion you could use:

When there’s un­cer­tainty in the fac­tors and they cor­re­late pos­i­tively, we may be un­der­es­ti­mat­ing the marginal cost-effec­tive­ness. When there’s un­cer­tainty in the fac­tors and they cor­re­late nega­tively, we may be over­es­ti­mat­ing the marginal cost-effec­tive­ness.

(And this is be­cause we’re tak­ing the product of the ex­pected val­ues rather than the ex­pected value of the product and not ex­plic­itly mod­el­ling cor­re­la­tions be­tween terms.)

If you do give it a glance, I’d be cu­ri­ous to hear your thoughts on the cri­tiques re­gard­ing the shape and size of the marginal re­turns graph. It’s these con­cerns that I found most com­pel­ling as fun­da­men­tal cri­tiques of us­ing ITN as more than a rough first-pass heuris­tic.

Is this about Ne­glect­ed­ness as­sum­ing diminish­ing marginal re­turns? I think if you try to model the fac­tors as they’re defined for­mally and math­e­mat­i­cally by 80,000 Hours, Tractabil­ity can cap­ture effects in the op­po­site di­rec­tion and, e.g. in­creas­ing marginal re­turns. At any rate, if the terms, as defined math­e­mat­i­cally, are mod­el­led prop­erly (and as­sum­ing no di­vi­sion by zero is­sues), then when we take the product, we get Good done /​ ex­tra re­sources, and there’s noth­ing there that im­plies an as­sump­tion of diminish­ing or in­creas­ing marginal re­turns, so if Ne­glect­ed­ness as­sumes diminish­ing marginal re­turns, then the other fac­tors as­sume in­creas­ing marginal re­turns to com­pen­sate.

How many ex­tra re­sources we con­sider in Ne­glect­ed­ness could be im­por­tant, though, and it could be the case that Good done /​ ex­tra re­sources is higher or lower de­pend­ing on the size of “ex­tra re­sources”. I think this is where we would see diminish­ing or in­creas­ing marginal re­turns, but no as­sump­tion ei­ther way.