and how finely we individuate them
The Banzhaf value should avoid this problem since it has the property of 2-Efficiency: “The 2-Efficiency property states that the allocation rule that satisfies it is immune against artificial merging or splitting of players.”
I like this angle! It seems useful to compare the Shapley value in this domain to the Banzhaf value. (Brief, dense description: If Shapley value attributes value to pivotal actors during the sequential process of coalition formation (averaged across all permutations of coalition formation orderings), Banzhaf value attributes value to critical actors without which any given coalition would fail. See Shapley-Shubik power index and Banzhaf power index for similar concepts in a slightly different context.)
This paper has a nice table of properties:
| Property | Shapley | Banzhaf |
|---|---|---|
| Efficiency | Yes | No |
| Dummy player property | Yes | Yes |
| Null player property | Yes | Yes |
| Symmetry | Yes | Yes |
| Anonymity | Yes | Yes |
| Additivity | Yes | Yes |
| Transfer property | Yes | Yes |
| 2-Efficiency | No | Yes |
| Total power | No | Yes |
| Strong monotonicity | Yes | Yes |
| Marginal contributions | Yes | Yes |
(“Additivity’ is the same as “linearity” here.)
Focusing on just the properties where they differ:
Efficiency: I’ve sometimes seen this called “full allocation” which is suggestive. It’s basically just whether the full value of the coalition is apportioned to actors of the coalition or if some of it is leftover.
2-Efficiency: “The 2-Efficiency property states that the allocation rule that satisfies it is immune against artificial merging or splitting of players.”
Total power: “The Total power property establishes that the total payoff obtained for the players is the sum of all marginal contributions of every player normalized by .”
I’d have to think about this more carefully, but it’s not immediately obvious to me which set of properties is better for the purpose at hand.
They actually have a working paper for an updated version that I was just able to dig up (the links from Google Scholar seem broken ATM): The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years of Research Findings.
The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings is a broader and (IMO) nice overview of this stuff. Here’s a snippet from the central table:
| Personnel measures | Validity (r) |
|---|---|
| GMA tests | .51 |
| Work sample tests | .54 |
| Integrity tests | .41 |
| Conscientiousness tests | .31 |
| Employment interviews (structured) | .51 |
| Employment interviews (unstructured) | .38 |
| Job knowledge tests | .48 |
| Job tryout procedure1 | .44 |
| Peer ratings | .49 |
| T & E behavioral consistency method | .45 |
| Reference checks | .26 |
| Job experience (years) | .18 |
| Biographical data measures | .35 |
| Assessment centers | .37 |
| T & E point method | .11 |
| Years of education | .10 |
| Interests | .10 |
| Graphology | .02 |
| Age | -.01 |
Thanks, I find this quite interesting!
To test the limits of the theory, it seems like it might be useful to come up with factors that work in the opposite direction. I’ve heard that caloric restriction and castration both contribute to longevity in humans and seem (arguably?) affectively negative. Smoking is arguably affectively positive and increases biological ageing. I have to guess we could come up with many other examples.
Obviously, it’s highly speculative at this point, but what would you guess the correlation coefficient is between cumulative welfare and biological ageing? How large does the correlation need to be before it’s useful?
Also, seems like full-on state-building—which is presumably what would need to happen in terra nullius—is a different (harder?) task than a charter city. As I understand things, charter cities typically rely on many services/institutions of their host polity.
I thought the The Psychology of Environmental Decisions had some good info:
We argue that to promote pro-environmental decisions and to achieve public consensus on the need for action we must address individual and collective understanding (cognition) of environmental problems, as well as individual and collective commitments to take action to mitigate or prevent those problems. We review literature pertaining to psychological predispositions, mental models, framing, psychological distance, and the social context of decisions that help elucidate how these goals of cognition and commitment can be achieved.
Overcoming public resistance to carbon taxes is a bit narrower (perhaps generalizable) but still good and relevant IMO. They outline five general reasons for opposition to carbon taxes:
The personal costs are perceived to be too high
Carbon taxes can be regressive
Carbon taxes could damage the wider economy
Carbon taxes are believed not to discourage high‐carbon behavior
Governments may want to tax carbon to increase their revenues
and outline some policy advice:
Phasing in carbon taxes over time
Earmarking tax revenues for additional climate change mitigation
Redistributing taxes to improve fairness
Information sharing and communication
Yeah, I’m not expecting RCTs. I just think that some attempt at causal inference would be great (e.g. instrumental variable, difference in differences). I also don’t think this is a purely procedural complaint (i.e. not just a rote repetition of “Correlation isn’t causation!”)--I think there are real risks around confounding and external validity.
I’m also fully onboard for the claim that institutions matter. For me, the uncertainty comes in when we ask “Can this intervention change the right institutions with the right direction and magnitude?”.
(Also, I don’t think it’ll be that productive to talk about without bringing more serious evidence to bear but even 0 doesn’t strike me as “very pessimistic”. There have been plenty of well-intentioned policies with a net negative effect.)
A 0.5% boost in annual GDP per capita growth doesn’t strike me as a very pessimistic “pessimistic” estimate.
Regulation and Growth looks to be one of the two citations on this (IMO crucial) parameter, but it’s just a correlational study of regulations vs growth. The other is China’s Special Economic Zones at 30 which looks to be basically a case study.
The Skeptics Guide to Institutions (four parts total) has some background from the skeptical perspective.
(For the record, I think charter cities are interesting. But I also think the domain is extremely complicated and it seems hard to get impact estimates that are even remotely reliable.)
I mentioned it in my comment elsewhere, but—from a quick look at the paper and the supplementary material—I don’t think it’s much like any of these. They don’t make any special mention that I could find of trying to translate purely economic measures into welfare. The only mention I could find about income adjustment is “rich/poor specifications” which appears to be about splitting the formula for growth of damages into one of two forms depending on whether the country is rich or poor.
More plainly, I think the final number should be interpreted as yet another estimate in the long line of social cost of carbon estimates. It seems to be measuring the same thing as all the others (i.e. not utility) and I don’t know where the idea of income-adjustment in this post is coming from.
Thanks, this is interesting! I quickly read through the core paper and am a bit confused.
It seems like you’re understanding income adjustment to be one of the main additions in the paper. Where are you seeing that? The title/abstract/etc. seem to be pitching greater spatial resolution as the main contribution. Greater spatial resolution helps with income adjustment but isn’t sufficient. As far as I can tell the paper primarily uses regular old GDP per capita (with the de rigeur acknowledgement that GDP isn’t a great welfare measure). The only income adjustment I see is a couple of mentions of rich/poor specifications and the supplementary information suggests that this is just splitting the formula for the growth of damages based on whether a country falls into the rich bin or poor bin.
They explain the increased cost not as due to income adjustment (as I understand things) but because:
The median estimates of the GSCC (Fig. 1) are significantly higher than the Inter-agency Working Group estimates, primarily due to the higher damages associated with the empirical macroeconomic production function
All that said, I wish it did use logarithmic utility because that seems like an important improvement!
(FYI: All the inline footnotes still link to a Google doc.)
(I think my other two recent comments sort of answer each of your questions.)
I’m a bit confused. In the GiveDirectly case for ‘value of increasing consumption’, you’re still holding the discount rate constant, right?
Nope, it varies. One way you can check this intuitively is: if the discount rate and all other parameters were held constant, we’d have a proper function and our scatter plot would show at most one output value for each input.
taking GiveWell’s point estimate as the prior mean, how do the cost-effectiveness estimates (and their uncertainty) change as we vary our uncertainty over the input parameters.
There are (at least) two versions I can think of:
Adjust all the input uncertainties in concert. That is, spread all the point estimates by ±20% or all by ±30% , etc. This would be computationally tractable, but I’m not sure it would get us too much extra. I think the key problem with the current approach which would remain is that we’re radically more uncertain about some of the inputs than the others.
Adjust all the input uncertainties individually. That is, spread point estimate 1 by ±20%, point estimate 2 by ±10%, etc. Then, spread point estimate 1 by ±10%, spread point estimate 2 by ±20%, etc. Repeat for all combinations of spreads and inputs. This would actually give us somewhat useful information, but would be computational intractable given the number of input parameters.
Short version:
Do the expected values of the output probability distributions equal the point estimates that GiveWell gets from their non-probabilistic estimates?
No, but they’re close.
More generally, are there any good write-ups about when and how the expected value of a model with multiple random variables differs from the same model filled out with the expected value of each of its random variables?
Don’t know of any write-ups unfortunately, but the linearity of expectation means that the two are equal if and (generally?) only if the model is linear.
Long version:
When I run the Python versions of the models with point estimates, I get:
| Charity | Value/$ |
|---|---|
| GiveDirectly | 0.0038 |
| END | 0.0211 |
| DTW | 0.0733 |
| SCI | 0.0370 |
| Sightsavers | 0.0394 |
| Malaria Consortium | 0.0316 |
| HKI | 0.0219 |
| AMF | 0.0240 |
The (mostly minor) deviations from the official GiveWell numbers are due to:
Different handling of floating point numbers between Google Sheets and Python
Rounded/truncated inputs
A couple models calculated the net present value of an annuity based on payments at the end of the each period instead of the beginning—I never got around to implementing this
Unknown errors
When I calculate the expected values of the probability distributions given the uniform input uncertainty, I get:
| Charity | Value/$ |
|---|---|
| GiveDirectly | 0.0038 |
| END | 0.0204 |
| DTW | 0.0715 |
| SCI | 0.0354 |
| Sightsavers | 0.0383 |
| Malaria Consortium | 0.0300 |
| HKI | 0.0230 |
| AMF | 0.0231 |
I would generally call these values pretty close.
It’s worth noting though that the procedure I used to add uncertainty to inputs doesn’t produce inputs distributions that have the original point estimate as their expected value. By creating a 90% CI at ±20% of the original value, the CI is centered around the point estimate but since log normal distributions aren’t symmetric, the expected value is not precisely at the the point estimate. That explains some of the discrepancy.
The rest of the discrepancy is presumably from the non-linearity of the models (e.g. there are some logarithms in the models). In general, the linearity of expectation means that the expected value of a linear model of multiple random variables is exactly equal to the linear model of the expected values. For non-linear models, no such rule holds. (The relatively modest discrepancy between the point estimates and the expected values suggests that the models are “mostly” linear.)
Oh, very cool! I like the idea of sampling from different GiveWell staffers’ values (though I couldn’t do that here since I regarded essentially all input parameters as uncertain instead of just the highlighted ones).
I hadn’t thought about the MPT connection. I’ll think about that more.
Thanks for your thoughts.
the thrust of what you’re saying is “we should do uncertainty analysis (use Monte Carlo simulations instead of point-estimates) as our cost-effectiveness might be sensitive to it”
Yup, this is the thrust of it.
But you haven’t shown that GiveWell’s estimates are sensitive to a reliance on point estimates (have you?)
I think I have—conditionally. The uncertainty analysis shows that, if you think the neutral uncertainty I use as input is an acceptable approximation, substantially different rankings are within the bounds of plausibility. If I put in my own best estimates, the conclusion would still be conditional. It’s just that instead of being conditional upon “if you think the neutral uncertainty I use as input is an acceptable approximation” it’s conditional upon “if you think my best estimates of the uncertainty are an acceptable approximation”.
So the summary point there is that there’s really no way to escape conditional conclusions within a subjective Bayesian framework. Conclusions will always be of the form “Conclusion C is true if you accept prior beliefs B”. This makes generic, public communication hard (as we’re seeing!), but offers lots of benefits too (which I tried to demonstrate in the post—e.g. an explicit quantification of uncertainty, a sense of which inputs are most influential).
here’s a new, really complicated methodology we could use
If I’ve given the impression that it’s really complicated, I think might have misled. One of the things I really like about the approach is that you pay a relatively modest fixed cost and then you get this kind of analysis “for free”. By which I mean the complexity doesn’t infect all your actual modeling code. For example, the GiveDirectly model here actually reads more clearly to me than the corresponding spreadsheet because I’m not constantly jumping around trying to figure out what the cell reference (e.g. B23) means in formulas.
Admittedly, some of the stuff about delta moment-independent sensitivity analysis and different distance metrics is a bit more complicated. But the distance metric stuff is specific to this particular problem—not the methodology in general—and the sensitivity analysis can largely be treated as a black box. As long as you understand what the properties of the resulting number are (e.g. ranges from 0-1, 0 means independence), the internal workings aren’t crucial.
I think it would actually be very useful for you to input your best guess inputs (and its likely to be more useful for you to do it than an average EA, given you’ve thought about this more)
Given the responses here, I think I will go ahead and try that approach. Though I guess even better would be getting GiveWell’s uncertainty on all the inputs (rather than just the inputs highlighted in the “User weights” and “Moral inputs” tab).
Sorry for adding even more text to what’s already a lot of text :). Hope that helps.
You looked at the overall recap and saw the takeaways there? e.g. Sensitivity analysis indicates that some inputs are substantially more influential than others, and there are some plausible values of inputs which would reorder the ranking of top charities.
These are sort of meta-conclusions though and I’m guessing you’re hoping for more direct conclusions. That’s sort of hard to do. As I mention in several places, the analysis depends on the uncertainty you feed into it. To maintain “neutrality”, I just pretended to be equally uncertain about each input. But, given this, any simple conclusions like “The AMF cost-effectiveness estimates have the most uncertainty.” or “The relative cost-effectiveness is most sensitive to the discount rate.” would be misleading at best.
The only way to get simple conclusions like that is to feed input parameters you actually believe in to the linked Jupyter notebook. Or I could put in my best guesses as to inputs and draw simple conclusions from that. But then you’d be learning about me as much as you’d be learning about the world as you see it.
Does that all make sense? Is there another kind of takeaway that you’re imagining?
The accuracy of technological forecasts, 1890-1940 is a paper I happened to already know about that seems somewhat relevant but I didn’t see mentioned: