Currently Research Director at Founders Pledge, but posts and comments represent my own opinions, not FP’s, unless otherwise noted.
I worked previously as a data scientist and as a journalist.
Matt_Lerner
Karma: 1,313
Thanks for this! It seems like much of the work that went into your CEA could be repurposed for explorations of other potentially growth- or governance-enhancing interventions. Since finding such an intervention would be quite high-value, and since the parameters in your CEA are quite uncertain, it seems like the value of information with respect to clarifying these parameters (and therefore the final ROI distribution) is probably very high.
Do you have a sense of what kind of research or data would help you narrow the uncertainty in the parameter inputs of your cost-effectiveness model?
On the face of it, it seems like researching and writing about “mainstream” topics is net positive value for EAs for the reasons you describe, although not obviously an optimal use of time relative to other competing opportunities for EAs. I’ve tried to work out in broad strokes how effective it might be to move money within putatively less-effective causes, and it seems to me like (for instance) the right research, done by the right person or group, really could make a meaningful difference in one of these areas.
Items 2.2 and 2.3 (in your summary) are, to me, simultaneously the riskiest and most compelling propositions to me. Could EAs really do a better job finding the “right answers” than there are to be found in existing work? I take “neglectedness” in the ITN framework to be a heuristic that serves mainly to forestall hubris in this regard: we should think twice before assuming we know better than the experts, as we’re quite likely to be wrong.
But I think there is still reason to suspect that there is value to be captured in mainstream causes. Here are a few reasons I think this might be the case.
“Outcome orientation” and a cost-benefit mindset are surprisingly rare, even in fields that are nominally outcomes-focused. This horse has already been beaten to death, but the mistakes, groupthink, and general confusion in many corners of epidemiology and public health during the pandemic suggests that consequences are less salient in these fields than I would have expected beforehand. Alex Tabarrok, a non-epidemiologist, seems to have gotten most things right well before the relevant domain experts simply by thinking in consequentialist terms. Zeynep Tufekci, Nate Silver, and Emily Oster are in similar positions.
Fields have their own idiosyncratic concerns and debates that eat up a lot of time and energy, IMO to the detriment of overall effectiveness. My (limited) experience in education research and tech in the developed world led me to conclude that the goals of the field are unclear and ill-defined (Are we maximizing graduation rates? College matriculation? Test scores? Are we maximizing anything at all?). Significant amounts of energy are taken up by debates and concerns about data privacy, teacher well-being and satisfaction, and other issues that are extremely important but which, ultimately, are not directly related to the (broadly defined) goals of the field. The drivers behind philanthropic funding seem, to me, to be highly undertheorized.
I think philanthropic money in the education sector should probably go to the developing world, but it’s not obvious to me that developed-world experts are squeezing out all the potential value that they could. Whether the scale of that potential value is large enough to justify improving the sector, or whether such improvements are tractable, are different questions.
There are systematic biases within disciplines, even when those fields or disciplines are full of smart, even outcomes-focused people. Though not really a cause area, David Shor has persuasively argued that Democratic political operatives are ideological at the cost of being effective. My sense is that this is also true to some degree in education.
There are fields where the research quality is just really low. The historical punching bag for this is obviously social psychology, which has been in the process of attempting to improve for a decade now. I think the experience of the replication crisis—which is ongoing—should cause us to update away from thinking that just because lots of people are working on a topic, that means that there is no marginal value to additional research. I think the marginal value can be high, especially for EAs, who are constitutionally hyper-aware of the pitfalls of bad research, have high standards of rigor, and are often quantitatively sophisticated. EAs are also relatively insistent on clarity, the lack of which seems to be a main obstacle to identifying bad research.
I think about this all the time. It seems like a really high-value thing to do not just for the sake of other communities but even from a strictly EA perspective— discourse norms seem to have a real impact on the outcome of decision-relevant conversations, and I have an (as-yet unjustified) sense that EA-style norms lead to better normative outcomes. I haven’t tried it, but I do have a few isolated, perhaps obvious observations.
For me at least, it is easier to hew to EA discussion norms when they are, in fact, accepted norms. That is, assuming the best intentions of an interlocutor, explaining instead of persuading, steelmanning, etc— I find it easier to do these things when I know they’re expected of me. This suggests to me that it might hard to institute such norms unilaterally.
EA norms don’t obviously all go together. You can imagine a culture where civility is a dominant norm but where views are still expressed and argued for in a tendentious way. This would suck in a community where the shared goal is some truth-seeking enterprise, but I imagine that the more substantive EA norms around debate and discussion would actually impose a significant cost on communities where truth-seeking isn’t the main goal!
Per the work of Robert Frank, it seems like there are probably institutional design decisions that can increase the likelihood of observing these norms. I’m not sure how much the EA Forum’s designers intended this, but it seems to me like hiding low-scoring answers, allowing real names, and the existence of strong upvotes/downvotes all play a role in culture on the forum in particular.
I guess a more useful way to think about this for prospective funders is to move things about again. Given that you can exert c/x leverage over funds within Cause Y, then you’re justified in spending c to do so provided you can find some Cause Y such that the distribution of DALYs per dollar meets the condition...
...which makes for a potentially nice rule of thumb. When assessing some Cause Y, you need only (“only”) identify a plausibly best or close-to-best opportunity, as well as the median one, and work from there.
Obviously this condition holds for any distribution and any set of quintiles, but the worked example above only indicates to me that it’s a plausible condition for the log-normal.
Under what circumstances is it potentially cost-effective to move money within low-impact causes?
This is preliminary and most likely somehow wrong. I’d love for someone to have a look at my math and tell me if (how?) I’m on the absolute wrong track here.
Start from the assumption that there is some amount of charitable funding that is resolutely non-cause-neutral. It is dedicated to some cause area Y and cannot be budged. I’ll assume for these purposes that DALYs saved per dollar is distributed log-normally within Cause Y:
I want to know how impactful it might, in general terms, be to shift money from the median funding opportunity in Cause Y to the 90th percentile opportunity. So I want the difference between the value of spending a dollar at those two points on the impact distribution.
The log-normal distribution has the following quantile function:
So the value to be gained by moving from p = 0.5 to p = 0.9 is given by
This simplifies down to
Or
Not a pretty formula, but it’s easy enough to see two things which were pretty intuitive before this exercise. First, you can squeeze out more DALYs from moving money in causes where the mean DALYs per dollar across all funding opportunities is higher, and, for a given average, moving money is higher-value where there’s more variation across funding opportunities (roughly, since variance is proportional to but not precisely given by sigma). Pretty obvious so far.
Okay, what about making this money-moving exercise cost-competitive with a direct investment in an effective cause, with a benchmark of $100/DALY? For that, and for a given investment amount $x, and a value c such that an expenditure of $c causes the money in cause Y to shift from the median opportunity to the 90th-percentile one, we’d need to satisfy the following condition:
Moving things around a bit...
Which, given reasonable assumptions about the values of c and x, holds true trivially for larger means and variances across cause Y. The catch, of course, is that means and variances of DALYs per dollar in a cause area are practically never large, let alone in a low-impact cause area. Still, the implication is that (a) if you can exert inexpensive enough leverage over the funding flows within some cause Y and/or (b) if funding opportunities within cause Y are sufficiently variable, cost-effectiveness is at least theoretically possible.
So just taking an example: Our benchmark is $100 per DALY, or 0.01 DALYs per dollar, so let’s just suppose we have a low-impact Cause Y that is between three and six orders of magnitude less effective than that, with a 95% CI of [0.00000001,0.00001], or one for which you can preserve a DALY for between $100,000 and $100 million, depending on the opportunity. That gives mu = −14.97 and sigma = 1.76. Plugging those numbers into the above, we get approximately...
...suggesting, I think, that if you can get roughly 4000:1 leverage when it comes to spending money to move money, it can be cost-effective to influence funding patterns within this low-impact cause area.
There are obviously a lot of caveats here (does a true 90th percentile opportunity exist for any Cause Y?), but this is where my thinking is at right now, which is why this is in my shortform and not anywhere else.
What do you see as the consequentialist value of doing journalism? What are the ways in which journalists can improve the world? And do you believe these potential improvements are measurable?
One thing to note here is that lots of commonly-used power law distributions have positive support. Political choices can and sometimes do have dramatically negative effects, and many of the catastrophes that EAs are concerned with are plausibly the result of those choices (like nuclear catastrophe, for instance).
So a distribution that describes the outcomes of political choices should probably have support on the whole real line, and you wouldn’t want to model choices with most simple power-law distributions. But you might be on to something—you might think of a hierarchical model in which there’s some probability that decisions are either good or bad, and that the degree to which they are good or bad is governed by a power law distribution. That’s the model I’ve been working with, but it seems incomplete to me.
I read this post with a lot of interest; it has started to seem more likely to me lately that spreading productive, resilient norms about decision-making and altruism is a more effective means of improving decisions in the long run than any set of particular institutional structures. The knock-on effects of such a phenomenon would, on a long time scale, seem to dwarf the effects of many other ostensibly effective interventions.
So I get excited about this idea. It seems promising.
But some reflection about what is commonly considered precedent for something like this makes me a little bit more skeptical.
I think we see another kind of self-correction mechanism in the belief system of science. It provides tools for recognising truth and discarding falsehood, as well as cultural impetus to do so; this leads not just to the propagation of existing scientific beliefs, but to the systematic upgrading of those beliefs; this isn’t drift, but going deeper into the well of truth.
I have a sense that a large part of the success of scientific norms comes down to their utility being immediately visible. Children can conduct and repeat simple experiments (e.g. baking soda volcano); undergraduates can repeat famous projects with the same results (e.g. the double slit experiment), and even non-experimentalists can see the logic at the core of contemporary theory (e.g. in middle school geometry, or at the upper level in real analysis). What’s more, the norms seem to be cemented most effectively by precisely this kind of training, and not to spread freely without direct inculcation: scientific thinking is widespread among the trained, and (anecdotally) not so common among the untrained. For many Western non-scientists, science is just another source of formal authority, not a process that derives legitimacy from its robust efficacy.
I can see a way clear to a broadening of scientific norms to include what you’ve characterized as “truth-seeking self-aware altruistic decision-making.” But I’m having trouble imaging how it could be self-propagating. It would seem, at the very least, to require active cultivation in exactly the way that scientific norms do—in other words, that it would require a lot of infrastructure and investment so that proto-truth-seeking-altruists can see the value of the norms. Or perhaps I am having a semantic confusion: is science self-propagating in that scientists, once cultivated, go on to cultivate others?
I very strongly upvoted this because I think it’s highly likely to produce efficiencies in conversation on the Forum, to serve as a valuable reference for newcomers to EA, and to act as a catalyst for ongoing conversation.
I would be keen to see this list take on life outside the forum as a standalone website or heavily moderated wiki, or as a page under CEA or somesuch, or at QURI.
I’m not sure why this is being downvoted. I don’t really have an opinion on this, but it seems at least worth discussing. OP, I think this is an interesting idea.
John Lewis Gaddis’ The Cold War: A New History contains a number of useful segments about the nuclear tensions between the U.S. and the U.S.S.R., insightful descriptions of policymakers’ thinking during these moments, and a consideration of counterfactual histories in which nuclear weapons might have been deployed. I found it pretty useful in terms of helping me get a picture of what decision-making looks like when the wrong decision means (potentially) the end of civilization.
How harmful is a fragmented resume? People seem to believe this isn’t much of a problem for early-career professionals, but I’m 30, and my longest tenure was for two and a half years (recently shorter). I like to leave for new and interesting opportunities when I find them, but I’m starting to wonder whether I should avoid good opportunities for the sake of appearing more reliable as a potential employee.
First, congratulations. This is impressive, you should be very proud of yourself, and I hope this is the beginning of a long and fruitful data science career (or avocation) for you.
What is going on here?
I think the simplest explanation is that your model fit better because you trained on more data. You write that your best score was obtained by applying XGBoost to the entire feature matrix, without splitting it into train/test sets. So assuming the other teams did things the standard way, you were working with 25%-40% more data to fit the model. In a lot of settings, particularly in the case of tree-based methods (as I think XGBoost usually is), this is a recipe for overfitting. In this setting, however, it seems like the structure of the public test data was probably really close to the structure of the private test data, so the lack of validation on the public dataset paid off for you.I think one interpretation of this is that you got lucky in that way. But I don’t think that’s the right takeaway. I think the right takeaway is that you kept your eye on the ball and chose the strategy that worked based on your understanding of the data structure and the available methods and you should be very satisfied.
I wonder if the forum shouldn’t encourage a class of post (basically like this one) that’s something like “are there effective giving opportunities in X context?” Although EA is cause-neutral, there’s no reason why members shouldn’t take the opportunity provided by serendipity to investigate highly specific scenarios and model “virtuous EA behavior.” This could be a way of making the forum friendlier to visitors like the OP, and a way for comments to introduce visitors to EA concepts in a way that’s emotionally relevant.
I also found this (ironically) abstract. There are more than enough philosophers on this board to translate this for us, but I think it might be useful to give it a shot and let somebody smarter correct the misinterpretations.
The author suggests that the “radical” part of EA is the idea that we are just as obligated to help a child drowning in a faraway pond as in a nearby one:
The morally radical suggestion is that our ability to act so as to produce value anywhere places the same moral demands on us as does our ability to produce value in our immediate practical circumstances
She notes that what she sees as the EA moral view excludes “virtue-oriented” or subjective moral positions, and lists several views (e.g. “Kantian constructivist”) that are restricted if one takes what she sees as the EA moral view. She maintains that such views, which (apparently) have a long history at Oxford, have a lot to offer in the way of critique of EA.
Institutional critique
In a nutshell, EA focuses too much on what it can measure, and what it can measure are incrementalist approaches that ignores the “structural, political roots of global misery.” The author says that the EA responses to this criticism (that even efforts at systemic change can be evaluated and judged effective) are fair. She says that these responses constitute a claim that the institutional critique is a criticism of how closely EA hews to its tenets, rather than of the tenets themselves. She disagrees with this claim.
Philosophical critique
This critique holds that EAs basically misunderstand what morality is—that the point of view of the universe is not really possible. The author argues that attempting to take this perspective actively “deprives us of the very resources we need to recognise what matters morally”—in other words, taking the abstract view eliminates moral information from our reasoning.
The author lists some of the features of the worldview underpinning the philosophical critique. Acting rightly includes:
acting in ways that are reflective of virtues such as benevolence, which aims at the well-being of others
acting, when appropriate, in ways reflective of the broad virtue of justice, which aims at an end—giving people what they are owed—that can conflict with the end of benevolence
She concludes:
In a case in which it is not right to improve others’ well-being, it makes no sense to say that we produce a worse result. To say this would be to pervert our grasp of the matter by importing into it an alien conception of morality … There is here simply no room for EA-style talk of “most good.”
So in this view there are situations in which morality is more expansive than the improvement of others’ well-being, and taking the abstract view eliminates these possibilities.
The philosophical-institutional critique
The author combines the philosophical and institutional critiques. The crux of this view seems to be that large-scale social problems have an ethical valence, and that it’s basically impossible to understand or begin to rectify them if you take the abstract (god’s eye) view, which eliminates some of this useful information:
Social phenomena are taken to be irreducibly ethical and such that we require particular modes of affective response to see them clearly … Against this backdrop, EA’s abstract epistemological stance seems to veer toward removing entirely it from the business of social understanding.
This critique maintains that it’s the methodological tools of EA (“economic modes of reasoning”) that block understanding, and articulates part of the worldview behind this critique:
Underlying this charge is a very particular diagnosis of our social condition. The thought is that the great social malaise of our time is the circumstance, sometimes taken as the mark of neoliberalism, that economic modes of reasoning have overreached so that things once rightly valued in a manner immune to the logic of exchange have been instrumentalised.
In other words, the overreach of economic thinking into moral philosophy is a kind of contamination that blinds EA to important moral concerns.
Conclusion
Finally, the author contends that EA’s framework constrains “available moral and political outlooks,” and ties this to the lack of diversity within the movement. By excluding more subjective strains of moral theory, EA excludes the individuals who “find in these traditions the things they most need to say.” In order for EA to make room for these individuals, it would need to expand its view of morality.
I’m curious to hear Michael’s response, but also interested to hear more about why you think this. I have the opposite intuition- presumably 1910 had its fair share of moonshots which seemed crazy at the time and which turned out, in fact, to be basically crazy, which is why we haven’t heard about them.
A portfolio which included Ford and Edison would have performed extremely well, but I don’t know how many possible 1910 moonshot portfolios would have included them or would have weighted them significantly enough to outperform the many failed other moonshots.
I’m really excited to see this!
I understand that, lead abatement itself aside, the alkalinity of the water supply seems to have an impact on lead absorption in the human body and its attendant health effects. I’m curious whether (1) this impact is significant (2) whether interventions to change the pH of water are competitive in terms of cost-effectiveness with other types of interventions and (3) whether this has been tried.
The venue of advocacy here will depend at least in part on the policies you decide are worth advocating. Even with hundreds of grassroots volunteers, it will be hard to ensure the fidelity of the message you are trying to communicate. It is hard at first blush to imagine how greater attention to pandemic preparedness could do harm, but it is not difficult that simply exhorting government to “do something” could have bad consequences.
Given the situation, it seems likely that governments preparing for future pandemics without clear guidance will prepare for a repeat of the pandemic that is already happening, rather than a different and worse one in future.
Once you select certain highly effective policy worth advocating (for example, an outbreak contingency fund), that’s the stage at which to determine the venue and the tactic. I’m not a bio expert, but it’s not difficult to imagine that once you identify a roster of potential policies, the most effective in expectation may involve, for example, lobbying Heathrow Airport Holdings or the Greater London Authority rather than Parliament.
The EA community overrates the predictive validity and epistemic superiority of forecasters/forecasting.
This seems to be true and also to be an emerging consensus (at least here on the forum).
I’ve only been forecasting for a few months, but it’s starting to seem to me like forecasting does have quite a lot of value—as valuable training in reasoning, and as a way of enforcing a common language around discussion of possible futures. The accuracy of the predictions themselves seems secondary to the way that forecasting serves as a calibration exercise. I’d really like to see empirical work on this, but anecdotally it does feel like it has improved my own reasoning somewhat. Curious to hear your thoughts.
To clarify, I’m not sure this is likely to be the best use of any individual EA’s time, but I think it can still be true that it’s potentially a good use of community resources, if intelligently directed.
I agree that perhaps “constitutionally” is too strong—what I mean is that EAs tend (generally) to have an interest in / awareness of these broadly meta-scientific topics.
In general, the argument I would make would be for greater attention to the possibility that mainstream causes deserve attention and more meta-level arguments for this case (like your post).