Error

Wow! It’s really great to see such an in-depth response to the definitional and foundational work that’s been taking place around IIDM over the past year, plus I love your hand-drawn illustrations! As the author or co-author of several of the pieces you cited, I thought I’d share a few thoughts and reactions to different issues you brought up. First, on the distinctions and delineations between the value-neutral and value-oriented paradigms (I like those labels, by the way):

• I don’t quite agree that Jess Whittlestone’s problem profile for 80K falls into what you’re calling the “value neutral” category, as she stresses at several points the potential of working with institutions that are working on “important problems” or similar. For example, she writes: “Work on ‘improving decision-making’ very broadly isn’t all that neglected. There are a lot of people, in both industry and academia, trying out different techniques to improve decision-making....However, there seems to be very little work focused on...putting the best-proven techniques into practice in the most influential institutions.” The definition of “important” or “influential” is left unstated in that piece, but from the context and examples provided, I read the intention as one of framing opportunities from the standpoint of broad societal wellbeing rather than organizations’ parochial goals.

• This segues nicely into my second response, which is that I don’t think the value-neutral version of IIDM is really much of a thing in the EA community. CES is sort of an awkward example to use because a core tenet of democracy is the idea that one citizen’s values and policy preferences shouldn’t count more than another’s; I’d argue that the impartial welfarist perspective that’s core to EA philosophy is rooted in similar ideas. By contrast, I think people in our community are much more willing to say that some organizations’ values are better than others, both because organizations don’t have the same rights as human beings and also because organizations can agglomerate disproportionate power more easily and scalably than people. I’ve definitely seen disagreement about how appropriate or effective it is to try to change organizations’ values, but not so much about the idea that they’re important to take into account in some way. [Edit: no longer endorsed, see comment below.]

• There is a third type of value-oriented approach that you don’t really explore but I think is fairly common in the EA community as well as outside of it: looking for opportunities to make a positive impact from an impartial welfarist perspective on a smaller scale within a non-aligned organization (e.g., by working with a single subdivision or team, or on one specific policy decision) without trying to change the organization’s values in a broader sense.

I appreciated your thought-provoking exploration of the two indirect pathways to impact you proposed. Regarding the second pathway (selecting which institutions will survive and flourish), I would propose that an additional complicating factor is that non-value-aligned institutions may be less constrained by ethical considerations in their option set, which could give them an advantage over value-aligned institution from the standpoint of maximizing power and influence.

I did have a few critiques about the section on directly improving the outcomes of institutions’ decisions:

• I think the 2x2 grid you use throughout is a bit misleading. It looks like you’re essentially using decision quality as a proxy for institutional power, and then concluding that intentions x capability = outcomes. But decision quality is only one input into institutional capabilities, and in the short term is dominated by institutional resources—e.g., the government of Denmark might have better average decision quality than the government of the United States, but it’s hard to argue that Denmark’s decisions matter more. For that reason, I think that selecting opportunities on the basis of institutional power/​positioning is at least as important as value alignment. The visualization approach you took in the “A few overwhelmingly harmful institutions” graph seems to be on the right track in this respect.

• One issue you don’t really touch on except in a footnote is the distinction between stated values and de facto values for institutions, or internal alignment among institutional stakeholders. For example, consider a typical private health insurer in the US. In theory, its goal is to increase the health and wellbeing of millions of patients—a highly value-aligned goal! Yet in practice, the organization engages in many predatory practices to serve its own growth, enrichment of core stakeholders, etc. So is this an altruistic institution or not? And does bringing its (non-altruistic) actions into greater alignment with its (altruistic) goals count as improving decision quality or increasing value alignment under your paradigm?

While overall I tend to agree with you that a value-oriented approach is better, I don’t think you give a fair shake to the argument that “value-aligned institutions will disproportionately benefit from the development of broad decision-making tools.” It’s important to remember that improving institutional decision-making in the social sector and especially from an EA perspective is a very recent concept. The professional world is incredibly siloed, and it’s not hard at all for me to imagine that ostensibly publicly available resources and tools that anyone could use would, in practice, be distributed through networks that ensure disproportionate adoption by well-intentioned individuals and groups. I believe that something like this is happening with Metaculus, for example.

One final technical note: you used “generic-strategy” in a different way that we did in the “Which Institutions?” post—our definition imagines a specific organization that is targeted through a non-specific strategy, whereas yours imagines a specific strategy not targeted to any specific organization. I agree that the latter deserves its own label, but suggest a different one than “generic-strategy” to avoid confusion with the previous post.

I’ve focused mostly on criticisms here for the sake of efficiency, but I really was very impressed with this article and hope to see more writing from you in the future, on this topic and others!

I’ve been skeptical of much of the IIDM work I’ve seen to date. By contrast, from a quick skim, this piece seemed pretty good to me because it has more detailed models of how IIDM may or may not be useful, and is opinionated in a few non-obvious but correct-seeming ways. I liked this a lot – thanks for publishing!

Like, if anyone feels like handing out prizes for good content, I’d recommend that this piece of work should receive a $10k prize (though perhaps I’d want to read it in full before fully recommending).

Hi Lizka, WOW – Thank you for writing this. Great to see Rethink Priorities working on this. Absolutely loving the diagrams here.

I have worked in this space for a number of years mostly here, have been advocating for this cause within EA since 2016 and advised both Jess/​80K and the effective institutions project on their writeups. Thought I would give some quick feedback. Let me know if it is useful.

I thought your disentanglement did a decent job. Here are a few thought I had on it.

1. I really like how you split IIDM into “A technical approach to IIDM” and “A value-aligning approach to IIDM.”

2. However I found the details of how you split it to be very confusing. It left me quite unsure what goes into what bucket. For example intuitively I would see increasing the “accuracy of governments” (i.e. aligning governments with the interests of the voters) as “value-aligning” yet you classify it as “technical”.

3. That said despite this, I very much agreed with the conclusion that “value-oriented IIDM makes more sense than value-neutral IIDM” and the points you made to that effect.
   

4. I didn’t quite understand what “(1) IIDM can improve our intellectual and political environment” was really getting at. My best guess is that by (1) you mean work that only indirectly leads to “(3) improved outcomes”. So value-oriented (1) would look like general value spreading. Is that correct?
   

5. I agree with “for the sake of clarity … we should generally distinguish between ‘meta EA’ work and IIDM work”. That said I think it is worth bearing in mind that on occasion the approaches might not be that different. For example I have been advising the UK government on how to asses high-impact risks which is relevant for EAs too.*
   

6. One institution can have many parts. Might be a thing to highlight if you do more disentanglement. E.g. Is a new office for future generations within a government, a new institution or improving an existing institution?
   

One other thought I had whilst reading.

• I think it is important not to assign value to IIDM based on what is “predictable”.
  
  For example you say “it would be extremely hard to produce candidate IIDM interventions that would have sufficiently predictable outcomes via this pathway, as the outcomes would depend on many very uncertain factors.” Predictions do matter but one of the key cases for IIDM is that it offers a solution to the unpredictable, the unknown unknows, to the uncertainty of the EA (and especially longtermist) endeavour. All the advice on dealing with high-uncertainty and things that are hard to predict suggest that interventions like IIDM are the kinds of interventions that should work – as set out by Ian David Moss here (from this piece).

Finally, at points you seemed uncertain about tractability of this work. I wanted to add that so far I have found it much much easier than I expected. Eg you say “it is possible that shifting the aims of institutions is generally very difficult or that the potential benefits from institutions is overwhelmingly bottlenecked by decision-making ability, rather than by the value-alignment of institutions’ existing aims”. (I am perhaps still confused about what you would count as shifting aims Vs decision-making ability see my point 1. above, but) my rough take on this is that I have found shifting the aims of government to be fairly easy and that there are not too many decision-making bottlenecks.

So super excited to see more EA work in this space.

* Oddly enough, despite being in EA for years, I think I have found it easier to influence the UK government to get better at risk identification work than the EA community. Not sure what to do about that. Just wanted to say that I would love to input if RP is working in this space.

Nice post : )

I mostly agree with your points, though am a bit more optimistic than it seems like you are about untargeted, value-neutral IIDM having a positive impact.

Your skepticism about this seems to be expressed here:

And yet, it seems possible that there are some institutions that cause an overwhelming amount of harm (e.g. the farming industry or some x-risk-increasing endeavors like gain-of-function research), and that the value-neutral version of IIDM fails to take that into account.

I think this is true, but it still seems like the aims of institutions are pro-social as a general matter—x-risk and animal suffering in your examples are side effects that aren’t means to the ends of the institutions, which are ‘increase biosecuirty’ and ‘make money’, and if improving decisionmaking helps orgs get at their ends more efficiently then we should think they will have fewer bad side effects if they have better decisonmaking. Also generally orgs’ aims (e.g. “make money”) will presuppose the firm’s, and therefore humanity’s survival, so it still seems good to me as a general matter for orgs to be able to pursue their aims more effectively.

Really nice and useful exploration, and I really liked your drawings.

(a) Maybe the average/​median institution’s goals are already aligned with public good

FWIW, I intuitively would’ve drawn the institution blob in your sketch higher, i.e. I’d have put fewer than (eyeballing) 30% of institutions in the negatively aligned space (maybe 10%?). In moments like this, including a quick poll into the forum to get a picture what others think would be really useful.

However, I don’t see a clear argument for how an abstract intervention that improves decision-making would also incidentally improve the value-alignment of an institution.

Other spontaneous ideas, besides choosing more representative candidates:

• increased coherence of the institution could lead to an overall stronger link between its mandate and its actions

• increased transparency and coherence could reduce corruption and rent-seeking

I know of efficient and technologically progressive institutions that seem extremely harmful, and of benign, well-meaning institutions that are slow to change and inefficient

Given what I said beforehand, I’d be interested in learning more about examples of harmful institutions that have generally high capacity.

An update: after a bit of digging, I discovered this post, “Should marginal longtermist donations support fundamental or intervention research?”, which contains a discussion on a topic that is quite close to “should EA value foundational (science/​decision theory) research,” (in the pathway (1) section of my post). The conclusions of the post I found do not fit into my vague impressions of “the consensus.” In particular, the conclusion of that post is that longtermist research hours should often be spent on fundamental research (which is defined by its goals).

I’m moderately confident that, from a longtermist perspective, $1M of additional research funding would be better allocated to fundamental rather than intervention research (unless funders have access to unusually good intervention research opportunities, but not to unusually good fundamental research opportunities)

(Disclaimer: the author, Michael, is employed at Rethink Priorities, where I am interning. I don’t know if he still endorses this post or its conclusions, but the post seems relevant here and very valuable as a reference.)

Thanks! Some brief ‘deep’ counterpoints. I don’t see how “decision-making quality” and
“values” or/​which is implied by “value alignment” can be orthogonal to each other, thus facilitating the above graphs. To my mind bad values would promote bad decisions. (And to my mind bad values would have some relation and thus alignment with ‘our’ values, hopefully a negative one, but most likely not a priori without a relation). Relatedly, I also don’t really believe in the existence of “value-neutrality”, and moreover I think it is a dangerous, or more mildly counter-productive, concept to deploy (with effects that are misleadingly regarded as ‘neutral’ and perhaps attended less to), e.g. the economy might be—or often is—regarded as neutral, yet very significantly disregards the interests of future generations, or non-human animals.

I suppose the post’s content fits with moral relativists’ and anti-realists’ worldviews, but with more difficulty with moral realists’, or then just to relate to a way that people often ‘happen to talk’.

Follow-up: Perhaps to put altruism and effectiveness—properly understood—on the y-axis and x-axis respectively would be better, i.e. communicate what we would want to communicate and not suffer from the above-mentioned shortcomings?