I think someone should research policy changes in democratic countries which counterfactually led to the world getting a lot better or worse (under a range of different moral theories, and under public opinion), and the extent to which these changes were technocratic or populist. This would be useful to establish the track records of technocracy and populism, giving us a better reason to generally lean one way or the other.
This is exactly the kind of thing that I think won’t work, because reality is underpowered.
I forgot to link this earlier, but it turns out that some such research already exists (minus the stipulation that it has to be in democratic countries, but I don’t think this is necessarily a fatal problem; there are key similarities with politics in non-democratic countries). In 2009, Daron Acemoglu (a highly-respected-including-by-EAs academic who studies governance) and some other people wrote a paper [PDF] arguing that the First French Empire created a natural experiment, and examining the results. Scott reviewed it in a follow-up post to his earlier exchange with Weyl. The authors’ conclusion (spoilered because Scott’s post encourages readers to try to predict the results in advance) is that
technocratic-ish policies got better results.
I consider this moderately strong evidence against heuristics in the opposite direction, but very weak evidence in favor of heuristics in the same direction. There are quite a lot of caveats, some of which Scott gets into in the post. One of these is that the broader technocracy-vs.-populism question subsumes a number of other heuristics, which, in real life, we can apply independently of that single-axis variable. (His specific example might be controversial, but I can think of others that are harder to argue with, such as (on the technocratic side) “policies have to be incentive-compatible”, or (on the populist side) “don’t ignore large groups of people when they tell you you’ve missed something”.) Once we do that, the value of a general catch-all heuristic in one direction or the other will presumably be much diminished.
Also, there are really quite a lot of researcher degrees-of-freedom in a project like this, which makes it very hard to have any confidence that the conclusions were caused by the underlying ground truth and not by the authors’ biases. And just on a statistical level, sample sizes are always going to be tiny compared to the size of highly multi-dimensional policyspace.
So that’s why I’m pessimistic about this research program, and think we should just try to figure stuff out on a case-by-case basis instead, without waiting for generally-applicable results to come in.
Since you mentioned it, I should clarify that I have no strong opinion on whether EA should be more technocratic or more populist on the current margin. (Though it’s probably fair to say that I’m basically in favor of the status quo, because arguments against it mostly consist of claims that EA has missed something important and obvious, and I tend to find these unpersuasive. I suppose one could argue this makes me pro-technocracy, if one thought the status quo was highly technocratic.) In any case, my contention is that it’s not a crucial consideration.
I think we are disagreeing in a general sense about the usefulness of imprecise and unreliable, but systematically obtained answers to big questions, when trying to answer smaller sub-questions. If we think these answers are less useful, we are less likely to decide that ‘technocracy vs populism in general’ is a crucial consideration. If we think these answers are more useful, we are more likely to decide that ‘technocracy vs populism in general’ is a crucial consideration.
I do agree the conclusion of Acemoglu’s paper (admittedly, it is too long for me to read) is only weak evidence in favour of more technocracy, but if other papers were able to identify more natural experiments and came to similar conclusions, in theory I think that could generate enough evidence for ‘more technocracy’ (or ‘more populism’) to be a sufficiently strong prior / heuristic to be useful when looking at individual cases, which is why I still think ‘technocracy vs populism’ is a crucial consideration.
Update: Having read another comment, it seems likely that expert opinion most replaces other expert opinion in the context of policymaking. That changes my mind on whether technocracy vs populism is a crucial consideration, since it is only relevant to ‘promoting evidence-based policy’, a very minor EA cause area.
This is exactly the kind of thing that I think won’t work, because reality is underpowered.
I forgot to link this earlier, but it turns out that some such research already exists (minus the stipulation that it has to be in democratic countries, but I don’t think this is necessarily a fatal problem; there are key similarities with politics in non-democratic countries). In 2009, Daron Acemoglu (a highly-respected-including-by-EAs academic who studies governance) and some other people wrote a paper [PDF] arguing that the First French Empire created a natural experiment, and examining the results. Scott reviewed it in a follow-up post to his earlier exchange with Weyl. The authors’ conclusion (spoilered because Scott’s post encourages readers to try to predict the results in advance) is that
technocratic-ish policies got better results.
I consider this moderately strong evidence against heuristics in the opposite direction, but very weak evidence in favor of heuristics in the same direction. There are quite a lot of caveats, some of which Scott gets into in the post. One of these is that the broader technocracy-vs.-populism question subsumes a number of other heuristics, which, in real life, we can apply independently of that single-axis variable. (His specific example might be controversial, but I can think of others that are harder to argue with, such as (on the technocratic side) “policies have to be incentive-compatible”, or (on the populist side) “don’t ignore large groups of people when they tell you you’ve missed something”.) Once we do that, the value of a general catch-all heuristic in one direction or the other will presumably be much diminished.
Also, there are really quite a lot of researcher degrees-of-freedom in a project like this, which makes it very hard to have any confidence that the conclusions were caused by the underlying ground truth and not by the authors’ biases. And just on a statistical level, sample sizes are always going to be tiny compared to the size of highly multi-dimensional policyspace.
So that’s why I’m pessimistic about this research program, and think we should just try to figure stuff out on a case-by-case basis instead, without waiting for generally-applicable results to come in.
Since you mentioned it, I should clarify that I have no strong opinion on whether EA should be more technocratic or more populist on the current margin. (Though it’s probably fair to say that I’m basically in favor of the status quo, because arguments against it mostly consist of claims that EA has missed something important and obvious, and I tend to find these unpersuasive. I suppose one could argue this makes me pro-technocracy, if one thought the status quo was highly technocratic.) In any case, my contention is that it’s not a crucial consideration.
Thank you for explaining all of this.
I think we are disagreeing in a general sense about the usefulness of imprecise and unreliable, but systematically obtained answers to big questions, when trying to answer smaller sub-questions. If we think these answers are less useful, we are less likely to decide that ‘technocracy vs populism in general’ is a crucial consideration. If we think these answers are more useful, we are more likely to decide that ‘technocracy vs populism in general’ is a crucial consideration.
I do agree the conclusion of Acemoglu’s paper (admittedly, it is too long for me to read) is only weak evidence in favour of more technocracy, but if other papers were able to identify more natural experiments and came to similar conclusions, in theory I think that could generate enough evidence for ‘more technocracy’ (or ‘more populism’) to be a sufficiently strong prior / heuristic to be useful when looking at individual cases, which is why I still think ‘technocracy vs populism’ is a crucial consideration.
Update: Having read another comment, it seems likely that expert opinion most replaces other expert opinion in the context of policymaking. That changes my mind on whether technocracy vs populism is a crucial consideration, since it is only relevant to ‘promoting evidence-based policy’, a very minor EA cause area.