Thank you so much for writing this. This is one of my central areas of interest, and I’ve been puzzled by the comparative lack of resources expended by the EA community on institutional decision-making given the apparently high degree of importance accorded to it by many of us.
This is a great guide. I agree that the central question here is whether or not deliberative democracy leads to better outcomes. If it does, or even if probably does, it seems that it’s easily one of the highest-value potential cause areas, since the levers that influence many other cause areas are within reach of democratic polities.
With that in mind, it seems clear to me that the primary way in which deliberation is EA-relevant is as a large-scale decision making mechanism. So it seems like relatively small-scale uses are not very important to us, and it also seems like information about these successes may not be useful given the likelihood that instituting these mechanisms at a large scale is likely to present very different problems of kind, not of degree. I’d love to hear your thoughts on that.
I have a few other thoughts about this review, and I’d like to hear your responses if you have the time.
• Basically all of the cross-country comparisons in this review suffer from reverse causation. Countries that have lots of deliberation and good outcomes don’t necessarily have the former causing the latter; the former could rather be just another instance of the latter. As enthused as I am about deliberative democracy, this scenario seems just as likely as the causal one. Is there any reason to view these correlations as suggestive of a causal effect?
• It seems like this review contains a relative paucity of research supporting the null hypothesis that deliberation does not improve decision making (or, for that matter, the alternative hypothesis that it actually worsens decision making). Were you unable to find studies taking this position? If not, how worried are you about the file-drawer effect here?
• Based on your reading of all this evidence, I’d love to hear your subjective first impressions- what do you personally feel is the “best bet” for enacting deliberative democracy on a large scale somewhere besides China? How far do you think this could feasibly go and how long would you expect such a change to take? Very wide confidence bands on these estimates are fine, of course.
It seems like this review contains a relative paucity of research supporting the null hypothesis that deliberation does not improve decision making (or, for that matter, the alternative hypothesis that it actually worsens decision making). Were you unable to find studies taking this position? If not, how worried are you about the file-drawer effect here?
One approach worth considering when engaging with topics like these is to adopt a systematic review rather than a narrative one. The part which would be particularly helpful re. selection worries is having a pre-defined search strategy, which can guard against inadvertently gathering a skewed body of evidence. (If there are lots of quantitative results, there are statistical methods to assess publication bias, but that typically won’t be relevant here—that said, there are critical appraisal tools you can use to score study quality which offers indirect indication, as well as value generally in getting a sense of how trustworthy the consensus of the literature is).
These are a lot more work, but may be worth it all the same. With narrative reviews on topics where I expect there to be a lot of relevant (primary) literature, I always have the worry one could tell a similarly persuasive story for many different conclusions depending on whether the author happened upon (or was more favourably disposed to) one or other region of the literature.
‘Quick and dirty’ systematisation can also help: e.g. ‘I used search term [x] in google scholar and took the first 20 results—of the relevant ones, 8 were favourable, and 1 was neutral’.
I’ve always wondered about the “first N Google results” strategy. Even in the absence of a file-drawer effect, isn’t this more likely to turn up papers making positive claims (on the assumption that e.g. rejections of the null are more likely to be cited than inconclusive results)?
I’m not sure how google scholar judges relevance (e.g. I can imagine eye-catching negative results also being boosted up the rankings) but I agree it is a source of distortion—I’d definitely offer it as ‘better than nothing’ rather than good. (Perhaps one tweak would be sample by a manageable date range rather than relevance, although one could worry about time trends).
A better option (although it has some learning curve and onerousness) is query a relevant repository, export all the results, and take a random sample from these.
Cross-country comparisons & reverse causation:
I agree that the cross country comparisons do not offer much causal inference and as noted in the piece would be an interesting area to find more or do more research in.
The file-drawer effect:
With regard to your and other commenters points on study selection and giving the other side a fair hearing, I have made some updates throughout the text, but especially in the “Impact of deliberation” preamble to emphasise some counterpoints or mixed findings in the literature, in addition to the already existing section on “Reasons to doubt deliberative mini publics”. Even after clarifying some uncertainty around the effects of deliberation and underscoring the need for more high-quality research in this area, we think the existing evidence base for deliberative reforms compares favourably with other interventions of this sort. For me, a greater concern is around institutionalising deliberation so that the effects have direct impact.
Enacting deliberative democracy on a large scale:
I am very hesitant to make a proposal for “enacting deliberative democracy on a large scale somewhere besides China” because as noted in the piece deliberative democracy is a much larger concept of a macro-political system consisting of various sites of deliberation compared to more limited democratic (or undemocratic) acts of deliberation . To me, the former is not obviously the best or most tractable proposal since it could involve changing the entire political system and ecosystem of institutions. There are possibilities for small scale deliberation to inform the larger system (without having to have mass deliberation) e.g. small scale deliberative councils or polling across a polity can offer advice as in the case of AmericaSpeaks, or inform the wider electorate as in the case of Citizens Initiative Reviews, and of course one could consider variants on Rupert Read’s Guardians of the future scheme, where a small number deliberate and then either advise or have an array of different powers to influence the legislature (or to influence the wider public through their oversight). These do not seem to be any harder to implement than other proposals such as approval voting or age-weighted voting, and have the pro of having already been adopted in a number of polities.
Thank you so much for writing this. This is one of my central areas of interest, and I’ve been puzzled by the comparative lack of resources expended by the EA community on institutional decision-making given the apparently high degree of importance accorded to it by many of us.
This is a great guide. I agree that the central question here is whether or not deliberative democracy leads to better outcomes. If it does, or even if probably does, it seems that it’s easily one of the highest-value potential cause areas, since the levers that influence many other cause areas are within reach of democratic polities.
With that in mind, it seems clear to me that the primary way in which deliberation is EA-relevant is as a large-scale decision making mechanism. So it seems like relatively small-scale uses are not very important to us, and it also seems like information about these successes may not be useful given the likelihood that instituting these mechanisms at a large scale is likely to present very different problems of kind, not of degree. I’d love to hear your thoughts on that.
I have a few other thoughts about this review, and I’d like to hear your responses if you have the time.
• Basically all of the cross-country comparisons in this review suffer from reverse causation. Countries that have lots of deliberation and good outcomes don’t necessarily have the former causing the latter; the former could rather be just another instance of the latter. As enthused as I am about deliberative democracy, this scenario seems just as likely as the causal one. Is there any reason to view these correlations as suggestive of a causal effect?
• It seems like this review contains a relative paucity of research supporting the null hypothesis that deliberation does not improve decision making (or, for that matter, the alternative hypothesis that it actually worsens decision making). Were you unable to find studies taking this position? If not, how worried are you about the file-drawer effect here?
• Based on your reading of all this evidence, I’d love to hear your subjective first impressions- what do you personally feel is the “best bet” for enacting deliberative democracy on a large scale somewhere besides China? How far do you think this could feasibly go and how long would you expect such a change to take? Very wide confidence bands on these estimates are fine, of course.
One approach worth considering when engaging with topics like these is to adopt a systematic review rather than a narrative one. The part which would be particularly helpful re. selection worries is having a pre-defined search strategy, which can guard against inadvertently gathering a skewed body of evidence. (If there are lots of quantitative results, there are statistical methods to assess publication bias, but that typically won’t be relevant here—that said, there are critical appraisal tools you can use to score study quality which offers indirect indication, as well as value generally in getting a sense of how trustworthy the consensus of the literature is).
These are a lot more work, but may be worth it all the same. With narrative reviews on topics where I expect there to be a lot of relevant (primary) literature, I always have the worry one could tell a similarly persuasive story for many different conclusions depending on whether the author happened upon (or was more favourably disposed to) one or other region of the literature.
‘Quick and dirty’ systematisation can also help: e.g. ‘I used search term [x] in google scholar and took the first 20 results—of the relevant ones, 8 were favourable, and 1 was neutral’.
I’ve always wondered about the “first N Google results” strategy. Even in the absence of a file-drawer effect, isn’t this more likely to turn up papers making positive claims (on the assumption that e.g. rejections of the null are more likely to be cited than inconclusive results)?
I’m not sure how google scholar judges relevance (e.g. I can imagine eye-catching negative results also being boosted up the rankings) but I agree it is a source of distortion—I’d definitely offer it as ‘better than nothing’ rather than good. (Perhaps one tweak would be sample by a manageable date range rather than relevance, although one could worry about time trends).
A better option (although it has some learning curve and onerousness) is query a relevant repository, export all the results, and take a random sample from these.
Thanks for your thoughts!
Cross-country comparisons & reverse causation: I agree that the cross country comparisons do not offer much causal inference and as noted in the piece would be an interesting area to find more or do more research in.
The file-drawer effect: With regard to your and other commenters points on study selection and giving the other side a fair hearing, I have made some updates throughout the text, but especially in the “Impact of deliberation” preamble to emphasise some counterpoints or mixed findings in the literature, in addition to the already existing section on “Reasons to doubt deliberative mini publics”. Even after clarifying some uncertainty around the effects of deliberation and underscoring the need for more high-quality research in this area, we think the existing evidence base for deliberative reforms compares favourably with other interventions of this sort. For me, a greater concern is around institutionalising deliberation so that the effects have direct impact.
Enacting deliberative democracy on a large scale: I am very hesitant to make a proposal for “enacting deliberative democracy on a large scale somewhere besides China” because as noted in the piece deliberative democracy is a much larger concept of a macro-political system consisting of various sites of deliberation compared to more limited democratic (or undemocratic) acts of deliberation . To me, the former is not obviously the best or most tractable proposal since it could involve changing the entire political system and ecosystem of institutions. There are possibilities for small scale deliberation to inform the larger system (without having to have mass deliberation) e.g. small scale deliberative councils or polling across a polity can offer advice as in the case of AmericaSpeaks, or inform the wider electorate as in the case of Citizens Initiative Reviews, and of course one could consider variants on Rupert Read’s Guardians of the future scheme, where a small number deliberate and then either advise or have an array of different powers to influence the legislature (or to influence the wider public through their oversight). These do not seem to be any harder to implement than other proposals such as approval voting or age-weighted voting, and have the pro of having already been adopted in a number of polities.
Again thanks!