As a software developer, I’m intimately familiar with the phenomenon of tiny subtle mistakes causing unexpected catastrophes… writing production software trains a certain paranoia when dealing with computers that’s hard to get elsewhere. Sometimes I wonder if most scientific simulation code has subtle errors in it that invalidate the results because the researchers don’t have enough programming experience to realize how easy it is to write code that’s incorrect and simulations are black boxes where errors don’t necessarily manifest themselves immediately & visibly.
One interesting idea might be for each person in your group to try to generate frameworks for evaluating a proposed regulation independently, and meet to compare spreadsheets later. Probably this would work even better with a group that had a diverse set of backgrounds (educationally, politically, etc.)
Because politics is apparently inherently interesting to almost everyone, you might be able to start blogging about whatever regulations you are looking in to and extract free cognitive labor from your readers to provide an additional perspective, critique your research, etc. In fact, maybe there is low-hanging fruit in just redirecting the attention of intelligent policy bloggers to policies that can actually be impacted in the near term instead of just whatever the fashionable discussion topics of the day are, then writing up any consensus they come to as a comment.
(This is a tougher one since there are also downsides to having politics discussions online, which I wrote about previously. Not sure where I currently stand, but leaning towards a blog promoting a high standard of discussion being the best option.)
Broadly my intuition is that some policy recommendations have the potential for 10x or 100x the impact relative to others, so you are better off aiming to produce a small number of high-quality, highly-targeted recommendations than a large number of lower-quality ones, at least in the long run. (Incidentally, are animal issues on your radar?)
I think we will start blogging in a limited capacity about regulations we are seriously considering working on and some that we considered and then dismissed. We probably aren’t going to blog about every regulation we look at since there are so many. Some comments are likely to be far more impactful than others, however the comments that are likely to have the most impact are also likely to have slower feedback and no nearby certain deadlines for implementation.
Our current priority list seems to be:
-Network early to get expert feedback and assistance
-Produce lots of comments early to get feedback and learn how to make influential comments
-Focus on high impact comments toward the end of our project trial
To some degree, these priorities can get jumbled by time sensitive opportunities, but as an overall aim, we think this is correct for moving forward.
Animal issues are on our radar, but I have yet to see anything lately relevant to factory farming of cows, pigs, or chickens. We have seen a lot of proposed rules about fisheries and species protection, but didn’t have the expertise to go after them yet. If there are experts we could consult on animal issues, they would likely be helpful unless their way of sorting policies into “worth going after” and “not worth it” is the same as ours and nothing new comes up/is noticed.
Some of these questions require semi-detailed responses, so I will respond with a few different comments. Richard had some examples/anecdotes about the level of impact policy comments could have:
“A good recent example of FDA making major changes as a result of public comment is the Food Safety Modernization Act (FSMA) Produce Safety rule. This rule was extensively changed as a result of feedback from the public, mostly the affected farmers. Some of the comments were merely self-interested lobbying, but some pointed out where FDA’s lack of understanding of specific situations would have led to an inefficient regulation. The changes were so extensive that FDA had to publish a reproposal of the rule.
This is a large and complex rule, so you can choose how much detail you want to look into. The overview is here:
You can do a text search in the Federal Register document for the phrase “Relevant Comments” to see the FDA responses to comments on various subjects and how those comments changed the regulation.
This is a link to the economic analysis of the reproposal, which has details on the costs and benefits of the changes in the reproposal:
As for spreadsheets, we could go through a cluster thinking way of producing estimates, but I am under the impression this would take a lot more time per person, and then when comparing spreadsheets at the end, we’d be finding errors that would have been easier to handle earlier if we worked together earlier and got faster feedback.
There certainly is value to avoiding groupthink though. Overall I do think using multiple sequential techniques could be a rather rigorous way to evaluate something, and make a very good comment, but we are also trying to get useful feedback by making a lot of comments and we are already commenting much slower than we desired to because of outreach.
I would like to do this, but I think it would be useful to have more comment drafting workers if we want to do this a lot.
This sounds like a pretty cool project. Some thoughts:
Do we have evidence, whether quantitative or anecdotal, about the level of impact that policy comments have?
Reading about your spreadsheet reminded me of this post on sequence thinking vs cluster thinking. I think spreadsheets are inherently sequence-ish in nature; a change to a single field can completely alter the result.
As a software developer, I’m intimately familiar with the phenomenon of tiny subtle mistakes causing unexpected catastrophes… writing production software trains a certain paranoia when dealing with computers that’s hard to get elsewhere. Sometimes I wonder if most scientific simulation code has subtle errors in it that invalidate the results because the researchers don’t have enough programming experience to realize how easy it is to write code that’s incorrect and simulations are black boxes where errors don’t necessarily manifest themselves immediately & visibly.
One interesting idea might be for each person in your group to try to generate frameworks for evaluating a proposed regulation independently, and meet to compare spreadsheets later. Probably this would work even better with a group that had a diverse set of backgrounds (educationally, politically, etc.)
Because politics is apparently inherently interesting to almost everyone, you might be able to start blogging about whatever regulations you are looking in to and extract free cognitive labor from your readers to provide an additional perspective, critique your research, etc. In fact, maybe there is low-hanging fruit in just redirecting the attention of intelligent policy bloggers to policies that can actually be impacted in the near term instead of just whatever the fashionable discussion topics of the day are, then writing up any consensus they come to as a comment.
(This is a tougher one since there are also downsides to having politics discussions online, which I wrote about previously. Not sure where I currently stand, but leaning towards a blog promoting a high standard of discussion being the best option.)
Broadly my intuition is that some policy recommendations have the potential for 10x or 100x the impact relative to others, so you are better off aiming to produce a small number of high-quality, highly-targeted recommendations than a large number of lower-quality ones, at least in the long run. (Incidentally, are animal issues on your radar?)
I think we will start blogging in a limited capacity about regulations we are seriously considering working on and some that we considered and then dismissed. We probably aren’t going to blog about every regulation we look at since there are so many. Some comments are likely to be far more impactful than others, however the comments that are likely to have the most impact are also likely to have slower feedback and no nearby certain deadlines for implementation.
Our current priority list seems to be: -Network early to get expert feedback and assistance -Produce lots of comments early to get feedback and learn how to make influential comments -Focus on high impact comments toward the end of our project trial
To some degree, these priorities can get jumbled by time sensitive opportunities, but as an overall aim, we think this is correct for moving forward.
Animal issues are on our radar, but I have yet to see anything lately relevant to factory farming of cows, pigs, or chickens. We have seen a lot of proposed rules about fisheries and species protection, but didn’t have the expertise to go after them yet. If there are experts we could consult on animal issues, they would likely be helpful unless their way of sorting policies into “worth going after” and “not worth it” is the same as ours and nothing new comes up/is noticed.
Some of these questions require semi-detailed responses, so I will respond with a few different comments. Richard had some examples/anecdotes about the level of impact policy comments could have:
“A good recent example of FDA making major changes as a result of public comment is the Food Safety Modernization Act (FSMA) Produce Safety rule. This rule was extensively changed as a result of feedback from the public, mostly the affected farmers. Some of the comments were merely self-interested lobbying, but some pointed out where FDA’s lack of understanding of specific situations would have led to an inefficient regulation. The changes were so extensive that FDA had to publish a reproposal of the rule.
This is a large and complex rule, so you can choose how much detail you want to look into. The overview is here:
http://www.fda.gov/Food/GuidanceRegulation/FSMA/ucm334114.htm
and it has links to further information. The full Federal Register notice is here:
https://www.federalregister.gov/articles/2014/09/29/2014-22447/standards-for-the-growing-harvesting-packing-and-holding-of-produce-for-human-consumption
You can do a text search in the Federal Register document for the phrase “Relevant Comments” to see the FDA responses to comments on various subjects and how those comments changed the regulation.
This is a link to the economic analysis of the reproposal, which has details on the costs and benefits of the changes in the reproposal:
http://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/EconomicAnalyses/UCM442801.pdf
Another rule where public comments led to major changes and a repropsal is the FSMA Preventative Controls rule:
http://www.fda.gov/Food/GuidanceRegulation/FSMA/ucm334115.htm
However, understanding the issues in that rule requires a bit more specialized knowledge.”
As for spreadsheets, we could go through a cluster thinking way of producing estimates, but I am under the impression this would take a lot more time per person, and then when comparing spreadsheets at the end, we’d be finding errors that would have been easier to handle earlier if we worked together earlier and got faster feedback.
There certainly is value to avoiding groupthink though. Overall I do think using multiple sequential techniques could be a rather rigorous way to evaluate something, and make a very good comment, but we are also trying to get useful feedback by making a lot of comments and we are already commenting much slower than we desired to because of outreach.
I would like to do this, but I think it would be useful to have more comment drafting workers if we want to do this a lot.