Strong-upvoted, thank you for the detailed writeup and BOTEC. I currently work in global health policy and your takeaways seem broadly right to me, I would include your reflections on the work itself in that section for folks who jump straight to that.
You handwaved away the internal 1.5 FTE for 6 months * overhead costs of conducting the work, but I’d be remiss not to mention that ~$100M in benefits from your Guesstimate vs order-of-mag $100k internal costs is a 1,000x ROI, within spitting distance of Open Phil’s funding bar.
The most surprising thing I got from your writeup was (emphasis mine)
I looked for some statistics on how many rule change requests end up resulting in rule changes to get a sense of the success rate. 548 rule change requests have been initiated. It’s hard to say how many of them were successful without spending more time, because AEMC doesn’t publish statistics on this, and some of the requests get merged into others. Also, not all rule change requests survive the consultation process intact. It could end up being quite different to what was originally proposed. A little over half of rule changes that have been initiated have commenced (incorporated in the live rules). Overall, I’m reminded of a hits-based giving approach.
Over half! I would’ve ballparked this at 10% give or take, so it’s good to reorient my gut-feel on hit rate to empirics.
I would indeed be keen for you to write more about what you specifically did during the project as per your offer, always good to have more case studies that go into the nuts and bolts for practitioners and folks wanting to test for fit in policy careers.
Thanks for the detailed response! I’ve included a few reflections on the work in the conclusion section. Fair point on the internal costs—I was thinking about this as a cost but not as an impact multiplier from funding. With some more work it could be used as justification for the existence of ECA and why consumers pay their salary. ~$200k seems right for staff time plus overhead.
Yeah, “over half” was quite surprising to me too. I wonder how much of this is because organisations may only lodge a rule change request if they have a decent sense that it is likely to be successful before doing so. If individuals and smaller/outside organisations took my advice and started lodging more rule change requests that ratio would surely change.
Thank you for nudging me to expand on the specfics of what I did. I think I’ll write something more detailed at some point, but for now I’ll just brain dump some dot points on what we did in those 6 months—hopefully that’s helpful for now.
ECA strategic planning to identify the energy consumer issues that are most important, tractable, and within ECA’s wheelhouse (see 3-year plan as the output)
Deep literature review (many rounds of this at each stage—I tried to just become an expert in the topic—I used AI pretty heavily as a learning companion)
Internal brainstorming with seniors to identify the problem and design a solution
Stakeholder mapping to identify who we need to involve, consult, mobilise, etc. (e.g., using IAP2 frameworks, plotting stakeholders on interest vs power axes)
Building a spreadsheet based on stakeholder mapping with details of key organisations including contacts
Identifying the value propositions of our rule change request—i.e. 4 main/unique values that different stakeholders would receive if the rule change were successful
Grouping stakeholders into one of 4 value propositions
Several rounds of seeking and incorporating feedback with key decision makers and stakeholders (sending dot point summaries, drafts, presenting to, meeting with, etc.)
Multiple rounds of internal drafting and review
Final checks (placed a high bar on being accurate and not having typos, etc.)
Lodged rule change
Offered briefings to interested parties
Media releases leading to some articles in industry press
Developed fact sheet to help stakeholders understand issue and lower the bar to making a submission
Social media posts to create awareness and (primarily) to encourage submissions
Running 4 workshops/briefings (one for each value proposition) to mobilise stakeholders to make a submission and collect feedback
Responding to government consultation papers (some more literature review, feedback, brainstorming for solutions to specific issues raised)
Commissioning external expert analysis as needed
I was the project leader for all of this (except the strategic planning which happened before I joined) but didn’t necessarily do all of it myself.
Yes definitely helpful, both for my own thinking and to be able to have something to point others to. With the caveat that learning from success stories requires some sort of survivorship bias adjustment, I think nuts-and-bolts writeups of technical policy reform success stories (as opposed to more high-level guides) are valuable and undersupplied, so if you ever get round to the more detailed writeup that would be great.
Strong-upvoted, thank you for the detailed writeup and BOTEC. I currently work in global health policy and your takeaways seem broadly right to me, I would include your reflections on the work itself in that section for folks who jump straight to that.
You handwaved away the internal 1.5 FTE for 6 months * overhead costs of conducting the work, but I’d be remiss not to mention that ~$100M in benefits from your Guesstimate vs order-of-mag $100k internal costs is a 1,000x ROI, within spitting distance of Open Phil’s funding bar.
The most surprising thing I got from your writeup was (emphasis mine)
Over half! I would’ve ballparked this at 10% give or take, so it’s good to reorient my gut-feel on hit rate to empirics.
I would indeed be keen for you to write more about what you specifically did during the project as per your offer, always good to have more case studies that go into the nuts and bolts for practitioners and folks wanting to test for fit in policy careers.
Thanks for the detailed response! I’ve included a few reflections on the work in the conclusion section. Fair point on the internal costs—I was thinking about this as a cost but not as an impact multiplier from funding. With some more work it could be used as justification for the existence of ECA and why consumers pay their salary. ~$200k seems right for staff time plus overhead.
Yeah, “over half” was quite surprising to me too. I wonder how much of this is because organisations may only lodge a rule change request if they have a decent sense that it is likely to be successful before doing so. If individuals and smaller/outside organisations took my advice and started lodging more rule change requests that ratio would surely change.
Thank you for nudging me to expand on the specfics of what I did. I think I’ll write something more detailed at some point, but for now I’ll just brain dump some dot points on what we did in those 6 months—hopefully that’s helpful for now.
ECA strategic planning to identify the energy consumer issues that are most important, tractable, and within ECA’s wheelhouse (see 3-year plan as the output)
Deep literature review (many rounds of this at each stage—I tried to just become an expert in the topic—I used AI pretty heavily as a learning companion)
Internal brainstorming with seniors to identify the problem and design a solution
Stakeholder mapping to identify who we need to involve, consult, mobilise, etc. (e.g., using IAP2 frameworks, plotting stakeholders on interest vs power axes)
Building a spreadsheet based on stakeholder mapping with details of key organisations including contacts
Identifying the value propositions of our rule change request—i.e. 4 main/unique values that different stakeholders would receive if the rule change were successful
Grouping stakeholders into one of 4 value propositions
Several rounds of seeking and incorporating feedback with key decision makers and stakeholders (sending dot point summaries, drafts, presenting to, meeting with, etc.)
Multiple rounds of internal drafting and review
Final checks (placed a high bar on being accurate and not having typos, etc.)
Lodged rule change
Offered briefings to interested parties
Media releases leading to some articles in industry press
Developed fact sheet to help stakeholders understand issue and lower the bar to making a submission
Social media posts to create awareness and (primarily) to encourage submissions
Running 4 workshops/briefings (one for each value proposition) to mobilise stakeholders to make a submission and collect feedback
Responding to government consultation papers (some more literature review, feedback, brainstorming for solutions to specific issues raised)
Commissioning external expert analysis as needed
I was the project leader for all of this (except the strategic planning which happened before I joined) but didn’t necessarily do all of it myself.
Yes definitely helpful, both for my own thinking and to be able to have something to point others to. With the caveat that learning from success stories requires some sort of survivorship bias adjustment, I think nuts-and-bolts writeups of technical policy reform success stories (as opposed to more high-level guides) are valuable and undersupplied, so if you ever get round to the more detailed writeup that would be great.