This is a follow-up post to our intermediate report on a series of workshops and an ‘Impact Challenge’ at the Dutch Ministry of Foreign Affairs (MFA) to improve institutional decision making.
TL;DR conclusion:
From an impact-perspective we are still uncertain about the magnitude of impact of this project. This will depend on the follow up of the winning idea which is institutionalizing Red teaming exercises to improve decision making. At the time of writing, pilot Red Teaming exercises are being incorporated into the development of two internal strategies of policy. We will keep you informed about the impact.
Outcomes that we have observed:
The Challenge has empowered civil servants that have an interest in working on improving the decision-making process and enhancing policy impact. We provided them with tools and subjects to continue the discussion in their own work environments e.g., a Head of Mission/Ambassador of a Dutch mission abroad introduced the five EA questions to his team.
According to our judgment all three ideas that made the final would contribute to improved decision making quality within the Ministry, if implemented (big uncertainty). The winning idea is followed up on.
The Challenge shows that, when given the time to strategize and think about topics of improving impact, experts themselves come up with good solutions, guided by a team that inspires them with new ways of thinking.
We perceived interest in EA-related ideas within senior civil service ranks and more horizontally, spread over different departments throughout the Ministry. Overall there is an appetite for enhancing policy impact and improving decision making. We believe that similar projects that focus on improving institutional decision-making have the potential to be well received and successful.
In our evaluation discussion we have been encouraged to look into incorporating an adapted version of this project in the Dutch civil servant traineeship program as well as the foreign service traineeship program. This way, EA-related content could be part of the basic mandatory training program of new Dutch civil servants. We are looking into following up on this.
We believe the Effective Altruism community would benefit from spending more resources on research on how to improve coordination and collective action problems in policy and governance (as Larks mentioned in this post). This would increase the effectiveness of facilitation and nudging participants to more effective ideas in similar projects. Simultaneously, as called upon by many EAs recently, we need more action-oriented EAs taking similar initiatives to this experiment, to learn and share lessons on Improving Institutional Decision-Making.
Introduction
Recently we have finished a pilot on organizing an Impact Challenge at the Dutch Ministry of Foreign Affairs with the aim to improve its institutional decision making and raise awareness on policy impact. This experiment gave us and the broader EA community insight into the potential for working on improving institutional decision making. We observed a clear demand in this government institution to improve their work processes and change internal cultures. This is also driven by a shift in norms and values where more space is opening up for diversity and inclusion topics, including a more open approach to diversity of thought. However, important challenges remain related to the intrinsic bureaucratic system as well as deeply rooted cultures where it can be hard for change agents to institutionalize positive change. We want to encourage more EAs within the civil service to start their own intrapreneurial projects and hope to provide some inspiration through this post by sharing our lessons learned. Dutch EAs at other governmental organizations already indicated interest in our program. We think there is a lot of value of having EAs as insiders in government to push for similar projects.
Design of the challenge
We designed a structured process for 38 civil servants that applied for the Challenge which we allocated over 6 teams. We started with two workshops covering subjects to raise awareness on policy impact incl. the five questions of EA as well as different decision making techniques and improving personal effectiveness exercises. This was followed by tailored working sessions with the 6 teams facilitated by our team. We have now finished facilitating the 10-month long program. We assisted civil servants to come up with their own solutions for obstacles they encounter in their work to increase the impact of their organization. In this post, we will elaborate on additional results since the last post and future plans.
Outcome of the Challenge
The Challenge had three goals:
1) Enhance awareness on policy impact;
2) Improve decision making (balancing between political reality and data/knowledge-driven decisions);
3) Foster open cultures that enable learning and adapting ability of the organization.
We have produced shared ownership of the “Impact challenge” with the Strategic Advisory Unit (ESA) and other relevant departments in the MFA that work on this topic.
At the closing session, which was recorded and viewed by others who could not attend, about 40 participants attended including representatives from important internal networks like the Staff Council as well as the Deputy Secretary General (DSG) representing the Senior Management Board. The DSG commented on the three final improvement plans developed during the Challenge:
1) Regular Red Teaming exercises to challenge policy advice and explore policy options
2) A Knowledge and Expertise platform that would allow for better utilization of inhouse expertise;
3) Specialized career tracks to enhance the organization’s strategic capacity on global priority areas.
This session took place as the final event of the Annual Strategy Days of the MFA. There was active engagement on the topic of enhancing impact by the team members as well as the jury members. One team analyzed their idea’s impact with EA questions around scope, neglectedness and tractability. The winning project on using Red Teaming exercises to safeguard internal critical debate has the potential to improve the decision-making process by 1) raising awareness on diverse policy impact and 2) fostering a culture in which there is more space to challenge thoughts on policy directions in an open positive learning environment. More and consistent Red Teaming exercises lead to more transparent and better decision making processes. Investing in this type of more knowledge-driven decision making can balance fast decision making driven by daily political developments and work pressure. The Strategic Advisory Unit is following up and setting up Red Teaming exercises for the development of two internal strategies on policy. If these pilots are successful, broader implementations and further institutionalization are possible.
The other two final plans are focused on improving knowledge-driven decisions and can be followed-up through existing initiatives like the recent establishment of the Innovation and Digitalization department that expressed interest in working out a knowledge platform. This platform would support employees in better finding and utilizing each other’s talents and experience. The internal Academy has been working on improving the offerings of specialist career tracks, therefore the third final plan from this Challenge can feed into their work.
All the ideas that made it to the final are focused on enhancing critical thinking and learning within the organization. The active participation and open positive feedback by people in positions at different levels within the MFA endorsed such a culture is developing and there is a willingness to continue to work on this.
Impact and Survey results
We generally think our impact with this project is:
Information and inspiration value for EAs that have similar aspirations
Improvements in decision making through follow-up of the ideas coming from the Impact Challenge (as well as the two other goals on raising awareness of policy impact and fostering more open critical working cultures)
Making civil servants within the Ministry more value aligned
Empowering individual civil servants that function as change agents for more long term effective policy making within the Ministry
To generate data about our Impact, we conducted three surveys among the participants of the course of the program/Challenge: (1) before the initial workshops, (2) after the initial workshops and the first follow-up sessions, and (3) after the final presentations. The response rates were 76% for (1), 50% for (2), and 53% for (3), with some questions in the surveys not receiving answers from every respondent.
Here are some key findings:
All age groups and seniority levels were represented, with more junior civil servants at the start of the program
Average rating of the program (in final survey): 3.8 out of 5.0
More than 50% of the teams and participants dropped out of the Challenge, especially the younger civil servants.
We observed one key improvement amongst participants: they reported achieving a better understanding of the organization’s structure and processes, which is what many participants initially felt they were lacking in order to have more positive impact in the Ministry.
The average learning experience of the participants was moderate to good.
Only some of the people who dropped off rated average or poor.
More than 50% would definitely participate in another challenge, another 25% are interested
→ Workshops alone do not yield the full learning experience as participants do not have the space to reflect on the content and practice it thoroughly.
In detail, we asked the participants to report how effective the Challenge was for them in a number of aspects. We present the average scores (on a scale from 1 to 5) ordered from highest to lowest ranking:
New network that provides useful insights in improving your work
3.6
Creating a movement within BZ to enhance overall impact of the organization
3.5
New perspective on enhancing policy impact
3.4
New tools to improve your policy impact
3.2
Creating a movement within BZ to improve the working environment
3.1
Enabling better decisions during the policy process
3.0
New perspective on enhancing personal effectiveness
2.9
New tools to improve personal effectiveness
2.9
New skills aimed at starting my own projects, including experimenting within those projects
2.8
Hence, the major benefits of the program seem to be about creating a network of like-minded people within the ministry who want to improve institutional decision making.
Apart from the survey we believe we have raised awareness about EA ideas in general at the Ministry. One illustration is that the Strategic Advisory Unit had invited Toby Ord as the keynote speaker of the Annual Strategy Days 2022. Lisa, has been introducing EA ideas since the start of her career at the Ministry in 2017 a.o. by inviting Peter Singer over for a lecture and by organizing an 80.000 hours workshop (thanks to support by Sam Hilton/HIPE). Through these initiatives she has also distributed almost 100 EA books amongst colleagues. In this Challenge we have also provided all participants with a copy of Doing Good Better.
Key Learnings
All the key learnings from our intermediate report still stand but we are happy to share some additional learnings
50% of the teams dropped off due to other obligations, especially after the summer break. Some measures could have reduced this:
Our collaborating partners at the Ministry explained that this happens regularly in training programs because of a culture where learning courses are not prioritized in the organization and therefore the drop-out rates and last minute cancellations are high;
We believe that if we would have shortened the duration of the program we could have kept participants more engaged;
Another solution we had in mind, but simply not the time to organize before the Challenge, is to get clear time commitments from participants and their managers from the start.
Earlier talks and agreements about how the ideas coming out of the Challenge would be followed up on and adopted. We would demand more buy-in for follow up from senior management earlier in the process next time;
Teams suffered from making clear and equal division of tasks to follow-up on their plans. We also noticed that there was a threshold for individuals to execute small scale testing of their ideas in their own work environments. We recommend putting more emphasis on assisting in project planning as well as supporting participants’ research and test skills in future initiatives.
Strong facilitation from EA-aligned individuals can support steering teams towards more scientifically sound ideas. Simultaneously, we’ve learned that understanding internal dynamics is required to come up with realistic, sensible solutions.
Lisa Gotoh, the co-organiser that works for the MFA, has built a strong reputation within the MFA, including positive exposure to senior civil servants in this way. Starting an intrapreneurial program like this can contribute to positive career capital.
Lisa’s experience working in the organization provided the team with a good understanding of the complexity of the policy making process and other interesting insights (information value) in order to continue improving the Challenge along the way. It also helped improve other activities from team members (e.g. Jan-Willem’s creation of Training For Good’s Impactful Policy Careers program.
We think that the Effective Altruism community could spend more resources on an even better understanding of measures that could improve the quality of decision making.
Future Work
As we are currently dealing with limited capacity we cannot ensure all suggested follow-ups:
We are happy to share all the materials (in English) of this Challenge and available for any questions or calls to share experiences for other EAs who may be interested in organizing similar projects.
Help kickstart/incorporate content in Dutch civil servant traineeship programs
Reiterate to a 3 month projects and try to institutionalize this into standardized programs in different Ministries
Acknowledgements
We would like to acknowledge that this project is funded by the Infrastructure Fund of the Center for Effective Altruism and the Dutch MFA as well as the time of volunteers in crafting content, translating, and organizing logistics. The core group is composed of the following five individuals; feel free to reach out to us if you have questions or comments!
Lisa M. Gotoh, Senior Policy Officer, Ministry of Foreign Affairs (currently posted at the Permanent Mission of the Kingdom of the Netherlands to the United Nations in New York)
Jan-Willem van Putten, former Managing Director, Effectief Altruïsme Nederland / current Director, Training For Good
Emil N. Iftekhar, PhD Candidate, Max Planck Institute for Dynamics and Self-Organization
Reijer Knol, QA Engineer, Dutch Ministry of Defense
Guiding civil servants to Improve Institutional Decision-Making through an ‘Impact Challenge’
This is a follow-up post to our intermediate report on a series of workshops and an ‘Impact Challenge’ at the Dutch Ministry of Foreign Affairs (MFA) to improve institutional decision making.
TL;DR conclusion:
From an impact-perspective we are still uncertain about the magnitude of impact of this project. This will depend on the follow up of the winning idea which is institutionalizing Red teaming exercises to improve decision making. At the time of writing, pilot Red Teaming exercises are being incorporated into the development of two internal strategies of policy. We will keep you informed about the impact.
Outcomes that we have observed:
The Challenge has empowered civil servants that have an interest in working on improving the decision-making process and enhancing policy impact. We provided them with tools and subjects to continue the discussion in their own work environments e.g., a Head of Mission/Ambassador of a Dutch mission abroad introduced the five EA questions to his team.
According to our judgment all three ideas that made the final would contribute to improved decision making quality within the Ministry, if implemented (big uncertainty). The winning idea is followed up on.
The Challenge shows that, when given the time to strategize and think about topics of improving impact, experts themselves come up with good solutions, guided by a team that inspires them with new ways of thinking.
We perceived interest in EA-related ideas within senior civil service ranks and more horizontally, spread over different departments throughout the Ministry. Overall there is an appetite for enhancing policy impact and improving decision making. We believe that similar projects that focus on improving institutional decision-making have the potential to be well received and successful.
In our evaluation discussion we have been encouraged to look into incorporating an adapted version of this project in the Dutch civil servant traineeship program as well as the foreign service traineeship program. This way, EA-related content could be part of the basic mandatory training program of new Dutch civil servants. We are looking into following up on this.
We believe the Effective Altruism community would benefit from spending more resources on research on how to improve coordination and collective action problems in policy and governance (as Larks mentioned in this post). This would increase the effectiveness of facilitation and nudging participants to more effective ideas in similar projects. Simultaneously, as called upon by many EAs recently, we need more action-oriented EAs taking similar initiatives to this experiment, to learn and share lessons on Improving Institutional Decision-Making.
Introduction
Recently we have finished a pilot on organizing an Impact Challenge at the Dutch Ministry of Foreign Affairs with the aim to improve its institutional decision making and raise awareness on policy impact. This experiment gave us and the broader EA community insight into the potential for working on improving institutional decision making. We observed a clear demand in this government institution to improve their work processes and change internal cultures. This is also driven by a shift in norms and values where more space is opening up for diversity and inclusion topics, including a more open approach to diversity of thought. However, important challenges remain related to the intrinsic bureaucratic system as well as deeply rooted cultures where it can be hard for change agents to institutionalize positive change. We want to encourage more EAs within the civil service to start their own intrapreneurial projects and hope to provide some inspiration through this post by sharing our lessons learned. Dutch EAs at other governmental organizations already indicated interest in our program. We think there is a lot of value of having EAs as insiders in government to push for similar projects.
Design of the challenge
We designed a structured process for 38 civil servants that applied for the Challenge which we allocated over 6 teams. We started with two workshops covering subjects to raise awareness on policy impact incl. the five questions of EA as well as different decision making techniques and improving personal effectiveness exercises. This was followed by tailored working sessions with the 6 teams facilitated by our team. We have now finished facilitating the 10-month long program. We assisted civil servants to come up with their own solutions for obstacles they encounter in their work to increase the impact of their organization. In this post, we will elaborate on additional results since the last post and future plans.
Outcome of the Challenge
The Challenge had three goals:
1) Enhance awareness on policy impact;
2) Improve decision making (balancing between political reality and data/knowledge-driven decisions);
3) Foster open cultures that enable learning and adapting ability of the organization.
We have produced shared ownership of the “Impact challenge” with the Strategic Advisory Unit (ESA) and other relevant departments in the MFA that work on this topic.
At the closing session, which was recorded and viewed by others who could not attend, about 40 participants attended including representatives from important internal networks like the Staff Council as well as the Deputy Secretary General (DSG) representing the Senior Management Board. The DSG commented on the three final improvement plans developed during the Challenge:
1) Regular Red Teaming exercises to challenge policy advice and explore policy options
2) A Knowledge and Expertise platform that would allow for better utilization of inhouse expertise;
3) Specialized career tracks to enhance the organization’s strategic capacity on global priority areas.
This session took place as the final event of the Annual Strategy Days of the MFA. There was active engagement on the topic of enhancing impact by the team members as well as the jury members. One team analyzed their idea’s impact with EA questions around scope, neglectedness and tractability. The winning project on using Red Teaming exercises to safeguard internal critical debate has the potential to improve the decision-making process by 1) raising awareness on diverse policy impact and 2) fostering a culture in which there is more space to challenge thoughts on policy directions in an open positive learning environment. More and consistent Red Teaming exercises lead to more transparent and better decision making processes. Investing in this type of more knowledge-driven decision making can balance fast decision making driven by daily political developments and work pressure. The Strategic Advisory Unit is following up and setting up Red Teaming exercises for the development of two internal strategies on policy. If these pilots are successful, broader implementations and further institutionalization are possible.
The other two final plans are focused on improving knowledge-driven decisions and can be followed-up through existing initiatives like the recent establishment of the Innovation and Digitalization department that expressed interest in working out a knowledge platform. This platform would support employees in better finding and utilizing each other’s talents and experience. The internal Academy has been working on improving the offerings of specialist career tracks, therefore the third final plan from this Challenge can feed into their work.
All the ideas that made it to the final are focused on enhancing critical thinking and learning within the organization. The active participation and open positive feedback by people in positions at different levels within the MFA endorsed such a culture is developing and there is a willingness to continue to work on this.
Impact and Survey results
We generally think our impact with this project is:
Information and inspiration value for EAs that have similar aspirations
Improvements in decision making through follow-up of the ideas coming from the Impact Challenge (as well as the two other goals on raising awareness of policy impact and fostering more open critical working cultures)
Making civil servants within the Ministry more value aligned
Empowering individual civil servants that function as change agents for more long term effective policy making within the Ministry
To generate data about our Impact, we conducted three surveys among the participants of the course of the program/Challenge: (1) before the initial workshops, (2) after the initial workshops and the first follow-up sessions, and (3) after the final presentations. The response rates were 76% for (1), 50% for (2), and 53% for (3), with some questions in the surveys not receiving answers from every respondent.
Here are some key findings:
All age groups and seniority levels were represented, with more junior civil servants at the start of the program
Average rating of the program (in final survey): 3.8 out of 5.0
More than 50% of the teams and participants dropped out of the Challenge, especially the younger civil servants.
We observed one key improvement amongst participants: they reported achieving a better understanding of the organization’s structure and processes, which is what many participants initially felt they were lacking in order to have more positive impact in the Ministry.
The average learning experience of the participants was moderate to good.
Only some of the people who dropped off rated average or poor.
More than 50% would definitely participate in another challenge, another 25% are interested
→ Workshops alone do not yield the full learning experience as participants do not have the space to reflect on the content and practice it thoroughly.
In detail, we asked the participants to report how effective the Challenge was for them in a number of aspects. We present the average scores (on a scale from 1 to 5) ordered from highest to lowest ranking:
Hence, the major benefits of the program seem to be about creating a network of like-minded people within the ministry who want to improve institutional decision making.
Apart from the survey we believe we have raised awareness about EA ideas in general at the Ministry. One illustration is that the Strategic Advisory Unit had invited Toby Ord as the keynote speaker of the Annual Strategy Days 2022. Lisa, has been introducing EA ideas since the start of her career at the Ministry in 2017 a.o. by inviting Peter Singer over for a lecture and by organizing an 80.000 hours workshop (thanks to support by Sam Hilton/HIPE). Through these initiatives she has also distributed almost 100 EA books amongst colleagues. In this Challenge we have also provided all participants with a copy of Doing Good Better.
Key Learnings
All the key learnings from our intermediate report still stand but we are happy to share some additional learnings
50% of the teams dropped off due to other obligations, especially after the summer break. Some measures could have reduced this:
Our collaborating partners at the Ministry explained that this happens regularly in training programs because of a culture where learning courses are not prioritized in the organization and therefore the drop-out rates and last minute cancellations are high;
We believe that if we would have shortened the duration of the program we could have kept participants more engaged;
Another solution we had in mind, but simply not the time to organize before the Challenge, is to get clear time commitments from participants and their managers from the start.
Earlier talks and agreements about how the ideas coming out of the Challenge would be followed up on and adopted. We would demand more buy-in for follow up from senior management earlier in the process next time;
Teams suffered from making clear and equal division of tasks to follow-up on their plans. We also noticed that there was a threshold for individuals to execute small scale testing of their ideas in their own work environments. We recommend putting more emphasis on assisting in project planning as well as supporting participants’ research and test skills in future initiatives.
Strong facilitation from EA-aligned individuals can support steering teams towards more scientifically sound ideas. Simultaneously, we’ve learned that understanding internal dynamics is required to come up with realistic, sensible solutions.
Lisa Gotoh, the co-organiser that works for the MFA, has built a strong reputation within the MFA, including positive exposure to senior civil servants in this way. Starting an intrapreneurial program like this can contribute to positive career capital.
Lisa’s experience working in the organization provided the team with a good understanding of the complexity of the policy making process and other interesting insights (information value) in order to continue improving the Challenge along the way. It also helped improve other activities from team members (e.g. Jan-Willem’s creation of Training For Good’s Impactful Policy Careers program.
We think that the Effective Altruism community could spend more resources on an even better understanding of measures that could improve the quality of decision making.
Future Work
As we are currently dealing with limited capacity we cannot ensure all suggested follow-ups:
We are happy to share all the materials (in English) of this Challenge and available for any questions or calls to share experiences for other EAs who may be interested in organizing similar projects.
Help kickstart/incorporate content in Dutch civil servant traineeship programs
Reiterate to a 3 month projects and try to institutionalize this into standardized programs in different Ministries
Acknowledgements
We would like to acknowledge that this project is funded by the Infrastructure Fund of the Center for Effective Altruism and the Dutch MFA as well as the time of volunteers in crafting content, translating, and organizing logistics. The core group is composed of the following five individuals; feel free to reach out to us if you have questions or comments!
Lisa M. Gotoh, Senior Policy Officer, Ministry of Foreign Affairs (currently posted at the Permanent Mission of the Kingdom of the Netherlands to the United Nations in New York)
Jan-Willem van Putten, former Managing Director, Effectief Altruïsme Nederland / current Director, Training For Good
Emil N. Iftekhar, PhD Candidate, Max Planck Institute for Dynamics and Self-Organization
Reijer Knol, QA Engineer, Dutch Ministry of Defense