I have a lot of admiration for your transparency and public commitment. In general, I think this kind of public commitment and pre-analysis planning for key evaluation results is laudable and pushes the sector towards evidence-based decision making.
That said, I do have some concerns about setting a pre-commitment on an effect size from an RCT, particularly at such an early stage of an intervention’s maturity. How are you thinking about mitigating the risk of a null result? I understand that in that scenario you’d pause for a few months and conduct exploratory research. But would there not be an opportunity, at earlier stages of your theory of change, to better test the inputs-to-outputs and outputs-to-outcomes links through things like:
simple observational studies
mixed methods
process evaluations, run in parallel with or prior to the RCT?
I’ll put my biases on the table. I think RCTs are incredibly useful tools, but they’re often not the most helpful tools for understanding the complex theory of change behind an intervention. They give you a single output, usually an effect size, that can be hard to disentangle and act on operationally, especially if that effect size turns out to be null.
Thanks Tony! Completely agree that the RCT are not the most helpful tools to understand ToCs. Both our first and second seasons were created through an iterative process that involved a lot of formative (qualitative research) that helped us get more insights in how listeners actually perceive the drama and its characters. See for example our (small) mix method evaluation of season 2 prior to the RCT (http://www.novah.ngo/qual-eval-twubakane-s2-infographic).
Hi NOVAH team,
I have a lot of admiration for your transparency and public commitment. In general, I think this kind of public commitment and pre-analysis planning for key evaluation results is laudable and pushes the sector towards evidence-based decision making.
That said, I do have some concerns about setting a pre-commitment on an effect size from an RCT, particularly at such an early stage of an intervention’s maturity. How are you thinking about mitigating the risk of a null result? I understand that in that scenario you’d pause for a few months and conduct exploratory research. But would there not be an opportunity, at earlier stages of your theory of change, to better test the inputs-to-outputs and outputs-to-outcomes links through things like:
simple observational studies
mixed methods
process evaluations, run in parallel with or prior to the RCT?
I’ll put my biases on the table. I think RCTs are incredibly useful tools, but they’re often not the most helpful tools for understanding the complex theory of change behind an intervention. They give you a single output, usually an effect size, that can be hard to disentangle and act on operationally, especially if that effect size turns out to be null.
Thanks Tony! Completely agree that the RCT are not the most helpful tools to understand ToCs. Both our first and second seasons were created through an iterative process that involved a lot of formative (qualitative research) that helped us get more insights in how listeners actually perceive the drama and its characters. See for example our (small) mix method evaluation of season 2 prior to the RCT (http://www.novah.ngo/qual-eval-twubakane-s2-infographic).