A quick drive-by comment on “4. Missed RCT Opportunity”: The sample size seems way too small for a RCT to be worth it. There’s not much statistical power to work with when researchers are studying a messy intervention with only 6 countries. And I imagine they’d struggle to attribute changes to the Technical Support Units unless it was something truly transformative (at least within the framework of the RCT).[1]
More broadly, I’m not aware of any commonly accepted way to do “small n” impact evaluation yet, especially with something as customized as Technical Assistance. This blog post from 3ie, a NGO to promote evidence-based policy making, talked about the issue 13 years ago and I think it’s still broadly true. The impact evaluation toolkit works best with (1) a precisely defined intervention, (2) a decent sample size (say n > 100), and (3) a very homogeneous sample. This grant, so far, looks to be the opposite of all 3.
I also recall the math for statistical inference gets strange when using very small sample sizes (say n<15) and may require assumptions that most people consider unrealistic. But I could be wrong here.
Thanks I’m going to edit that I think you are right.
To make an RCT work sample success wise you would need district level randomisation probably to work, and that wouldn’t make sense here when it’s only a central government level intervention.
A quick drive-by comment on “4. Missed RCT Opportunity”: The sample size seems way too small for a RCT to be worth it. There’s not much statistical power to work with when researchers are studying a messy intervention with only 6 countries. And I imagine they’d struggle to attribute changes to the Technical Support Units unless it was something truly transformative (at least within the framework of the RCT).[1]
More broadly, I’m not aware of any commonly accepted way to do “small n” impact evaluation yet, especially with something as customized as Technical Assistance. This blog post from 3ie, a NGO to promote evidence-based policy making, talked about the issue 13 years ago and I think it’s still broadly true. The impact evaluation toolkit works best with (1) a precisely defined intervention, (2) a decent sample size (say n > 100), and (3) a very homogeneous sample. This grant, so far, looks to be the opposite of all 3.
I also recall the math for statistical inference gets strange when using very small sample sizes (say n<15) and may require assumptions that most people consider unrealistic. But I could be wrong here.
Drive by hit job successful 🤣
Thanks I’m going to edit that I think you are right.
To make an RCT work sample success wise you would need district level randomisation probably to work, and that wouldn’t make sense here when it’s only a central government level intervention.