If you don’t already have it, I would strongly recommend getting a copy of Gerber & Green’s Field Experiments. I would also very strongly recommend that you (or EA Cameroon) engage an experimental methodology expert for this project, rather than pose the question on the forum (I am not such an expert).
It is very difficult to address all of these questions in a broad way, since the answers depend on:
The smallest effect size you would hope to observe
Your available resources
The population within each cluster
The total population
Your analysis methodology
I’m a little confused about the setup. You say that there are 6 groups— so how would it be possible to have “6 intervention + 3 non-intervention?” Sorry if I’m misunderstanding.
In general, and particularly in this context, it makes sense to split your clusters evenly between treatment and control. This is the setup that minimizes the standard error of the difference between groups. When the variance is larger, smaller effect sizes are difficult to detect. The smaller the number of clusters in your control group, for example, the larger the effect size that you would have to detect in order to make a statistically defensible claim.
With such a small number of clusters, effect sizes would have to be very large in order to be statistically distinguishable from zero. If indeed 50% of the population in these groups is already masked, 6 clusters may not be enough to see an effect.
Can we get some clarification on some of your questions? Particularly:
How important, in terms of statistical power is to include all clusters
If you have only 6 to choose from, then the answer is very important. But I’m not sure this is the sense in which you mean this.
How many persons should be observed at each place?
My inclination here is to say “as many as possible.” But this is constrained by your resources and your method of observation. Can you say more about the data collection plan?
Thank you. I was not able to get (a pdf of) Field Experiments, but downloaded the “Field Experimental Designs for the Study of Media Effects,” also co-authored by Green. They point out “robust cluster standard errors” to estimate “individual-level average treatment effect” (172).
To answer your points:
The smallest effect size you would hope to observe
20%. From 5⁄10 to 6⁄10 or equivalent % increase
Your available resources
Researchers in all of the campaign clusters and some of the non-campaign ones. They can count whether e. g. few hundreds of individuals wear face covering
The population within each cluster
Different, average of 180,000⁄6 = 30,000.
The total population
Since we are just looking to estimate the impact of the 180,000-person campaign and not to generalize it, this should be 180,000x2 (180,000 participating and an equal number of non-participants who are the nearest geographically and in characteristics).
Your analysis methodology
Probit, logit or simple linear regression, but open to suggestions
I meant 6 groups in the intervention area, and some number of groups (e. g. 3 or 6) in the non-intervention area.
OK. So 3 intervention clusters and 3 non-intervention clusters are better than 6 intervention clusters and 3 non-intervention clusters but 6+6 may be necessary? Would the answer depend on the intra-cluster correlation coefficient (ρ)? Perhaps, the texts that generally talk about clustering assume relatively significant between cluster variability and low within cluster variability (so high ρ). However, in this study, how people respond to the messaging may not depend much on their ‘cluster assignment,’ but much more on their individual characteristics that, on average, may be comparable across the clusters and the studied population.
I should ask EA Cameroon about the possibility of different average responses in different villages.
Do you know of any online sample size calculator that includes clusters?
I refer you to Sindy’s comment (she is actually an expert) but I want to note and verify that it sounds as if you may not actually be thinking of collecting individual-level data, and that you’re thinking of making observations at the village level (e.g. what % of people in this village wear masks?). So it’s not just the case that you wouldn’t have enough clusters to make a statistical claim, but you may actually be talking about doing an experiment in which the units are villages… so n = 6 to 12. Then of course you’d have considerable error in the village-level estimate, and uncertainty about the representativeness about the sample within each village. I agree with Sindy that you probably don’t want an RCT here.
If you don’t already have it, I would strongly recommend getting a copy of Gerber & Green’s Field Experiments. I would also very strongly recommend that you (or EA Cameroon) engage an experimental methodology expert for this project, rather than pose the question on the forum (I am not such an expert).
It is very difficult to address all of these questions in a broad way, since the answers depend on:
The smallest effect size you would hope to observe
Your available resources
The population within each cluster
The total population
Your analysis methodology
I’m a little confused about the setup. You say that there are 6 groups— so how would it be possible to have “6 intervention + 3 non-intervention?” Sorry if I’m misunderstanding.
In general, and particularly in this context, it makes sense to split your clusters evenly between treatment and control. This is the setup that minimizes the standard error of the difference between groups. When the variance is larger, smaller effect sizes are difficult to detect. The smaller the number of clusters in your control group, for example, the larger the effect size that you would have to detect in order to make a statistically defensible claim.
With such a small number of clusters, effect sizes would have to be very large in order to be statistically distinguishable from zero. If indeed 50% of the population in these groups is already masked, 6 clusters may not be enough to see an effect.
Can we get some clarification on some of your questions? Particularly:
If you have only 6 to choose from, then the answer is very important. But I’m not sure this is the sense in which you mean this.
My inclination here is to say “as many as possible.” But this is constrained by your resources and your method of observation. Can you say more about the data collection plan?
Thank you. I was not able to get (a pdf of) Field Experiments, but downloaded the “Field Experimental Designs for the Study of Media Effects,” also co-authored by Green. They point out “robust cluster standard errors” to estimate “individual-level average treatment effect” (172).
To answer your points:
The smallest effect size you would hope to observe
20%. From 5⁄10 to 6⁄10 or equivalent % increase
Your available resources
Researchers in all of the campaign clusters and some of the non-campaign ones. They can count whether e. g. few hundreds of individuals wear face covering
The population within each cluster
Different, average of 180,000⁄6 = 30,000.
The total population
Since we are just looking to estimate the impact of the 180,000-person campaign and not to generalize it, this should be 180,000x2 (180,000 participating and an equal number of non-participants who are the nearest geographically and in characteristics).
Your analysis methodology
Probit, logit or simple linear regression, but open to suggestions
I meant 6 groups in the intervention area, and some number of groups (e. g. 3 or 6) in the non-intervention area.
OK. So 3 intervention clusters and 3 non-intervention clusters are better than 6 intervention clusters and 3 non-intervention clusters but 6+6 may be necessary? Would the answer depend on the intra-cluster correlation coefficient (ρ)? Perhaps, the texts that generally talk about clustering assume relatively significant between cluster variability and low within cluster variability (so high ρ). However, in this study, how people respond to the messaging may not depend much on their ‘cluster assignment,’ but much more on their individual characteristics that, on average, may be comparable across the clusters and the studied population.
I should ask EA Cameroon about the possibility of different average responses in different villages.
Do you know of any online sample size calculator that includes clusters?
I refer you to Sindy’s comment (she is actually an expert) but I want to note and verify that it sounds as if you may not actually be thinking of collecting individual-level data, and that you’re thinking of making observations at the village level (e.g. what % of people in this village wear masks?). So it’s not just the case that you wouldn’t have enough clusters to make a statistical claim, but you may actually be talking about doing an experiment in which the units are villages… so n = 6 to 12. Then of course you’d have considerable error in the village-level estimate, and uncertainty about the representativeness about the sample within each village. I agree with Sindy that you probably don’t want an RCT here.
OK, thank you.