The response rate issue seems key to me: if we had known the study would be substantially underpowered we would probably have not run it, or at least figured out how to run it differently.
It would have been awesome if we could have funnelled all 200K people who saw one of the two pages into taking a survey. However, the retargeting required the participants to click on yet another ad advertising the survey (at ~$1 in cost-per-click and the incentive offered per person), and fewer than 2% of our original population did so (the âresponse rateâ).
This low response rate was lower than we expected (despite doing actual piloting of the study to determine a guess at the response rate) and led to a large degree of subjects inadvertently dropping out[6] and we werenât able to get a large enough sample size despite paying so much money.
Whatâs the right methodology for a response rate pilot?
Youâre trying to learn what the (cumulative) response rate is as a function of money/âtime. You need a small enough sample (audience) that you can afford to really probe the dimensions of this space and pull out all the responses youâre going to be able to get. So if your full study will have 200k participants, you should have your pilot sample just be ~1k. Then ramp up your spending, and see how many responses you get over time. This tells you the total number of responses you can pull out of a 1k sample, and how much money/âtime it will probably take to get a given response rate.
(The pilot study in this case didnât actually measure response rate, just response cost, and used CPC ads with a very large sample size in a way that only measured the cost of the first few clicks. Since the first few clicks from a sample are always the cheapest, this wasnât a useful approach.)
The response rate issue seems key to me: if we had known the study would be substantially underpowered we would probably have not run it, or at least figured out how to run it differently.
Whatâs the right methodology for a response rate pilot?
Youâre trying to learn what the (cumulative) response rate is as a function of money/âtime. You need a small enough sample (audience) that you can afford to really probe the dimensions of this space and pull out all the responses youâre going to be able to get. So if your full study will have 200k participants, you should have your pilot sample just be ~1k. Then ramp up your spending, and see how many responses you get over time. This tells you the total number of responses you can pull out of a 1k sample, and how much money/âtime it will probably take to get a given response rate.
(The pilot study in this case didnât actually measure response rate, just response cost, and used CPC ads with a very large sample size in a way that only measured the cost of the first few clicks. Since the first few clicks from a sample are always the cheapest, this wasnât a useful approach.)