The response rate issue seems key to me: if we had known the study would be substantially underpowered we would probably have not run it, or at least figured out how to run it differently.
It would have been awesome if we could have funnelled all 200K people who saw one of the two pages into taking a survey. However, the retargeting required the participants to click on yet another ad advertising the survey (at ~$1 in cost-per-click and the incentive offered per person), and fewer than 2% of our original population did so (the “response rate”).
This low response rate was lower than we expected (despite doing actual piloting of the study to determine a guess at the response rate) and led to a large degree of subjects inadvertently dropping out[6] and we weren’t able to get a large enough sample size despite paying so much money.
What’s the right methodology for a response rate pilot?
You’re trying to learn what the (cumulative) response rate is as a function of money/time. You need a small enough sample (audience) that you can afford to really probe the dimensions of this space and pull out all the responses you’re going to be able to get. So if your full study will have 200k participants, you should have your pilot sample just be ~1k. Then ramp up your spending, and see how many responses you get over time. This tells you the total number of responses you can pull out of a 1k sample, and how much money/time it will probably take to get a given response rate.
(The pilot study in this case didn’t actually measure response rate, just response cost, and used CPC ads with a very large sample size in a way that only measured the cost of the first few clicks. Since the first few clicks from a sample are always the cheapest, this wasn’t a useful approach.)
The response rate issue seems key to me: if we had known the study would be substantially underpowered we would probably have not run it, or at least figured out how to run it differently.
What’s the right methodology for a response rate pilot?
You’re trying to learn what the (cumulative) response rate is as a function of money/time. You need a small enough sample (audience) that you can afford to really probe the dimensions of this space and pull out all the responses you’re going to be able to get. So if your full study will have 200k participants, you should have your pilot sample just be ~1k. Then ramp up your spending, and see how many responses you get over time. This tells you the total number of responses you can pull out of a 1k sample, and how much money/time it will probably take to get a given response rate.
(The pilot study in this case didn’t actually measure response rate, just response cost, and used CPC ads with a very large sample size in a way that only measured the cost of the first few clicks. Since the first few clicks from a sample are always the cheapest, this wasn’t a useful approach.)