Awesome, thanks for diving to this level of detail. You mention a lot of good points, some of which we’ve thought of, some not. I’ve started emailing statistical consulting companies, we’ll see what comes back.
I do want to pose this question in another way that I think reflects more accurately my doubts about the necessity for a statistician. I mean, I definitely agree having someone on board with that skill set would be nice … so would having a world class add agency designing the pamphlet, and a small army of volunteers to hand them out, etc. But is it necessary? So, let me frame up the question this way. Let’s say we run this study, and afterwards publish a report. We say, this is how we calculated our baseline values (and give the data), these are the resulting spikes in our tracked metrics (and give the data), these are the assumptions we used in calculating our success criteria, and these are the conclusions that we’ve made. How can this possibly be bad or counterproductive? Would you look at the data and be like, “well they didn’t use the best possible calculation for baseline, so I’m throwing it all out”? You follow what I’m asking here? I just fail to see how collecting the data and doing the calculations we proposed—even if they’re not perfect—could possibly be bad or counterproductive. Maybe we’re leaving some value on the table by not consulting a statistician, but I don’t understand the mode in which our entire effort fails by not consulting one.
A statistically sound study design is important for two major reasons I can see. Firstly it will maximise your chance of answering the question you are trying to answer (ie be adequately powered, have robust confidence intervals etc). But in addition it will help make sure you are studying what you think you are studying. Giving adequate consideration to sampling, randomisation, controls etc are all key, as is using the correct tests to measure your results, and these are all things a good stats person will help with. Having a ‘precise’ result is no good if you didn’t study what you thought you were studying, and a small p value is meaningless if you didn’t make the right comparison.
Regarding why I think bad data is worse than no data, I think it comes to a question of human psychology. We love numbers and measurement. It’s very hard for us to unhear a result even when we find out later it was exaggerated or incorrect. (For example the MMR vaccine and Wakefield’s discredited paper). Nick Bostrum refers to ‘data fumes’ - unreliable bits of information that permeate out ideas and to which we give excessive attention.
Awesome, thanks for diving to this level of detail. You mention a lot of good points, some of which we’ve thought of, some not. I’ve started emailing statistical consulting companies, we’ll see what comes back.
I do want to pose this question in another way that I think reflects more accurately my doubts about the necessity for a statistician. I mean, I definitely agree having someone on board with that skill set would be nice … so would having a world class add agency designing the pamphlet, and a small army of volunteers to hand them out, etc. But is it necessary? So, let me frame up the question this way. Let’s say we run this study, and afterwards publish a report. We say, this is how we calculated our baseline values (and give the data), these are the resulting spikes in our tracked metrics (and give the data), these are the assumptions we used in calculating our success criteria, and these are the conclusions that we’ve made. How can this possibly be bad or counterproductive? Would you look at the data and be like, “well they didn’t use the best possible calculation for baseline, so I’m throwing it all out”? You follow what I’m asking here? I just fail to see how collecting the data and doing the calculations we proposed—even if they’re not perfect—could possibly be bad or counterproductive. Maybe we’re leaving some value on the table by not consulting a statistician, but I don’t understand the mode in which our entire effort fails by not consulting one.
(Sorry for taking so long to reply)
A statistically sound study design is important for two major reasons I can see. Firstly it will maximise your chance of answering the question you are trying to answer (ie be adequately powered, have robust confidence intervals etc). But in addition it will help make sure you are studying what you think you are studying. Giving adequate consideration to sampling, randomisation, controls etc are all key, as is using the correct tests to measure your results, and these are all things a good stats person will help with. Having a ‘precise’ result is no good if you didn’t study what you thought you were studying, and a small p value is meaningless if you didn’t make the right comparison.
Regarding why I think bad data is worse than no data, I think it comes to a question of human psychology. We love numbers and measurement. It’s very hard for us to unhear a result even when we find out later it was exaggerated or incorrect. (For example the MMR vaccine and Wakefield’s discredited paper). Nick Bostrum refers to ‘data fumes’ - unreliable bits of information that permeate out ideas and to which we give excessive attention.