I’m not sure what our initial run on the pamphlets will be, but I’m thinking in the 5K-15K range. I haven’t done any analysis to figure out how many we’d need to hand out to get good statistics; not even really sure how to go about doing that, to be honest.
Please talk to a real statistician if you’re designing an experiment! Random Internet people picking your design apart is actually pretty good as far as review goes (if they’re the right Internet people), but actual statisticians are orders of magnitude better. Experiment design is very tricky and good statisticians are aware of both lots of tools to make your job easier, and lots of pitfalls for you to avoid. To quote Ronald Fisher:
To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.
I figure most people don’t know a statistician (I don’t) but there are plenty of people in LessWrong discussion who know how to do a power calculation so it might be good to start there (or just to dig a bit deeper here).
It really won’t help address the problem I’m talking about at all, which is unknown design flaws/statistical techniques/study design tools. Once you’ve figured out that you have a problem like “how should I power my study?”, smart people plus Internet is fine; I’m worried about the other 10 issues we haven’t noticed yet. That’s the kind of thing that statisticians are useful for.
Fortunately, it turns out you can still talk to statisticians even if you don’t know them personally. If you’re spending money on your study, you could even go so far as to hire a consultant. I also know statisticians and would be happy to refer Jonathon.
As someone who did a lot of study design in undergraduate, is currently a “data scientist”, and considers myself smart, I can confirm that I still make approximately 10 huge mistakes every time I run a study.
Yeah, if you give me the contact info of a statistician that you recommend that would be great. I don’t know if we have the budget for it, but I would definitely reach out.
I’m checking for people who would be interested in doing it pro bono. If that doesn’t work, I’m 99% sure you can find some people to fund a couple consultant-hours.
I don’t know if we have the budget for it
Not to put too fine a point on it, but if the alternative is TLYCS designing the experiment themselves, this is pretty much like running a charity that spends nothing on overhead. It looks good on paper, but in reality, that last bit of money is a huge effectiveness multiplier.
Please talk to a real statistician if you’re designing an experiment! Random Internet people picking your design apart is actually pretty good as far as review goes (if they’re the right Internet people), but actual statisticians are orders of magnitude better. Experiment design is very tricky and good statisticians are aware of both lots of tools to make your job easier, and lots of pitfalls for you to avoid. To quote Ronald Fisher:
Statistics Without Borders may be a good place to start.
I figure most people don’t know a statistician (I don’t) but there are plenty of people in LessWrong discussion who know how to do a power calculation so it might be good to start there (or just to dig a bit deeper here).
It really won’t help address the problem I’m talking about at all, which is unknown design flaws/statistical techniques/study design tools. Once you’ve figured out that you have a problem like “how should I power my study?”, smart people plus Internet is fine; I’m worried about the other 10 issues we haven’t noticed yet. That’s the kind of thing that statisticians are useful for.
Fortunately, it turns out you can still talk to statisticians even if you don’t know them personally. If you’re spending money on your study, you could even go so far as to hire a consultant. I also know statisticians and would be happy to refer Jonathon.
Makes sense. Also, if the EA survey is redone, that might be an even more important place to have a statistician.
As someone who did a lot of study design in undergraduate, is currently a “data scientist”, and considers myself smart, I can confirm that I still make approximately 10 huge mistakes every time I run a study.
Yeah, if you give me the contact info of a statistician that you recommend that would be great. I don’t know if we have the budget for it, but I would definitely reach out.
I’m checking for people who would be interested in doing it pro bono. If that doesn’t work, I’m 99% sure you can find some people to fund a couple consultant-hours.
Not to put too fine a point on it, but if the alternative is TLYCS designing the experiment themselves, this is pretty much like running a charity that spends nothing on overhead. It looks good on paper, but in reality, that last bit of money is a huge effectiveness multiplier.
I’d consider funding this if it’s “worth it” and not too much money. I’m sure others would as well.
I’m fairly surprised the EA movement doesn’t have official statisticians. The EAA movement has a lot of people claiming to be official statisticians.