IMHO, the āmath hardā parts of meta-analysis are figuring out what questions you want to ask, what are sensible inclusion criteria, and what statistical models are appropriate. Asking how much time this takes is the same as asking, where do ideas come from?
The ābodybuilding hardā part of meta-analysis is finding literature. The evaluators didnāt care for our search strategy, which you could charitably call ābespokeā and uncharitably call āad hoc and fundamentally unreplicable.ā But either way, I read about 1000 papers closely enough to see if they qualified for inclusion, and then, partly to make sure I didnāt duplicate my own efforts, I recorded notes on every study that looked appropriate but wasnāt. I also read, or at least read the bibliographies of, about 160 previous reviews. Maybe youāre a faster reader than I am, but ballpark, this was 500+ hours of work.
Regarding the computational aspects, the git history tells the story, but specifically making everything computationally reproducible, e.g. writing the functions, checking my own work, setting things up to be generalizableāa week of work in total? Iām not sure.
As I reread reviewer 2ā²s comments today, it occurred to me that some of their ideas might be interesting test cases for what Claude Code is and is not capable of doing. Iām thinking particularly of trying to formally incorporate my subjective notes about uncertainty (e.g. the many places where I admit that the effect size estimates involved a lot of guesswork) into some kind of...supplementary regression term about how much weight an estimate should get in meta-analysis? Like maybe Iād use Wasserstein-2 distance, as my advisor Don recently proposed? Or Bayesian meta-analysis? This is an important problem, and I donāt consider it solved by RoB2 or whatever, which means that fixing it might be, IDK, a whole new paper which takes however long that does? As my co-authors Don and Betsy & co. comment in a separate paper on which I was an RA: > Too often, research syntheses focus solely on estimating effect sizes, regardless of whether the treatments are realistic, the outcomes are assessed unobtrusively, and the key features of the experiment are presented in a transparent manner. Here we focus on what we term landmark studies, which are studies that are exceptionally well-designed and executed (regardless of what they discover). These studies provide a glimpse of what a meta-analysis would reveal if we could weight studies by quality as well as quantity. [the point being, meta-analysis is not well-suited for weighing by quality.]
Itās possible that some of the proposed changes would take less time than that. Maybe risk of bias assessment could be knocked out in a week?. But itās been about a year since the relevant studies were in my working memory, which means Iād probably have to re-read them all, and across our main and supplementary dataset, thatās dozens of papers. How long does it take you to read dozens of papers? Iād say I can read about 3-4 papers a day closely if Iām really, really cranking. So in all likelihood, yes, weeks of work, and thatās weeks where I wouldnāt be working on a project about building empathy for chickens. Which admittedly Iām procrastinating on by writing this 500+ word comment š
Love talking nitty gritty of meta-analysis š
IMHO, the āmath hardā parts of meta-analysis are figuring out what questions you want to ask, what are sensible inclusion criteria, and what statistical models are appropriate. Asking how much time this takes is the same as asking, where do ideas come from?
The ābodybuilding hardā part of meta-analysis is finding literature. The evaluators didnāt care for our search strategy, which you could charitably call ābespokeā and uncharitably call āad hoc and fundamentally unreplicable.ā But either way, I read about 1000 papers closely enough to see if they qualified for inclusion, and then, partly to make sure I didnāt duplicate my own efforts, I recorded notes on every study that looked appropriate but wasnāt. I also read, or at least read the bibliographies of, about 160 previous reviews. Maybe youāre a faster reader than I am, but ballpark, this was 500+ hours of work.
Regarding the computational aspects, the git history tells the story, but specifically making everything computationally reproducible, e.g. writing the functions, checking my own work, setting things up to be generalizableāa week of work in total? Iām not sure.
The paper went through many internal revisions and changed shape a lot from its initial draft when we pivoted in how we treated red and processed meat. Thatās hundreds of hours. Peer review was probably another 40 hour workweek.
As I reread reviewer 2ā²s comments today, it occurred to me that some of their ideas might be interesting test cases for what Claude Code is and is not capable of doing. Iām thinking particularly of trying to formally incorporate my subjective notes about uncertainty (e.g. the many places where I admit that the effect size estimates involved a lot of guesswork) into some kind of...supplementary regression term about how much weight an estimate should get in meta-analysis? Like maybe Iād use Wasserstein-2 distance, as my advisor Don recently proposed? Or Bayesian meta-analysis? This is an important problem, and I donāt consider it solved by RoB2 or whatever, which means that fixing it might be, IDK, a whole new paper which takes however long that does? As my co-authors Don and Betsy & co. comment in a separate paper on which I was an RA:
> Too often, research syntheses focus solely on estimating effect sizes, regardless of whether the treatments are realistic, the outcomes are assessed unobtrusively, and the key features of the experiment are presented in a transparent manner. Here we focus on what we term landmark studies, which are studies that are exceptionally well-designed and executed (regardless of what they discover). These studies provide a glimpse of what a meta-analysis would reveal if we could weight studies by quality as well as quantity. [the point being, meta-analysis is not well-suited for weighing by quality.]
Itās possible that some of the proposed changes would take less time than that. Maybe risk of bias assessment could be knocked out in a week?. But itās been about a year since the relevant studies were in my working memory, which means Iād probably have to re-read them all, and across our main and supplementary dataset, thatās dozens of papers. How long does it take you to read dozens of papers? Iād say I can read about 3-4 papers a day closely if Iām really, really cranking. So in all likelihood, yes, weeks of work, and thatās weeks where I wouldnāt be working on a project about building empathy for chickens. Which admittedly Iām procrastinating on by writing this 500+ word comment š