Thanks for this post. Apologies I have not had to read through in detail, but I would suggest that perhaps:
The search criteria that you have used has missed a significant number of papers within the field. Looking at the country distribution you posted, this becomes more obvious; I would suggest looking at the What Works papers that were produced several years ago, where quite extensive literature reviews were being conducted
I think you do acknowledge this weakly, but there is such wide-spanning heterogeneity in the studies that you have included (and the programs the use), that I think tighter sub-group analysis is needed to tease out meaningful conclusions
A lot of work in this space has been done in the last 5-6 years; whilst not a specific limitiation of your work, just something to bear in mind!
I do not think we have missed a significant portion of primary prevention papers in our time period. Looking at that page, I am seeing some things that had midpoint evaluations in 2018. Looking at this group’s final report (https://www.whatworks.co.za/documents/publications/390-what-works-to-prevent-vawg-final-performance-evaluation-report-mar-2020/file), I do not see anything that qualifies as a primary prevention of sexual violence. We did a painstaking systematic search and I’m reasonably confident we got just about everything that meets our criteria. As to whether we might have chosen different criteria—too late now, but for my own curiosity, what would you suggest?
We have many subgroup analyses in the paper, though for my tastes, I wish we could have done more in terms of grouping studies together by theoretical approach and then analyzing them. This turned out to be really hard in practice because there was so much overlapping content but also so many bespoke delivery mechanisms. This heterogeneity is one reason my next meta-analysis (https://forum.effectivealtruism.org/posts/k9qqGZtmWz3x4yaaA/environmental-and-health-appeals-are-the-most-effective) sets strict quality-related inclusion criteria and then compares theoretical approaches head-to-head.
Definitely, and this is arguably the main limitation of this paper: we’re a few years out of date. Basically what happened was, we started this paper pre-pandemic (the first conversation I was part of was summer 2017!), did our search, and then lost a few years to the pandemic and also a lot of the team worked on a different meta-analysis in the interim (https://www.annualreviews.org/content/journals/10.1146/annurev-psych-071620-030619, on which I was an RA). I still think we’ve got some interesting and general lessons from reviewing 30+ years worth of papers.
Thanks for this post. Apologies I have not had to read through in detail, but I would suggest that perhaps:
The search criteria that you have used has missed a significant number of papers within the field. Looking at the country distribution you posted, this becomes more obvious; I would suggest looking at the What Works papers that were produced several years ago, where quite extensive literature reviews were being conducted
I think you do acknowledge this weakly, but there is such wide-spanning heterogeneity in the studies that you have included (and the programs the use), that I think tighter sub-group analysis is needed to tease out meaningful conclusions
A lot of work in this space has been done in the last 5-6 years; whilst not a specific limitiation of your work, just something to bear in mind!
Hi Akhil,
Thanks for engaging.
I do not think we have missed a significant portion of primary prevention papers in our time period. Looking at that page, I am seeing some things that had midpoint evaluations in 2018. Looking at this group’s final report (https://www.whatworks.co.za/documents/publications/390-what-works-to-prevent-vawg-final-performance-evaluation-report-mar-2020/file), I do not see anything that qualifies as a primary prevention of sexual violence. We did a painstaking systematic search and I’m reasonably confident we got just about everything that meets our criteria. As to whether we might have chosen different criteria—too late now, but for my own curiosity, what would you suggest?
We have many subgroup analyses in the paper, though for my tastes, I wish we could have done more in terms of grouping studies together by theoretical approach and then analyzing them. This turned out to be really hard in practice because there was so much overlapping content but also so many bespoke delivery mechanisms. This heterogeneity is one reason my next meta-analysis (https://forum.effectivealtruism.org/posts/k9qqGZtmWz3x4yaaA/environmental-and-health-appeals-are-the-most-effective) sets strict quality-related inclusion criteria and then compares theoretical approaches head-to-head.
Definitely, and this is arguably the main limitation of this paper: we’re a few years out of date. Basically what happened was, we started this paper pre-pandemic (the first conversation I was part of was summer 2017!), did our search, and then lost a few years to the pandemic and also a lot of the team worked on a different meta-analysis in the interim (https://www.annualreviews.org/content/journals/10.1146/annurev-psych-071620-030619, on which I was an RA). I still think we’ve got some interesting and general lessons from reviewing 30+ years worth of papers.
DM’d you