On surveys- To summarise, the concern you raise is that the programs aim to reduce social acceptability of VAWG, and mostly use local interviewers to assess incidence of violence, which may introduce social acceptability bias. Although this seems plausible, interviews were trained, conducted privately and used validated questionnaires. To quote one paper
The study was conducted in accordance with WHO guidelines for the safe and ethical collection of data on violence against women [[24]]. These guidelines seek to minimize reporting biases and risk of harm to both respondents and interviewers. At both baseline and follow-up, interviewers received at least three weeks of training on the ethical and methodological issues surrounding the conduct of a survey relating to IPV and HIV, as well as ongoing support during the course of the survey. Interviewers were all from the local area, and interviewed respondents of the same sex as themselves. Interviews were conducted in private settings, in Luganda or English, and were concluded by providing information on additional support services in the area. At baseline, interviewers conducting the baseline survey were blinded as to the allocation of the intervention. It was not, however, possible to keep follow-up interviewers blinded.
As a result, I think this risk of bias is quite low. Also, I think that inherently, any impact evaluation of interventions in this space would require surveys.
3. External validity- A valid concern. I have two comments: (1)There are a number of studies in different settings which show positive results, suggesting external validity. (2) Although cultural and social drivers of violence vary, the intervention is co-designed with community and quite locally tailored, which mitigates some of the concern around external validity.
4. Meta-comment- I think that some of my estimates of the persistence of effects were quite conservative, which may counterbalance slightly smaller discounts for external and internal validity
Re item 4, it’s fair to note that I haven’t checked how conservative you’ve been on other assumptions, so if I did a replication of your work and it ended up being similar, then I agree that could be a reason.
Hey Sanjay, thanks for your comment
Internal validity-I think it is important to bear in mind there are a number of high quality RCT’s conducted with low quality of bias- given that, although this might not feel like it has as strong an evidence as bednets or vaccines, it does have a strong evidence base. You can see the cRCTs here- Abramsky et al (2014), Dunkle et al (2020), Leight et al (2020), Wagman et al (2015), Ogum Alangea et al (2020), Le Roux et al (2020), Chatterji et al (2020).
On surveys- To summarise, the concern you raise is that the programs aim to reduce social acceptability of VAWG, and mostly use local interviewers to assess incidence of violence, which may introduce social acceptability bias. Although this seems plausible, interviews were trained, conducted privately and used validated questionnaires. To quote one paper
As a result, I think this risk of bias is quite low. Also, I think that inherently, any impact evaluation of interventions in this space would require surveys.
3. External validity- A valid concern. I have two comments: (1)There are a number of studies in different settings which show positive results, suggesting external validity. (2) Although cultural and social drivers of violence vary, the intervention is co-designed with community and quite locally tailored, which mitigates some of the concern around external validity.
4. Meta-comment- I think that some of my estimates of the persistence of effects were quite conservative, which may counterbalance slightly smaller discounts for external and internal validity
Re item 4, it’s fair to note that I haven’t checked how conservative you’ve been on other assumptions, so if I did a replication of your work and it ended up being similar, then I agree that could be a reason.