At Animal Ask we did later hear some of that feedback ourselves and one of our early projects failed for similar reasons. Our programs are very group-led, as in we select our research priorities based on groups looking to pursue new campaigns. This means the majority of our projects tend to focus on policy rather than corporate work, given more groups consider new country-specific campaigns and want research to inform this decision.
In the original report from CE, they do account for the consolidation of corporate work behind a few asks. They expected the research on corporate work to be ‘ongoing’ deeper’ and ‘more focused research’. So strategically would look more like research throughout the previous corporate campaign to inform the next with a low probability of updating any specific ask. The expectation is that it could be many years between the formation of corporate asks.
So in fact this consolidation was highlighted in the incubation program as a reason success could have so much impact. As with the large amount of resources the movement devotes to these consolidated corporate asks ensuring these are optimised is essential.
As Ren outlined we have a couple of recent, more detailed evaluations and we have found that the main limitations on our impact are factors only a minority of advisors in the animal space highlighted. These are constraints from other organisation stakeholders either upper management (when the campaigns team had updated on our findings but there was momentum behind another campaign) or funders (particular individual or smaller donors who are typicaly less research motivated than OPP, EAAWF, ACE etc.)
You can see this was the main concern for CE researchers in the original report. “Organizations in the animal space are increasingly aware of the importance of research, but often there are many factors to consider, including logistical ease, momentum, and donor interest. It is possible that this research would not be the determining factor in many cases”.
Awesome great post, fantastic to see variations of this intervention being considered. The main concerns we focused on with unguided vs guided delivery methods for self-help were recruitment costs and retention.
As you show in the relative engagement in recent trials retention seems to play out in favour of unguided given the lower costs. I think this is a fair read, from what I’ve seen from comparing the net effect size of these interventions the no less than half figure is about right too.
For recruitment, given the average rate of download for apps we discounted organic growth as you have. Then the figure cited for cost per install is correct (can see another source here) but for general installs. For the unguided self-help intervention, we want a subset of people with the/a condition we are targeting (~5-15%). We then want to turn those installs into active engaged users of the program. I don’t know how much these would increase the costs compared to the $0.02 - $0.10 figure. Plausibly 6-20 times for targeting the subgroup with the condition for installs (although even the general population may benefit a bit from the intervention). Then the cost of turning those installs into active users. Some stats suggest that on day one there is a 23.01% retention rate and then as low as 2.59% by day 30 (source, similar to your 3.3% real-world app data). So again looking at maybe a 4 to 40 times increase in recruitment costs depending. Overall that could increase costs for recruiting an active user by 24 to 800 times (Or $0.48 to $8).
Although these would be identical across unguided vs guided the main consideration then becomes does it costs more to recruit more users or to guide existing users and increase the follow-through/ effect size. Adding the above factors quickly to your CEA for recruitment cost gives a cost per active user of 9.6 (1.6 to 32) and cost-effectiveness for a guided of mean 24 (12 to 47) and unguided of mean 21 (4.8 to 66). Taking all of this at face value which version looks more cost-effective will depend a lot on exactly where and how the intervention is being employed. Even then there is uncertainty with some of these figures in a real-world setting.
For the unguided app model you outline I agree if successful this would be incredibly cost-effective. Although at present I’d still be uncertain which version would look best. Ultimately that’s up to groups implementing this family of interventions, like Kaya Guides, to explore through implementing and experimenting.