Thomas Billington’s EAForum account. I am the co-founder of Fish Welfare Initiative. I also work as a Monitoring, Evaluation, and Learning Associate at The Mission Motor.
My particular areas of passion are:
Creating change for farmed animals in low and middle-income countries
Bringing better monitoring and evaluation to the animal movement
Cool! Would be keen to sign on to a mailing list :)
Strong agree.
And I understand why this is a problem. It can be hard to independently create contacts in these spaces from scratch, and there is an aspect of not knowing what you don’t know at play. I’m almost certain I am committing the same mistake in multiple places in my work.
Would be interested to think about solutions here. Like perhaps a group such as Consultants For Impact could take on a role of knowledge dispersal, doing things like getting project management experts to give a talks at EAGs?
For sure!
I would say that EAs are missing large parts of M&E, including:
- The formal setting of key questions / assumptions that form the basis of what you will focus on trying to answer
- Creating formal monitoring frameworks (e.g. a log frame) that takes these questions / assumptions and identifies practical indicators and a method of measuring them
- I think EAs don’t use the full diversity of M&E tools. In my experience we tend to over-index on surveys (vs., say, interviews, focus group discussion, or observational data)
- I think considering the frequency of use of surveys, we could generally up-skill in high-quality survey design
- Using a diverse set of evaluation types (EAs generally know about RCTs, but these are such a narrow slice of the available evaluation types)
In general I think we care about M&E but lack experience in the formal processes of it, especially monitoring. So application is patchy and not generally in line with best practices.
I should perhaps clarify that I am mostly talking about the non-global development side of EA. I think their norms for M&E are significantly better.
Intrac’s M&E universe is one place to see an overview of what M&E entails. I think also The Mission Motor intends to create more resources on these topics in the future :)
Congrats on launching!
Excited to follow your work.
My advice for EAs who want to skill up in a neglected area:
Get a good mentor who broadly aligns with EA values but already has the skill. I’ve had three of these relationships (with experts in fish welfare, M&E, and agricultural development). In each case, it was symbiotic: I could offer EA knowledge and connections in return for them teaching me their skillset.
Book calls with as many relevant people as possible in the field. Shamelessly ask for help, even from people only tangentially related to what you’re trying to learn. The more you snowball contacts, the better your chances of finding a mentor. Also, having lots of people explain the fundamentals to you repeatedly is very useful, as it helps you learn the core ideas of the field.
Read a lot. Try to consume a lot of media from the chosen field. You know you’re doing well when you start to notice consistent underlying patterns and premises (again, the core ideas).
Start applying it before you’re ready. Look for small projects you can use as practice for the new skill.
In general, when learning a new skill, Andrea Gunn’s talk on training leaders offers a lot of good insight. I also made a one-page summary of her talk.
I have historically been able to do this upskilling as a side project to my existing job.
Glad it’s helpful!
Have some level of scepticism that the total “util count” from life on earth will be net-positive. I’m also in general wary of impact that is too long-term speculative.
That’s the one!
Weird that I remember it being half the size.
Anecdotal evidence, but I would attribute at least part of the reason I ended up in an EA career to a 2 day event of ~12 people hosted by EA London.
Tom from The Mission Motor here (have also done some more general M&E consulting in the EA space).
In my opinion, there is a lot of room for growth in M&E best practice in the (non-global development) EA space. There is often an appetite for M&E, but there is a real lack of knowledge and expertise. This to me presents a big opportunity where behaviour change is needed and also easier to obtain (though still not easy).
You can also have a snowball effect if you change the norms in the EA space. Personally, I feel like current concepts of measuring progress and impact in (again, mostly non-global dev) EA spaces are rudimentary. For example, we tend to focus on speculative cost-effective analysis as the primary method of assessing projects, even when they are still in the ideation phase. I believe this is, at least in part, because there is not an understanding of alternate M&E tools (e.g. setting up a good monitoring system for evaluating a theory of change).
I exclude the global development space as ideas from groups like IPA and IDInsight have permeated more there.
So my opinion would be strongly pro-working with EAs.
Hey, Tom from Fish Welfare Initiative here.
We really appreciate the time and thought given to evaluating FWI and the fish welfare space more generally from a current cost-effectiveness lens. Of course, we are aware that FWI’s programming as it currently stands is not as cost-effective as cage-free or broiler campaigns, we understand that this is the prioritization of some donors, and so we think this is something that is important for potential supporters to understand (see our best arguments against donating to FWI).
Our general stance on the value of Fish Welfare Initiative is that we are a project with both high levels of learning for the movement and overall promise to become significantly more cost-effective in the future. We believe that working in low and middle income countries like India is critical for the long-term success of our movement, and thus having research and action firmly rooted in the field is necessary. We, therefore, find comparing our work at this stage to the estimated effectiveness of some of the biggest successes of our movement to only be a small negative update.
The broader point of the post, however, is also that aquatic animal welfare projects like FWI may struggle to become as cost-effective as other projects when discounted for fishes’ “welfare range”. We find this valid as a concern for some to have. However, we also feel that welfare ranges are a relatively nascent field of study, that true success in the animal movement likely involves significant work for fishes, and that the learnings from FWI’s work cross-apply beyond just fishes. So again, we find this only a small negative update.
Also to note, FWI does not endorse the numbers used by IPA or this review as to the magnitude of suffering alleviated by FWI’s programming. These numbers are based on Ambitious Impact’s original cost-effectiveness calculations, which are a far-cry from our actual work. Real numbers on how much suffering we alleviate on average per fish is something we are working on building a process for (see our welfare assessment protocol), but the in-field complexities mean that we do not believe it can be assessed through secondary research.
FWI, of course, is a biased opinion on all this (although we do believe we have a very intimate understanding of the ground situation), and so it would be reasonable to take our opinion with some salt. We are always open to feedback, and thank you again to Vasco for putting this together.
Fwiw, one area where we do strongly agree with you is on your assessment of Shrimp Welfare Project. They’re awesome, and we encourage people to support them.
Aaron!
Thanks for posting :) I’m coming at this as someone who spent a lot of time running on-farm research at Fish Welfare Initiative (and planning to do more through my new charity). I broadly agree, but I’d add a few caveats:
The core issue with farms as “welfare labs” is heterogeneity (variability). Especially in LMICs, farms are messy, uncontrolled environments where confounding variables easily creep in. That creates a lot of statistical noise. If you’re aiming for high certainty, farms can make that difficult.
Relatedly, on-farm research makes it harder to isolate specific effects. You mention the benefit of insights collected “under real commercial conditions”, and I agree that ultimately effectiveness in the real world is what matters. But there’s also strong value in isolating variables to understand mechanisms. If we’re testing whether pigs prefer straw or wood shavings, we may not want to simultaneously capture differences in how farmers manage those materials. Otherwise, when results don’t support our hypothesis, we don’t know whether we’re observing animal preference or management differences. That’s why the difference between efficacy (does it work in controlled conditions?) and effectiveness (does it work in real-world conditions?) is useful.
So in sum: on-farm research is a valuable tool, but it can’t replace controlled research, and I’d hesitate to frame it as more valuable.
I also think two ideas may be getting conflated: monitoring existing farm conditions vs running experiments on farms. Both can be useful, but both need a clear use-case for the data. I agree, though, that there’s strong potential on both fronts.