Concerns with ACE research

Summary

I out­line some con­cerns about ACE’s re­search. I show that some of ACE’s older re­search is of low qual­ity, and should be re­moved from the web­site—ACE’s new Re­search Direc­tor agrees with this. More im­por­tantly, ACE’s re­search on the im­pact of cor­po­rate cam­paigns is flawed, and con­se­quently ACE’s re­search does not provide much rea­son to be­lieve that their recom­mended char­i­ties ac­tu­ally im­prove an­i­mal welfare. This is not a crit­i­cism of ACE’s recom­mended char­i­ties. I con­clude with some thoughts about how ACE could im­prove and note a cause for op­ti­mism, as a new Re­search Direc­tor ar­rives.

***

ACE’s re­search has been crit­i­cised in the past, most no­tably in a De­cem­ber 2016 blog­post by Har­ri­son Nathan. ACE’s re­search has im­proved since then with some of the most se­ri­ous prob­lems be­ing re­solved. For ex­am­ple, their Novem­ber 2017 leaflet­ing re­port was of a good stan­dard, and was a ma­jor im­prove­ment on their pre­vi­ous leaflet­ing re­port, which was of poor qual­ity. How­ever, I have ex­am­ined ACE’s re­search in 2018 and 2017 and be­lieve that it still con­tains some se­ri­ous flaws.

Eval­u­at­ing the im­pact of an­i­mal char­i­ties is gen­er­ally more difficult than eval­u­at­ing the im­pact of char­i­ties car­ry­ing out di­rect health in­ter­ven­tions be­cause ev­i­dence is sparse and much hinges on difficult ques­tions about an­i­mal sen­tience. Con­se­quently, ACE’s re­search team faces a harder prob­lem than GiveWell’s. How­ever, this point notwith­stand­ing, ACE’s re­search still falls short of what we should ex­pect. The prob­lems con­cern the failure to re­move low qual­ity older re­search and, more im­por­tantly, the rea­son­ing for judge­ments about the effec­tive­ness of cor­po­rate cam­paigns.

1. Old research

a. Many of ACE’s older in­ter­ven­tion re­ports are of low qual­ity, and should have been re­moved from ACE’s web­site.

b. ACE now ac­knowl­edges in pri­vate that these re­ports are of low qual­ity and for the most part does not rely on them in its char­ity re­views, but in many cases ACE fails to in­form reader of this fact.

2. Cor­po­rate campaigns

a. ACE does not have up to date re­search of suffi­cient qual­ity on the welfare effects of cor­po­rate cam­paigns.

b. ACE also does not check whether their recom­mended char­i­ties are gen­uinely causally re­spon­si­ble for the cor­po­rate policy suc­cesses that they claim.

The prob­lems I dis­cuss here per­tain only to the re­views of The Hu­mane League (THL) and An­i­mal Equal­ity, as I have not had time to look into their re­search on the Good Food In­sti­tute. This piece is in no way a cri­tique of ACE’s recom­mended char­i­ties.

I pub­lish this cri­tique firstly in the hope that it will en­courage an im­prove­ment in stan­dards at ACE; sec­ondly, that it will en­courage ex­ter­nal scrutiny of ACE re­search go­ing for­ward; and fi­nally that it will provide in­for­ma­tion rele­vant to in­di­vi­d­ual donors. In Au­gust 2018, ACE ap­pointed a new Re­search Direc­tor, Toni Adle­berg, who should not be held re­spon­si­ble for the mis­takes dis­cussed here. My in­ter­ac­tions with Ms Adle­berg and other mem­bers of ACE’s cur­rent re­search staff have been very pos­i­tive, and I am op­ti­mistic that there will be im­prove­ments in ACE’s re­search in the fu­ture.

Dis­clo­sure: I in­terned at ACE for a few months in 2015. I am cur­rently an un­paid ex­ter­nal re­search con­sul­tant for ACE, and have thus far car­ried out about 2 hours of work for them in this role. ACE re­viewed this piece prior to pub­li­ca­tion. The views ex­pressed here are my own, not those of Founders Pledge.

1. ACE’s view on the im­pact of their recom­mended charities

Be­fore we be­gin, it is use­ful to dis­t­in­guish three plat­forms in which ACE pre­sents its views:

1. In­ter­ven­tion reports

2. Cost-effec­tive­ness analyses

3. Cur­rent all-things-con­sid­ered view ex­pressed in char­ity reviews

The view ex­pressed in each these three things are of­ten differ­ent. For ex­am­ple, the view ex­pressed in the in­ter­ven­tion re­port on in­ves­ti­ga­tions is differ­ent to the view ex­pressed in the cost-effec­tive­ness analy­ses of in­ves­ti­ga­tions. ACE is also at pains to point out that their cost-effec­tive­ness analy­ses are only sup­posed to give a very rough pic­ture of the cost-effec­tive­ness of their char­i­ties, and they say that cost-effec­tive­ness analy­ses play “only a limited role in our over­all opinions of which char­i­ties and in­ter­ven­tions are most effec­tive”.[1]

With this clar­ified, we can now out­line ACE’s cur­rent view on THL and An­i­mal Equal­ity have im­pact. The in­ter­ven­tions pur­sued by these two or­gani­sa­tions can be di­vided into grass­roots out­reach and cor­po­rate out­reach. As I am defin­ing the term, grass­roots out­reach in­cludes things like leaflet­ing, on­line ads, so­cial me­dia out­reach, hu­mane ed­u­ca­tion, and un­der­cover in­ves­ti­ga­tions.

Cor­po­rate out­reach ac­counts for the ma­jor­ity of the mod­el­led im­pact of both char­i­ties in the cost-effec­tive­ness analy­ses of THL and An­i­mal Equal­ity: for THL, ~90% of the mod­el­led im­pact is from cor­po­rate out­reach, and ~10% from on­line out­reach; and for An­i­mal Equal­ity, >90% of the mod­el­led im­pact is from cor­po­rate out­reach. This seems con­sis­tent with ACE’s all-things-con­sid­ered view of which in­ter­ven­tions are likely to be im­pact­ful. ACE says that “[cor­po­rate cam­paigns] can be highly im­pact­ful when im­ple­mented thought­fully”.[2] ACE seems to be pes­simistic about grass­roots out­reach, say­ing:

“THL works to effect change through sev­eral differ­ent kinds of out­reach, in­clud­ing leaflet­ing, on­line ads, and ed­u­ca­tion. While there is lit­tle ev­i­dence available about the effec­tive­ness of these in­ter­ven­tions, we do not cur­rently recom­mend the use of leaflet­ing or on­line ads as we sus­pect that they are not as effec­tive as some other means of pub­lic out­reach.”[3]

Thus, ACE’s view as of Au­gust 2018 is that grass­roots ad­vo­cacy has close to no effect, though ACE does es­ti­mate that THL’s on­line out­reach is benefi­cial. I dis­cuss this in sec­tion 2b.

2. Grass­roots outreach

In this sec­tion, I dis­cuss ACE’s re­search on the im­pact of grass­roots out­reach. Due to lack of data on the im­pact of most forms of grass­roots out­reach, in their cost-effec­tive­ness analy­ses, ACE es­ti­mates the im­pact of most forms of grass­roots out­reach – hu­mane ed­u­ca­tion, in­ves­ti­ga­tions and An­i­mal Equal­ity’s i360 video out­reach – by as­sum­ing that they were X times as effec­tive as leaflet­ing. For ex­am­ple, we would, the ar­gu­ment goes, in­tu­itively ex­pect hu­mane ed­u­ca­tion to be ~5 times as cost-effec­tive as leaflet­ing, and leaflet­ing spares X an­i­mals from fac­tory farm­ing, so hu­mane ed­u­ca­tion spares ~5x an­i­mals from fac­tory farm­ing.

Prior to Novem­ber 2017, ACE be­lieved that leaflet­ing pro­duced fairly sub­stan­tial benefits. How­ever, in Novem­ber 2017, ACE pub­lished a new leaflet­ing re­port, stat­ing that their view on the im­pact of leaflet­ing had changed. They now be­lieve that leaflet­ing has tiny im­pact, and is in fact very slightly harm­ful. How­ever, ACE con­tinues to in­fer the im­pact of other forms of grass­roots out­reach from the im­pact of leaflet­ing, us­ing a scal­ing fac­tor. Thus, be­cause they judge hu­mane ed­u­ca­tion to be ~5 times as effec­tive as leaflet­ing and they es­ti­mate leaflet­ing to be harm­ful, they now es­ti­mate that the mean effect of hu­mane ed­u­ca­tion is 5x as harm­ful as leaflet­ing. It is not clear that con­tin­u­ing to use a scal­ing fac­tor makes sense when the sign of the effect has changed, but that is per­haps open to de­bate. They con­tinue to use this ap­proach for mod­el­ling the im­pact of hu­mane ed­u­ca­tion, in­ves­ti­ga­tions, and An­i­mal Equal­ity’s video out­reach.

Set­ting the speci­fics of the cost-effec­tive­ness anal­y­sis to one side, as men­tioned, ACE’s ap­par­ent cur­rent all-things-con­sid­ered view is that all forms of grass­roots out­reach (with the pos­si­ble ex­cep­tion of on­line ads) have close to zero effect. I agree that grass­roots out­reach is likely to have very small effects be­cause se­cur­ing dietary change through ad­vo­cacy ap­pears to be very difficult, with the best data sug­gest­ing that the per­centage of veg*ns has barely in­creased since the 1990s, as dis­cussed in this ex­cel­lent re­cent ACE blog.

2a. Investigations

Of THL and An­i­mal Equal­ity, only An­i­mal Equal­ity car­ries out in­ves­ti­ga­tions. The Feb 2016 in­ter­ven­tion re­port on in­ves­ti­ga­tions on ACE’s web­site eval­u­ates the im­pact of in­ves­ti­ga­tions in a differ­ent way to the cost-effec­tive­ness analy­ses of An­i­mal Equal­ity, and pro­duces a differ­ent es­ti­mate of their im­pact. The qual­ity of the in­ter­ven­tion re­port is low. One ma­jor con­cern is that the im­pact calcu­la­tions are done in a table, rather than in a spread­sheet or Guessti­mate model, mak­ing it difficult to un­der­stand im­por­tant in­puts, calcu­la­tions and out­puts.

Direct suffer­ing avoided per in­ter­ven­tion unit

An ‘in­ter­ven­tion unit’ is the mea­sure of the length of an in­ter­ven­tion. For in­ves­ti­ga­tions, 1 in­ter­ven­tion unit is 1 in­ves­ti­ga­tion. A unit of suffer­ing is a year of farmed cap­tivity, or equiv­a­lent, averted.

ACE pro­vides pes­simistic and re­al­is­tic (but not op­ti­mistic) es­ti­mates of ‘di­rect suffer­ing per in­ter­ven­tion unit’. No ex­pla­na­tion is given for these figures and they are ac­tu­ally the same as the ‘in­di­rect suffer­ing avoided per in­ter­ven­tion unit’. As­sum­ing that di­rect and in­di­rect suffer­ing averted are in­tended to be differ­ent, this sug­gests a mis­take that should have been no­ticed in a re­view of the page. Here is a screen­shot (I have added the column la­bels from a sep­a­rate screen­shot); these num­bers ap­pear with­out any ex­plana­tory calcu­la­tions.

(Screen­shot: 15th Au­gust 2018, about 95% of the way down the page)

Indi­rect suffer­ing avoided per in­ter­ven­tion unit

This is a screen­shot of the rele­vant part of the table on ‘in­di­rect suffer­ing avoided per in­ter­ven­tion unit’ (again, I have added the column la­bels from a differ­ent screen­shot fur­ther up the page):

(Screen­shot: 15th Au­gust 2018, about 95% of the way down the page)

There are three prob­lems with this. Firstly, ACE es­ti­mates the reach of a cam­paign stat­ing:

‘A per­son does not have to be di­rectly con­tacted by a staff mem­ber in or­der to be “reached”. They must merely en­counter the cam­paign in some ca­pac­ity, in­clud­ing liv­ing un­der a le­gal ju­ris­dic­tion be­ing tar­geted by a leg­is­la­tive cam­paign.’

The con­di­tion speci­fied in the sec­ond sen­tence is clearly in­ap­pro­pri­ate: ob­vi­ously, not ev­ery­one who lives in a le­gal ju­ris­dic­tion tar­geted by a cam­paign should count as be­ing reached by the cam­paign.

Se­condly, the op­ti­mistic es­ti­mate can­not be cor­rect and is there­fore not a rea­son­able ren­der­ing of ‘op­ti­mistic’. It im­plies that each un­der­cover in­ves­ti­ga­tion makes 2.8% of those reached veg­e­tar­ian or ve­gan. If, as es­ti­mated in the Feb 2016 in­ter­ven­tion re­port, each cam­paign will reach ~9m peo­ple, this would im­ply that each cam­paign will cre­ate ~250,000 new veg­e­tar­i­ans or ve­g­ans. Since An­i­mal Equal­ity runs 20 in­ves­ti­ga­tions a year,[4] this would im­ply that they alone will cre­ate ~5m new veg­e­tar­i­ans or ve­g­ans per year ex­clu­sively through their un­der­cover in­ves­ti­ga­tions. It is some­what difficult to es­ti­mate how many veg­e­tar­i­ans there are in the US be­cause ac­cu­rately track­ing dietary habits is challeng­ing and peo­ple tend to lie about their an­i­mal product con­sump­tion. While up to 6% of peo­ple in the US self-iden­tify as veg­e­tar­ian or ve­gan, but stud­ies that ask peo­ple whether they ate meat or fish on two con­sec­u­tive 24 hour pe­ri­ods show that the veg­e­tar­i­anism rate re­mains be­low 1.5%. See this ACE blog for more. Even on the higher es­ti­mate based on self-iden­ti­fi­ca­tion, the es­ti­mate of the im­pact of in­ves­ti­ga­tions can­not be cor­rect.

Thirdly, no ex­pla­na­tion is pro­vided for the pes­simistic or re­al­is­tic es­ti­mates; as you can see in the above screen­shot, head­line figures ap­pear with­out any calcu­la­tions. This is con­cern­ing. When I asked ACE about this in 2017, they said that the ba­sis for the figure was an­other page on the im­pacts of me­dia cov­er­age on meat de­mand, which is not refer­enced at that point on the cost-effec­tive­ness anal­y­sis of un­der­cover in­ves­ti­ga­tions.[5]

I pointed all of this out to ACE in Septem­ber 2017, and they ac­knowl­edged that this was a mis­take, but the page has still not been al­tered as of Au­gust 2018. I think this page should be im­me­di­ately re­moved. In con­ver­sa­tion with ACE’s new re­search team, they have told me that they were already con­sid­er­ing re­mov­ing these old in­ter­ven­tion re­ports be­fore I sent them this re­port, which is an en­courag­ing sign.[6]

2b. On­line ads

As I men­tioned above, in ACE’s cost-effec­tive­ness anal­y­sis, on­line ads are re­spon­si­ble for around 10% of THL’s im­pact. There are two main prob­lems with ACE’s treat­ment of on­line ads.

Firstly, it mod­els the effect of on­line ads in a differ­ent way to how it mod­els other similar forms of grass­roots out­reach. As dis­cussed above, ACE uses the ‘leaflet­ing scal­ing fac­tor’ ap­proach for all other forms of grass­roots out­reach. It does not use this ap­proach for on­line ads, de­spite the fact that it seems that what­ever ra­tio­nale jus­tifies us­ing the ‘leaflet­ing scal­ing fac­tor’ ap­proach for other forms of grass­roots out­reach must also ap­ply to on­line ads. For ex­am­ple, An­i­mal Equal­ity’s i360 video out­reach seems func­tion­ally similar to on­line ads, and yet a differ­ent ap­proach is taken for each.

Se­condly, in its cost-effec­tive­ness anal­y­sis of THL’s on­line ads, ACE re­lies on an Au­gust 2016 in­ter­ven­tion re­port on on­line ads. This re­port is highly opaque and it is very un­clear how ACE ar­rived at cer­tain cru­cial figures, such as num­ber of veg­e­tar­i­ans cre­ated per click on an on­line ad. Here is the cru­cial sec­tion in the cost-effec­tive­ness es­ti­mate of on­line ads:

Source: in­ter­ven­tion re­port, on­line ads (as of 10th Au­gust 2018)

The rea­son­ing pre­sented for these num­bers is limited, and be­cause the num­bers are pre­sented in a table rather than a spread­sheet, it is not clear how they are com­bined to­gether.

When I asked about this in Oc­to­ber 2017, ACE said the re­searcher who made the calcu­la­tion has left and that the fac­tors bear­ing on it were not weighted in a for­mal way that could be pub­li­cly ex­plained.[7] In­deed, in the com­pre­hen­sive re­view of An­i­mal Equal­ity, ACE says “There is lit­tle ev­i­dence, thus far, about the effec­tive­ness of on­line out­reach...”[8]

In sum­mary, ACE has used figures which it can­not pub­li­cly ex­plain and which it now re­jects, to es­ti­mate the im­pact of one of its recom­mended charities

3. Cor­po­rate campaigns

In their cost-effec­tive­ness analy­ses, ACE es­ti­mates that cor­po­rate cam­paigns ac­count for the vast ma­jor­ity of the im­pact of their recom­mended char­i­ties. It also ap­pears to be ACE’s all-things-con­sid­ered view in its char­ity re­views that cor­po­rate cam­paigns are one of the high­est im­pact in­ter­ven­tions. For ex­am­ple, in its re­view of THL, ACE says of in­ter­ven­tions aiming to in­fluence in­dus­try “We find that these in­ter­ven­tions can be highly im­pact­ful when im­ple­mented thought­fully”.[9]

How­ever, ACE’s re­search on cor­po­rate cam­paigns is flawed in four ways. Th­ese flaws are the most im­por­tant ones I have found, given how im­por­tant cor­po­rate cam­paigns are in ACE’s cur­rent re­search.

  1. In the Novem­ber 2014 in­ter­ven­tion re­port on cor­po­rate cam­paigns, ACE did not carry out a com­pre­hen­sive liter­a­ture re­view of the ev­i­dence on the welfare effects of cage-free sys­tems. They re­lied only on a pa­per by De Mol et al (2006), which seems quite badly flawed, as dis­cussed in a re­port by the Open Philan­thropy Pro­ject pub­lished in Septem­ber 2017 (spe­cific sec­tion of the re­port on De Mol et al (2006)).[10] The ev­i­dence on the welfare benefits of cage-free sys­tems is much more un­clear than sug­gested in ACE’s as­sess­ment of cor­po­rate cam­paigns.

  2. ACE no longer en­dorses the con­clu­sions in its in­ter­ven­tion re­port on cor­po­rate cam­paigns, but does not in­form the reader of this on the in­ter­ven­tion re­port page or any­where in the re­views of its recom­mended char­i­ties. The in­ter­ven­tion re­port on cor­po­rate cam­paigns ap­par­ently plays no role in the cost-effec­tive­ness es­ti­mate of cor­po­rate cam­paigns or in the char­ity re­views, but this is not ex­plained to the reader. This leaves a ma­jor jus­tifi­ca­tory gap in ACE’s re­search. In its cost-effec­tive­ness analy­ses, ACE es­ti­mates that their mean es­ti­mate of the “pro­por­tional im­prove­ment in welfare due to cage-free poli­cies” is ~0.05,[11] but pro­vides only a one sen­tence ex­pla­na­tion for this es­ti­mate. This cru­cial figure should be defended with mul­ti­ple page re­port and re­view of the sci­ence on the welfare effects of cages.

  3. ACE have been aware of the prob­lems with the De Mol et al (2006) pa­per at least since Septem­ber 2017, and told me in Oc­to­ber 2017 that re­vis­ing the cor­po­rate cam­paigns in­ter­ven­tion re­port was a high pri­or­ity for the com­ing year.[12] As of Au­gust 2018, this re­port has still not been up­dated and ACE have told me that it will very likely be up­dated in early 2019.[13] The re­port has now not been up­dated for al­most four years. This progress seems much too slow given how im­por­tant cor­po­rate cam­paigns are in ACE’s eval­u­a­tions, and ac­cord­ing to con­ven­tional wis­dom in effec­tive an­i­mal ad­vo­cacy.

  4. In its re­view of THL’s and An­i­mal Equal­ity’s cor­po­rate out­reach, ACE re­lies only on the char­i­ties’ self-re­ported cor­po­rate policy suc­cesses, which it then dis­counts by an ar­bi­trary un­cer­tainty fac­tor: ~0.4 for both An­i­mal Equal­ity and THL. ACE does not check with third party news sources, ex­perts or with the com­pa­nies them­selves on whether the claims of the char­i­ties are ac­cu­rate.[14] This seems to me like ba­sic due dili­gence that one should carry out when as­sess­ing cor­po­rate ad­vo­cacy.

Conclusion

ACE’s re­search has im­proved some­what over the last year or so, but se­ri­ous prob­lems re­main. I be­lieve that at pre­sent, donors in­ter­ested in an­i­mal welfare would learn lit­tle from ACE’s re­search. Some of the prob­lems (those re­gard­ing grass­roots out­reach) chiefly con­cern the failure to re­move older poor qual­ity re­search and to ad­e­quately com­mu­ni­cate their views to con­sumers of their re­search. Other prob­lems (those re­gard­ing cor­po­rate out­reach) are highly rele­vant to cru­cial judge­ments about whether ACE’s recom­mended char­i­ties have any im­pact at all, or are even harm­ful (which is pos­si­ble on some ap­par­ently rea­son­able views on the effects of cage-free sys­tems).

1. Grass­roots outreach

a. ACE be­lieves that grass­roots out­reach has small to zero im­pact, which seems right.

b. How­ever, ACE’s in­ter­ven­tion re­ports on many forms of grass­roots out­reach are of poor qual­ity and should have been re­moved from ACE’s web­site.

c. ACE does not com­mu­ni­cate their cur­rent view of the qual­ity of some of these re­ports, nor does it ex­plain that the con­clu­sions therein di­verge from their cur­rent view.

2. Cor­po­rate outreach

a. Cor­po­rate out­reach ac­counts for most of the recom­mended char­i­ties’ im­pact in ACE cost-effec­tive­ness analy­ses and in their all-things-con­sid­ered view of the im­pact of their char­i­ties.

b. In spite of that, ACE still does not have a high qual­ity up to date re­view of cor­po­rate out­reach or of the welfare effects cage-free sys­tems.

c. ACE does not in­de­pen­dently check the claims of char­i­ties about their cor­po­rate policy suc­cesses.

To be clear, I do not think that these prob­lems can be ex­plained by the fact that cost-effec­tive­ness analy­ses for an­i­mal in­ter­ven­tions are ex­tremely un­cer­tain and should not be taken overly liter­ally. The main prob­lems per­tain to re­search ques­tions ab­solutely cru­cial to figur­ing out whether an­i­mal char­i­ties are benefi­cial or harm­ful, which have not been ad­e­quately an­swered by ACE.

As I men­tioned at the start of this brief, ACE re­search has im­proved over the last two years, and I am op­ti­mistic that with the ar­rival of their new Re­search Direc­tor, Toni Adle­berg, ACE’s re­search will im­prove at a faster pace in the fu­ture. Go­ing for­ward, I think the main im­prove­ments would come from:

1. Re­mov­ing the older lower qual­ity in­ter­ven­tion re­ports from the ACE web­site. As I have men­tioned, it is a pos­i­tive sign that ACE were already con­sid­er­ing do­ing this be­fore I sent them this re­port.

2. Pro­duc­ing a re­port on the welfare effects of cor­po­rate cam­paigns and in­de­pen­dently check­ing char­i­ties’ claimed cor­po­rate policy suc­cesses.

3. Reg­u­larly up­dat­ing in­ter­ven­tion re­ports and char­ity re­views to re­flect ACE’s cur­rent views, rather than up­dat­ing them whole­sale ev­ery >1 year.

I should also add that I do not think it would be wise for ACE to rely less on quan­tifi­ca­tion in cost-effec­tive­ness analy­ses in the fu­ture. Quan­tify­ing the im­pact of an­i­mal char­i­ties is very difficult, but with­out quan­tify­ing im­pact, com­par­i­sons be­tween char­i­ties be­come much more difficult.



[1] See crite­rion 3 of the re­view of THL here—https://​​an­i­malchar­i­tye­val­u­a­tors.org/​​char­ity-re­view/​​the-hu­mane-league/​​#c3

[2] See the ‘in­fluenc­ing in­dus­try’ sec­tion un­der Cri­te­rion 2 here—https://​​an­i­malchar­i­tye­val­u­a­tors.org/​​char­ity-re­view/​​the-hu­mane-league/​​#c2

[3] See the ‘in­fluenc­ing pub­lic opinion’ sec­tion un­der Cri­te­rion 2 here—https://​​an­i­malchar­i­tye­val­u­a­tors.org/​​char­ity-re­view/​​the-hu­mane-league/​​#c2

[4] See ACE’s An­i­mal Equal­ity cost-effec­tive­ness anal­y­sis.

[5] Con­ver­sa­tion with Alli­son Smith, former Re­search Direc­tor at ACE, 12th Oc­to­ber 2017.

[6] Con­ver­sa­tion with Toni Adle­berg and Jamie Spur­geon, 28th Au­gust 2018.

[7] Con­ver­sa­tion with Alli­son Smith of ACE, 12th Oc­to­ber 2017.

[8] https://​​an­i­malchar­i­tye­val­u­a­tors.org/​​char­ity-re­view/​​an­i­mal-equal­ity/​​#c1

[9] See the ‘in­fluenc­ing in­dus­try’ sec­tion un­der Cri­te­rion 2 here—https://​​an­i­malchar­i­tye­val­u­a­tors.org/​​char­ity-re­view/​​the-hu­mane-league/​​#c2

[10] Rudi M. De Mol et al., “A Com­puter Model for Welfare Assess­ment of Poul­try Pro­duc­tion Sys­tems for Lay­ing Hens,” NJAS Wa­gen­ingen Jour­nal of Life Sciences 54, no. 2 (Oc­to­ber 25, 2006): 157–68.

[11] See the cost-effec­tive­ness anal­y­sis of THL at https://​​www.getguessti­mate.com/​​mod­els/​​9604?to­ken=Ckr­b­zlMlrvv6XJOuZ3wmLJwtvnkJoD2aIY79V6DuAe3Wk5t0WP-q4O5Eu9eDuL-jzqz5w1I_YARjtTakfkWesA

[12] Con­ver­sa­tion with Alli­son Smith of ACE, 12th Oc­to­ber 2017.

[13] Email cor­re­spon­dence with Jamie Spur­geon of ACE, July 17th 2018.

[14] Email cor­re­spon­dence with Jamie Spur­geon of ACE, July 17th 2018.