Awesome! It is really good to finally see something like this happening! Luke and I have often talked about the need for it.
These are some quick, messy and rambling thoughts:
Next steps
In terms of the next steps, if I was in your shoes I’d look to do a single small easy project next as a test run.
I’d establish goals, resources/collaborative capacity (e.g., what do we collectively want to achieve in x time, and what do we have to get there). Then select someone to lead the project and who will support them and how and go from there. Also, get everyone on some project management platform like Asana.
[Note that I am not saying this is what you should do, just giving you my perspective based on what I think helped with keeping READI going. I am not a great project manager!]
Once everything is tested etc I’d do some research ideation and prioritisation with partner orgs. Once done you could then start tackling these research questions in order of priority.
I’d probably favour focusing on big and general issues (e.g, what is the best messaging for x in context y) rather than the needs of specific organisations (e.g., what should org do here on their website). Context shifting can be very demanding/inefficient and the advisory work might not generalise as much.
Risks, pitfalls, precautions
Related to the above, I actually think that the big challenge with a project like this is sustaining the coordination and collaboration.
I’d focus on keeping communication costs as low as possible. Make sure that you understand the needs of the key people in the collaboration and ensure that their cost-benefit ratios can sustain their involvement e.g., if people need publications/conversion rate improvements to justify their involvement then make sure that the projects are set up so that they will deliver those outcomes. Lots of one-to-many communication and not too many calls to reduce communication costs.
Research ideas
I have some ideas for the research questions/approaches (these are copied over from other places):
What if you got a range of different EA orgs who are promoting the same behaviour (e..g, donation) then implemented a series of interventions (e.g., a change in choice architecture) across their websites/campaigns (maybe on just a portion of traffic). Then use A/B testing to see what worked.
A research replication/expansion project (MANEA Labs? :P) where teams of researchers test relevant lab findings in the field with EA orgs. This database could offer some initial ideas to prioritise. Related to that, here is a post about this lab research which suggests that donor getting more choice potentially reduces their donation rates. Would be interesting to test in field on an EA charity aggregator like the TLYCS.
Potentially, using the EA survey, you could explore how different demographics, personality types and identities (e.g., social justice activist/climate change activist) interact with different moral views or arguments for key EA behaviours such as giving to effective charities/caring about the longterm etc. Could guide targeting.
Other comments
Unfortunately, I have no time to help at the moment so all I can offer are ideas/feedback for now.
More generally, I suspect that there will be some good opportunities to leverage the EA behaviour science community for some of this work (as I am sure you already know). Particularly if you can connect good data/research opportunities with upcoming academics keen to for publications. I’ll mention this initiative in the next edition.
I suspect that there will be some good opportunities to leverage the EA behaviour science community for some of this work (as I am sure you already know). Particularly if you can connect good data/research opportunities with upcoming academics keen to for publications. I’ll mention this initiative in the next edition.
I agree, and thanks for mentioning this. Perhaps a major part of our task, to make this work, is to come up with good systems and easy frameworks and ground rules for facilitating and enabling these cooperations. E.g.,
How can researchers/academics sign up and indicate their interests and suggestions? How is this information shared?
What must the researchers provide and promise (and commit to sharing with the community)
How do the partner organizations specify their interests and the areas they are willing to test and experiment with?
What must the partner organisations agree to do and share?
We provide some templates, generally-accepted protocols, tools (both statistical, ‘standard language’ templates, and IT/web design tools), and possible sources of funding (and coordination) for advertisements
Maybe we also organize some academic sources of value and credibility, like a conference
a major part of our task, to make this work, is to come up with good systems and easy frameworks and ground rules for facilitating and enabling these cooperations.
Agree! However, I’d personally avoid putting in much effort into any of that until there is clear evidence that enough researchers will get involved if you provide that.
Potentially, using the EA survey, you could explore how different demographics, personality types and identities (e.g., social justice activist/climate change activist) interact with different moral views or arguments for key EA behaviours such as giving to effective charities/caring about the longterm etc. Could guide targeting.
This could be very useful. Obviously the EA Survey is a particular slice, and probably not the group we are going for when we think of outreach. Patterns within this group (EA survey takers) may not correspond to the patterns in the relevant populations. Nonetheless, it is a start, and certainly a dataset I have a good handle on.
But I think we should also reach out and spend some resources to survey broader (non-EA or only adjacent-to-adjacent, or relevant representative) groups
I especially like the idea of “personality types and identities (e.g., social justice activist/climate change activist) interact with different moral views or arguments for key EA behaviours such as giving to effective charities/caring about the longterm etc.”
Recent work on de-biasing and awareness gets at this a bit, at least breaking things up by political affiliation. It might be worth our digging more closely into the Fehr, Mollerstrom, and Perez-T paper and data. It seems like a very powerful experiment tied to a rich dataset.
Yeah, I think that I forgot to add the other part of this idea which is that we would compare the EA survey against a sample from the general public so that we have both EA and non-EA responses to questions of interest. This might give a sense of where EAs and Non-EAs differ and therefore how best to message them. I think that all types of segementation, like political affiliation, would be very useful for EA organisational marketing but that’s probably worth validating with those orgs.
Related to that, here is a post about this lab research which suggests that donor getting more choice potentially reduces their donation rates. Would be interesting to test in the field on an EA charity aggregator like the TLYCS.
Absolutely. Your LinkedIn post outlines the need for some robust ‘real-world’ testing in this area, to supplement the small-stakes Prolific and Mturk samples in the authors use (which I need to read more carefully).
The ‘charity aggregator/charity choice platform’ is one particular relevant environment worth testing on, as distinct from the ‘specific charitable appeal’.
As to the ‘give the donors choice’ in particular, I envision some potentially countervailing things (pros/cons) of giving choice, some of which may be more relevant in a context involving ‘people seriously considering donating’ rather than ‘people asked to do an Mturk/Prolific study.’
Quick thoughts on this...
Cons of enabling choice (some examples): Choice paralysis, raising doubts, repugnance of ‘Sophie’s choice’, departure from ‘identifiable victim’ frame (although I read something recently suggesting that the evidence for the IVE may not be as strong as claimed!), calculating/comparing mindset may push out the empathetic/charitable mindset
Pros: Standard ‘allows better matching of consumer desires and options’, gives stronger sense of tangibility and agency (‘I gave to person A for schoolbooks’) reducing the sense of a futile ‘drop in the bucket’, seems more considerate and respectful of donor, appealing to their ‘wise judgment’
Actually consider this to be one of the more well replicated and evidenced findings. The 2016 meta-analysis HERE: supported it. However, I’ve recently been exposed to something or some discussion that seemed pretty credible suggesting this no longer should be taken as a robust result.
It may have come from metascience2021 (COS etc conference) -- if I dig it up again, I’ll post it and maybe write a twitter on it.
add and incorporate the parts of other recent reviews that are most relevant to effective giving, meta-analysis, and curation work (including yours)
work to find, build, and improve systems for assessing the state of the evidence on each barrier/motivator
summarize these into key ‘action points’ for our market testing project (in our gitbook/wiki)
But this part of the project is a bit distinct, requiring particular skills and focus. It might be worth trying to organize a way to divide things up so that this ‘literature synthesis and meta-analysis’ is ‘owned’ by a single member of our team. If we are ‘all doing everything’ context-switching can really be a drag.
Yeah, in hindsight, I that offers a better initial source for picking ideas to prioritise than the database which is just individual papers. I like the ideas you mentioned about the website and I think that there is definitely a need for more work on it—ideally with a single product owner who can improve the presentation and user experience and so on. I think it has a lot of promise!
What if you got a range of different EA orgs who are promoting the same behaviour (e..g, donation) then implemented a series of interventions (e.g., a change in choice architecture) across their websites/campaigns (maybe on just a portion of traffic). Then use A/B testing to see what worked.
This is very much what we are focused on doing; hoping to do soon. Donations and pledges are the obvious common denominator across many of our partner orgs. Being able to test a comparable change (in choice architecture, or messaging/framing, etc.) will be particularly helpful in allowing us to measure the robustness of any effect across contexts and platforms.
There is some tradeoff between ‘run the most statistically powerful test of a single pairing, A vs B’ and ‘run tests of across a set of messages A through Z using an algorithm (e.g., Kasy’s ‘Exploration sampling’) designed to yield the highest-value state of learning (cf ’reinforcement learning). We are planning to do some of each, compromising between these approaches.
Related to the above, I actually think that the big challenge with a project like this is sustaining the coordination and collaboration.
I’d focus on keeping communication costs as low as possible. Make sure that you understand the needs of the key people in the collaboration and ensure that their cost-benefit ratios can sustain their involvement e.g., if people need publications/conversion rate improvements to justify their involvement then make sure that the projects are set up so that they will deliver those outcomes.
Lots of one-to-many communication and not too many calls to reduce communication costs.
Yep! This makes me think of Christian and Griffiths ‘Algorithms to Live By’ on how Economies of scale work in reverse when something becomes ‘a sock matching problem’, or ‘everyone has to shake hands with everyone else’. I hope our use of Gitbook wiki (and maybe I’ll embed some videos) helps solve this prob. We’re also hoping to have efficient meetings with the whole group on particular themes (but not ’everyone has to weigh in on everyone else’s position in real time I guess).
‘Think about the needs and medium-term wins/outcomes for the collaborators’—very good idea.
We actually had some discussion of organizational issues which we cut out of the post (as it’s ‘our problem to sort out’). But this did include things like the risk of “‘too many stakeholders and goals leading to death by committee’, and ‘spreading ourselves too thin and being overambitious, with too many targets’”
This makes me think of Christian and Griffiths ‘Algorithms to Live By’ on how Economies of scale work in reverse when something becomes ‘a sock matching problem’, or ‘everyone has to shake hands with everyone else’.
Once everything is tested etc I’d do some research ideation and prioritisation with partner orgs. Once done you could then start tackling these research questions in order of priority.
By “once everything is tested etc” I am guessing that you may and once we have a good understanding and track record of doing and gaining value from each type of advertising, split-testing, etc?
We are working to identify the key research questions are, considering
the importance for EA organisations and fundraisers of these ‘tools’ (how much could these be used)
the existing evidence, both on general psychological providers and specifically in the charitable donation (and effective action) contexts
the extent to which we can practically conduct trials and learn from these … (feasibility and applicability of interventions, the observability of relevant outcomes, generalizability of evidence across platforms and contexts..)
I’d probably favour focusing on big and general issues (e.g, what is the best messaging for x in context y) rather than the needs of specific organisations (e.g., what should org do here on their website). Context shifting can be very demanding/inefficient and the advisory work might not generalise as much.
This is a good point, and I mostly agree, but of course we need to strike some balance. I suspect some things that seem like ‘very specific needs and practices for organization A’ might have greater applicability and relevance across organizations, and specific observations may lead to themes to study further. I’m also concerned about asserting and claiming to “test of general principles and theories of behavior” that prove to be extremely context-dependent. This is a major challenge of social science (and marketing), I think.
I’d establish goals, resources/collaborative capacity (e.g., what do we collectively want to achieve in x time, and what do we have to get there). Then select someone to lead the project and who will support them and how and go from there.
Agreed. I think we are still in the process of coming together and agreeing on which areas we can collaborate on and what the key priorities are. I suspect that it will be helpful to do some work and some small trials before meeting to decide on these overarching goals, to have a better sense of things. (Of course we have outlined these overarching goals in general, more or less as discussed above, but we need to fill in the specifics and the most promising angles and avenues.) I’m not sure we need one single ‘leader of everything’ but it will be good to have someone to ‘take ownership of each specific project’ (the whole ‘assigned to, reports to’ thing) as well as overall coordination.
Also, get everyone on some project management platform like Asana.
We are currently using a combination of Slack (conversation), Gitbook/Github (documentation and organized discussion of plans and information), and Airtable (structured information as data). I think you are right that we need to use these tools in specific ways that enable project management, keeping track of where we are on each project, who needs to do what, etc.
I’d still probably recommend Asana or a similar task manager if you can get everyone to try them. Micheal Noetel introduced me to it and he uses it very well with several research teams using concepts like ‘sprints’ and various other software design inspired approaches.
Thanks for the comments Peter, sorry I’m slow in responding. This is very helpful!
Next steps
In terms of the next steps, if I was in your shoes I’d look to do a single small easy project next as a test run.
Agreed!
[Aside: I am getting results from some ongoing related projects with relevant non-EA orgs, which I hope to write up soon, and I think Josh does/has/is as well. I believe Luke is also trialing some things atm. ]
We are indeed planning some small trials very soon, as you suggest, and I hope we can write these up and share them soon as well. It will be important to share what we learn not from the actual ‘outcomes of the trials’ but from the processes, considerations, etc.
Awesome! It is really good to finally see something like this happening! Luke and I have often talked about the need for it.
These are some quick, messy and rambling thoughts:
Next steps
In terms of the next steps, if I was in your shoes I’d look to do a single small easy project next as a test run.
I’d establish goals, resources/collaborative capacity (e.g., what do we collectively want to achieve in x time, and what do we have to get there). Then select someone to lead the project and who will support them and how and go from there. Also, get everyone on some project management platform like Asana.
[Note that I am not saying this is what you should do, just giving you my perspective based on what I think helped with keeping READI going. I am not a great project manager!]
Once everything is tested etc I’d do some research ideation and prioritisation with partner orgs. Once done you could then start tackling these research questions in order of priority.
I’d probably favour focusing on big and general issues (e.g, what is the best messaging for x in context y) rather than the needs of specific organisations (e.g., what should org do here on their website). Context shifting can be very demanding/inefficient and the advisory work might not generalise as much.
Risks, pitfalls, precautions
Related to the above, I actually think that the big challenge with a project like this is sustaining the coordination and collaboration.
I’d focus on keeping communication costs as low as possible. Make sure that you understand the needs of the key people in the collaboration and ensure that their cost-benefit ratios can sustain their involvement e.g., if people need publications/conversion rate improvements to justify their involvement then make sure that the projects are set up so that they will deliver those outcomes. Lots of one-to-many communication and not too many calls to reduce communication costs.
Research ideas
I have some ideas for the research questions/approaches (these are copied over from other places):
What if you got a range of different EA orgs who are promoting the same behaviour (e..g, donation) then implemented a series of interventions (e.g., a change in choice architecture) across their websites/campaigns (maybe on just a portion of traffic). Then use A/B testing to see what worked.
A research replication/expansion project (MANEA Labs? :P) where teams of researchers test relevant lab findings in the field with EA orgs. This database could offer some initial ideas to prioritise. Related to that, here is a post about this lab research which suggests that donor getting more choice potentially reduces their donation rates. Would be interesting to test in field on an EA charity aggregator like the TLYCS.
Potentially, using the EA survey, you could explore how different demographics, personality types and identities (e.g., social justice activist/climate change activist) interact with different moral views or arguments for key EA behaviours such as giving to effective charities/caring about the longterm etc. Could guide targeting.
Other comments
Unfortunately, I have no time to help at the moment so all I can offer are ideas/feedback for now.
More generally, I suspect that there will be some good opportunities to leverage the EA behaviour science community for some of this work (as I am sure you already know). Particularly if you can connect good data/research opportunities with upcoming academics keen to for publications. I’ll mention this initiative in the next edition.
I agree, and thanks for mentioning this. Perhaps a major part of our task, to make this work, is to come up with good systems and easy frameworks and ground rules for facilitating and enabling these cooperations. E.g.,
How can researchers/academics sign up and indicate their interests and suggestions? How is this information shared?
What must the researchers provide and promise (and commit to sharing with the community)
How do the partner organizations specify their interests and the areas they are willing to test and experiment with?
What must the partner organisations agree to do and share?
We provide some templates, generally-accepted protocols, tools (both statistical, ‘standard language’ templates, and IT/web design tools), and possible sources of funding (and coordination) for advertisements
Maybe we also organize some academic sources of value and credibility, like a conference
Agree! However, I’d personally avoid putting in much effort into any of that until there is clear evidence that enough researchers will get involved if you provide that.
This could be very useful. Obviously the EA Survey is a particular slice, and probably not the group we are going for when we think of outreach. Patterns within this group (EA survey takers) may not correspond to the patterns in the relevant populations. Nonetheless, it is a start, and certainly a dataset I have a good handle on.
But I think we should also reach out and spend some resources to survey broader (non-EA or only adjacent-to-adjacent, or relevant representative) groups
I especially like the idea of “personality types and identities (e.g., social justice activist/climate change activist) interact with different moral views or arguments for key EA behaviours such as giving to effective charities/caring about the longterm etc.”
Recent work on de-biasing and awareness gets at this a bit, at least breaking things up by political affiliation. It might be worth our digging more closely into the Fehr, Mollerstrom, and Perez-T paper and data. It seems like a very powerful experiment tied to a rich dataset.
Yeah, I think that I forgot to add the other part of this idea which is that we would compare the EA survey against a sample from the general public so that we have both EA and non-EA responses to questions of interest. This might give a sense of where EAs and Non-EAs differ and therefore how best to message them. I think that all types of segementation, like political affiliation, would be very useful for EA organisational marketing but that’s probably worth validating with those orgs.
Absolutely. Your LinkedIn post outlines the need for some robust ‘real-world’ testing in this area, to supplement the small-stakes Prolific and Mturk samples in the authors use (which I need to read more carefully).
The ‘charity aggregator/charity choice platform’ is one particular relevant environment worth testing on, as distinct from the ‘specific charitable appeal’.
As to the ‘give the donors choice’ in particular, I envision some potentially countervailing things (pros/cons) of giving choice, some of which may be more relevant in a context involving ‘people seriously considering donating’ rather than ‘people asked to do an Mturk/Prolific study.’
Quick thoughts on this...
Cons of enabling choice (some examples): Choice paralysis, raising doubts, repugnance of ‘Sophie’s choice’, departure from ‘identifiable victim’ frame (although I read something recently suggesting that the evidence for the IVE may not be as strong as claimed!), calculating/comparing mindset may push out the empathetic/charitable mindset
Pros: Standard ‘allows better matching of consumer desires and options’, gives stronger sense of tangibility and agency (‘I gave to person A for schoolbooks’) reducing the sense of a futile ‘drop in the bucket’, seems more considerate and respectful of donor, appealing to their ‘wise judgment’
Thanks—this is very useful. I’ll add it to my post so that more people read!
Where did you read about the research challenging the IVE btw? I’d be interested to read that.
Actually consider this to be one of the more well replicated and evidenced findings. The 2016 meta-analysis HERE: supported it. However, I’ve recently been exposed to something or some discussion that seemed pretty credible suggesting this no longer should be taken as a robust result.
It may have come from metascience2021 (COS etc conference) -- if I dig it up again, I’ll post it and maybe write a twitter on it.
Ah, just getting this turn of phrase—“Many Labs Project + EA”. That would be very nice to engineer and organize.
I’m glad to see that this is coming along; we’ve discussed it before. I’m working continue to update and build the “Increasing effective charitable giving: The puzzle, what we know, what we need to know next which works to outline the theories and evidence on the ‘barriers to effective giving’ and promising ‘tools’ for surmounting these. In doing this I hope to
add and incorporate the parts of other recent reviews that are most relevant to effective giving, meta-analysis, and curation work (including yours)
work to find, build, and improve systems for assessing the state of the evidence on each barrier/motivator
summarize these into key ‘action points’ for our market testing project (in our gitbook/wiki)
But this part of the project is a bit distinct, requiring particular skills and focus. It might be worth trying to organize a way to divide things up so that this ‘literature synthesis and meta-analysis’ is ‘owned’ by a single member of our team. If we are ‘all doing everything’ context-switching can really be a drag.
Yeah, in hindsight, I that offers a better initial source for picking ideas to prioritise than the database which is just individual papers. I like the ideas you mentioned about the website and I think that there is definitely a need for more work on it—ideally with a single product owner who can improve the presentation and user experience and so on. I think it has a lot of promise!
This is very much what we are focused on doing; hoping to do soon. Donations and pledges are the obvious common denominator across many of our partner orgs. Being able to test a comparable change (in choice architecture, or messaging/framing, etc.) will be particularly helpful in allowing us to measure the robustness of any effect across contexts and platforms.
There is some tradeoff between ‘run the most statistically powerful test of a single pairing, A vs B’ and ‘run tests of across a set of messages A through Z using an algorithm (e.g., Kasy’s ‘Exploration sampling’) designed to yield the highest-value state of learning (cf ’reinforcement learning). We are planning to do some of each, compromising between these approaches.
Yep! This makes me think of Christian and Griffiths ‘Algorithms to Live By’ on how Economies of scale work in reverse when something becomes ‘a sock matching problem’, or ‘everyone has to shake hands with everyone else’. I hope our use of Gitbook wiki (and maybe I’ll embed some videos) helps solve this prob. We’re also hoping to have efficient meetings with the whole group on particular themes (but not ’everyone has to weigh in on everyone else’s position in real time I guess).
‘Think about the needs and medium-term wins/outcomes for the collaborators’—very good idea.
We actually had some discussion of organizational issues which we cut out of the post (as it’s ‘our problem to sort out’). But this did include things like the risk of “‘too many stakeholders and goals leading to death by committee’, and ‘spreading ourselves too thin and being overambitious, with too many targets’”
Thanks! I hadn’t thought about that!
By “once everything is tested etc” I am guessing that you may and once we have a good understanding and track record of doing and gaining value from each type of advertising, split-testing, etc?
We are working to identify the key research questions are, considering
the importance for EA organisations and fundraisers of these ‘tools’ (how much could these be used)
the existing evidence, both on general psychological providers and specifically in the charitable donation (and effective action) contexts
the extent to which we can practically conduct trials and learn from these … (feasibility and applicability of interventions, the observability of relevant outcomes, generalizability of evidence across platforms and contexts..)
This is a good point, and I mostly agree, but of course we need to strike some balance. I suspect some things that seem like ‘very specific needs and practices for organization A’ might have greater applicability and relevance across organizations, and specific observations may lead to themes to study further. I’m also concerned about asserting and claiming to “test of general principles and theories of behavior” that prove to be extremely context-dependent. This is a major challenge of social science (and marketing), I think.
Agreed. I think we are still in the process of coming together and agreeing on which areas we can collaborate on and what the key priorities are. I suspect that it will be helpful to do some work and some small trials before meeting to decide on these overarching goals, to have a better sense of things. (Of course we have outlined these overarching goals in general, more or less as discussed above, but we need to fill in the specifics and the most promising angles and avenues.) I’m not sure we need one single ‘leader of everything’ but it will be good to have someone to ‘take ownership of each specific project’ (the whole ‘assigned to, reports to’ thing) as well as overall coordination.
We are currently using a combination of Slack (conversation), Gitbook/Github (documentation and organized discussion of plans and information), and Airtable (structured information as data). I think you are right that we need to use these tools in specific ways that enable project management, keeping track of where we are on each project, who needs to do what, etc.
I’d still probably recommend Asana or a similar task manager if you can get everyone to try them. Micheal Noetel introduced me to it and he uses it very well with several research teams using concepts like ‘sprints’ and various other software design inspired approaches.
Thanks for the comments Peter, sorry I’m slow in responding. This is very helpful!
Agreed!
[Aside: I am getting results from some ongoing related projects with relevant non-EA orgs, which I hope to write up soon, and I think Josh does/has/is as well. I believe Luke is also trialing some things atm. ]
We are indeed planning some small trials very soon, as you suggest, and I hope we can write these up and share them soon as well. It will be important to share what we learn not from the actual ‘outcomes of the trials’ but from the processes, considerations, etc.