This post is very useful among other reasons to see how thoughtful the work came about, what went into it, why you think it worked, and what metrics and data you look at.
Thanks for getting it out the door when you did. It meant I actually had time to read it.
At ACE we have this Better for Animals resource that reviews evidence for various interventions. It also covers documentaries briefly. Pro-animal docs’ lessons might not be particularly applicable to AI safety ones, but maybe it gets you thinking about those CTAs.
Because that’s the main question I have after reading this post, what are you evaluating for?
It is very impressive what you all did and I envy/admire the results. You made a beautiful product that people respect and many, many people consumed. But that does not equal changing the path we’re on. Or maybe it does if awareness is the thing that you think shifts our future.
I’m not suggesting you didn’t achieve your purpose, just that it’s not crystal clear from this retrospective what that was and how you measure getting there. Maybe in the next evaluation report, you can share the purpose and outcomes you set for the video project, or the theory of change.
If it’s primarily awareness that you’re going for than the currently listed data points seem great. If you want public discussion, than optimizing for those comments might be the thing to try and the CTA can be something around asking people to leave in the comments (or when they share the video) about what worries/questions they have. If you need people to talk to their representatives, then your CTA and content maybe need another slight adjustment to prime them for that. (IDK if public discussion or political activism are useful steps on your path to impact. Again, I know embarrassingly little about this cause area.)
I admit that I’m curious in large part because I hope to learn from it for my own work. There are so many animal welfare documentaries and quite a few voices (and some funders) saying we need more of them based on their personal experience or intuition. But we don’t have clear evidence that that these are a cost-effective method to help animals.
Now, the goals and context for AI safety are very different from those for animal welfare so it might work and work differently for you. But if I better understand what you’re trying to do, then I can more easily see what the animal advocacy community should (not) copy.
Separate but somewhat related: how do you think about the use of videos as a path to change against different AI timelines? Do short timelines mean that time-consuming, high-production value video are less or more useful than some other alternative, e.g. high volume of low-cost, quick posts at strategic spaces. This is not my area of expertise so I’m honestly just wondering. (Again partially so I can learn from it to help animals better.)
Anyway, here’s that resource and thanks again for sharing these insights:
Basically you’re right about all of the above, and we don’t yet have an end metric we’ve figured out, so we don’t know how much impact we’re having! It’s the biggest open question for our team right now, and I hope to get clarity on it in the next quarter or two and share that.
Re: Timelines—we don’t have super strong views on timelines, but the amount of weight I put on short timelines do I think mean that any longer leads than an average of 3 months per video seem bad to me. Whether we should try to go for e.g. 1/week I’m not yet sure on.
Hey Chana,
This post is very useful among other reasons to see how thoughtful the work came about, what went into it, why you think it worked, and what metrics and data you look at.
Thanks for getting it out the door when you did. It meant I actually had time to read it.
At ACE we have this Better for Animals resource that reviews evidence for various interventions. It also covers documentaries briefly. Pro-animal docs’ lessons might not be particularly applicable to AI safety ones, but maybe it gets you thinking about those CTAs.
Because that’s the main question I have after reading this post, what are you evaluating for?
It is very impressive what you all did and I envy/admire the results. You made a beautiful product that people respect and many, many people consumed. But that does not equal changing the path we’re on. Or maybe it does if awareness is the thing that you think shifts our future.
I’m not suggesting you didn’t achieve your purpose, just that it’s not crystal clear from this retrospective what that was and how you measure getting there. Maybe in the next evaluation report, you can share the purpose and outcomes you set for the video project, or the theory of change.
If it’s primarily awareness that you’re going for than the currently listed data points seem great. If you want public discussion, than optimizing for those comments might be the thing to try and the CTA can be something around asking people to leave in the comments (or when they share the video) about what worries/questions they have. If you need people to talk to their representatives, then your CTA and content maybe need another slight adjustment to prime them for that. (IDK if public discussion or political activism are useful steps on your path to impact. Again, I know embarrassingly little about this cause area.)
I admit that I’m curious in large part because I hope to learn from it for my own work. There are so many animal welfare documentaries and quite a few voices (and some funders) saying we need more of them based on their personal experience or intuition. But we don’t have clear evidence that that these are a cost-effective method to help animals.
Now, the goals and context for AI safety are very different from those for animal welfare so it might work and work differently for you. But if I better understand what you’re trying to do, then I can more easily see what the animal advocacy community should (not) copy.
Separate but somewhat related: how do you think about the use of videos as a path to change against different AI timelines? Do short timelines mean that time-consuming, high-production value video are less or more useful than some other alternative, e.g. high volume of low-cost, quick posts at strategic spaces. This is not my area of expertise so I’m honestly just wondering. (Again partially so I can learn from it to help animals better.)
Anyway, here’s that resource and thanks again for sharing these insights:
https://docs.google.com/document/d/1O0ylEEQJMQMTBlHDHcNZwvTgifi7TNyd6GpabC_4VT0/edit?usp=drivesdk
One more caveat: some of the studies we looked at might be based on a time when media was consumed very differently.
Basically you’re right about all of the above, and we don’t yet have an end metric we’ve figured out, so we don’t know how much impact we’re having! It’s the biggest open question for our team right now, and I hope to get clarity on it in the next quarter or two and share that.
Re: Timelines—we don’t have super strong views on timelines, but the amount of weight I put on short timelines do I think mean that any longer leads than an average of 3 months per video seem bad to me. Whether we should try to go for e.g. 1/week I’m not yet sure on.
Thanks so much for your comment!