Marcus and Austin, thank you for this research. Bringing more rigour to AI safety communications is so important.
Marcus, in the comments you say what you want from a video is “engagement/thought [i.e. learning effectiveness] or career change or protests or letters to the government, etc [i.e. call to action effectiveness]. This is why in the future, article reads/book reads are going to be weighted higher. Since reading a book is more effortful/engaging than passively watching a video.”
I’d like here to suggest that NON-passive video watching is technically easy and works well for certain educational use cases. However, it’s extremely rare—and indeed it’s not mentioned in your post or any of the comments. But perhaps you could add it to your list of non-passive media to investigate next?
Why bother? Well, let’s look at the effectiveness of YouTube…
1 – YouTube is an ad medium that measures exposure, not effectiveness
* The post sets out to measure how “cost-effective” AI safety YouTube videos are, which is a great goal. However, YT offers zero data on learning effectiveness — whether viewers understood, updated beliefs, or changed behavior. Thus all that can be measured is cost and watch-time. Not “effectiveness”.
* As has been pointed out, comment and comment analysis could be a proxy for quality of engagement; as well as citations of videos, likes, shares, stories of people taking action because of seeing a video etc. But these are all very hard to measure and quantify. I’ve certainly watched great videos without commenting on them. And we all know the quality of YT comments is wildly variable. (“YouTube comments are the Star Wars Cantina of the internet” as someone once said.)
* Therefore IMHO, a more accurate title would really be “What is the cost per view of AI Safety YouTubers?”
* This is NOT a criticism of the post—it’s inherent in the nature of YouTube, which is an advertising medium not an education medium: advertisers don’t care whether you understand a YT video, they just want their ad to be seen. But this fact is widely overlooked.
* So does this mean we abandon video? No—there are ways to use video combined with interactivity to make it possible to directly measure learning effectiveness.
2 – Some non-passive video formats actually do measure learning effectiveness
* Participatory interactive video-based workshops (like AI Basics: Thrills or Chills?) use interactivity to measure learning effectiveness directly.
* Example: participants answer “How important is the AI revolution?” at the start and end of the workshop. Average figures are:
* Start: ~20% Internet-level, ~70% Industrial Revolution-level, ~10% Evolution of humans-level
* end: ~0% Internet-level, ~60% Industrial Revolution-level, ~40% Evolution of humans-level
* This shows clear belief updates in high-quality audiences (Cornell, Cambridge, UK Cabinet Office). This demonstrates effectiveness in changing understanding—data that’s unavailable from regular YouTube videos.
* Medical education example: A gamified interactive video-based CPR training system (Lifesaver) showed 29% better outcomes than traditional face-to-face training in peer-reviewed medical research when learners were tested 6 months after training. Cost per session about 0.03% of the cost of f2f training. More than 1m people trained since launch.
* Again, this is measuring learning effectiveness rather than just views; apparently “many lives” have been saved by people who learned from Lifesaver but data has not been published.
* Due disclosure: I made both of these projects so obviously I’m not impartial. But I made them precisely because I wanted to make video more effective in training.
* Neither of these projects are on YouTube—so could an interactive approach like this work at scale there?
3 – It would be possible to do learning effectiveness measurement at scale with a non-passive YouTube video—but this is currently neglected
* Adapting interactivity to YouTube is technically straightforward—see for example the comic branching YT video “In Space with Markiplier”. Here the choices are just for fun; but as AI Basics shows, such choices can be meaningful.
* Such an approach could enable learning effectiveness measurement at scale on YT—tracking what people think and learn. Thus combining YouTube’s massive reach with genuine learning outcome data.
* The main barrier is conceptual, not technical. Most creators and funders lack experience with interactive video, making this obvious step seem foreign. But without it, you’re just measuring views—not effectiveness.
* Such a project would be novel but would at least make it possible to assess cost-effectiveness in learning impact.
Sorry to have gone on at such length—overall, I think using video in a non-passive way could offer a way towards measurable effectiveness, as well as reach. And research on this little-used approach would be very useful!
Marcus and Austin, thanks again for raising the vital but usually ignored question of “effectiveness” in AI safety communications on YouTube!
More about interactive videos I’ve worked on that might give context:
Trailer for Lifesaver—Emergency skills training
Trailer for AI Basics: Thrills or Chills? - AI and AI Safety awareness for non-technical people
More about the research on Lifesaver I talked about
Plus an example of interactive videos on YouTube:
In Space with Markiplier Non-serious interactive YT video from a leading YouTuber
Such a great post, thank you @Catherine Low🔸