I’m a product manager with a software, data science, and healthcare background. I worked on the EA Forum from 2022-2023, and previously spent 8 years as a software engineer and product manager in healthcare tech.
Sharang Phadke
Karma: 1,023
Hey Rocky, thanks for these thoughts
Searching with quotations is actually supported, and it does give you the desired behavior. The issue is that the snippets of text from posts that we’re highlighting are not properly focusing in on the search term. We’re planning to fix this soon.
Thanks for this feedback, we’ll keep that in mind. We definitely won’t overshadow the main common-to-everyone components of the frontpage.
Thanks Nathan! Do you happen to have an example of another site that does paragraph-level reactions well? The main question that pops to mind is is clutter vs value
Hi there, thanks for expressing your interest in post votes. I am pretty excited about trying them out on posts after lots of people gave us good reasons for this in our last feedback post. A major reason we started with the comment level was that turned out to be technically much easier, and we thought we’d be able to learn and iterate faster before considering the post level.
It’s worth clarifying that the Online team’s budget was much less than $2M budget before 2022, and a lot of the development of the Forum happened between 2018 and 2021 with one developer (JP). The budget and team grew in the last 12 months with FTX funding and significant growth in the EA movement, and now we’re evaluating what portion of the $2M budget to put towards continuing to develop the Forum versus new projects we’re exploring. It’s also worth noting that a portion of the $2M budget goes towards the content team (Lizka), and moderation and support contractors, who I think are super valuable but not the bulk of our budget, so I’ll instead focus on the product/engineering team and the platform in the next part of my response.
When I think of how the Forum compares to other platforms, I see some major tradeoffs. Compared to social media sites like reddit, twitter, and facebook groups, the EA Forum encourages long form content and high quality discussion much better, is a more focused space (with no ads or non-EA content), and indexes and organizes content better. Compared to individual or group blogs, the Forum is much more open (anyone can post), and prioritizes discussion better (commenting features are more hidden on many blogs).
Put together, I think these mean that the EA Forum is a more attractive space for EAs, and ultimately the network of users is a key factor in determining the value of the site.
This isn’t to say that the Forum can claim 100% counterfactual value for every interaction that happens in this space (compared to a world where it didn’t exist and we had a subreddit), but I do think the Forum has been valuable. I wish we could measure this value in a really clean way without too much effort (and we have near term plans to do more work here), but on the other hand I think it’s important we avoid the trap of focusing too much on measurement and evaluation for a public good that has fairly diffuse impact.
Hope that was helpful.
Thanks for this great question. It’s hard to answer this both because our model of how the Forum creates value includes lots of diffuse second and third order effects, as well as because well, predicting the future is hard! My guess is that the biggest factor here is the size of the EA movement. I would guess that the value of the Forum will scale as the EA movement and community scales.
Here’s a summary of how I think about the value of the Forum:
The Forum is an educational resource, providing different sorts of useful information to different users, from announcements and updates to novel research ideas and opinions.
The Forum helps segments of the EA community create common knowledge and make collective progress on important ideas, as well as coordinate on opportunities.
The Forum helps folks learn and get excited about EA ideas and causes and helps build a sense of community around them. This has the effect of motivating folks to work on the important ideas that are discussed in the space.
Often when we try to ask individual users how the Forum has been impactful for them, we hear that they enjoy reading and have learned about important ideas or made useful connections on the Forum. EA organizations tell us the Forum helps them establish their brand and ideas and later on recruit funding or employees.
Based on this, my guess is that the network of users that participate in the Forum is a huge factor in its value—a bigger EA movement would mean coordination and education would simply have a bigger audience and more potential for impact.
In terms of going big or maintaining what we have, I think we’re probably near to the edge of the impact we can get with tech improvements given the current size of the EA community, though there’s disagreement about this. A content specialist could add a lot of value in ways not through the tech, through content curation, special events, etc. That said, we do plan to take big swings that are outside of the core product of the EA forum, as JP mentions elsewhere.
Yes we are working on reactions now, and hope to roll something out within a week or two!
Thanks for flagging the issue with AI Narration. We’ve known that edited posts were a possibility, but hadn’t prioritized recreating the narration. My guess is this still doesn’t quite come to the top of the priority list since most posts do not change after the first publish, but we’re going to be tracking this and prioritizing over time.
Thanks for the suggestion on making the link icon copy the url! Someone actually suggested this just this week, and we’ve been working on it!
I really like this point, but I also empathize with the desire for strong public voices focused on pitching EA ideas to the world. It does feel like it’s easier to promote ideas when consistent sets of individuals are spokespeople and public influencers. I hope organizations and the community overall can promote a diverse set of individuals to that sort of platform without blind hero worship in the way you’ve described here!
What data infrastructure, broadly speaking, would make OWID’s work much easier and help your team investigate interesting and new data categories? For instance, what data have you found really hard to get a hold of in the past? What important data categories are particularly important but poorly organized out in the wild?
Relatedly, how does OWID prioritize what to focus on next in a way that prioritizes impactful research?
To be frank I don’t have a clear threshold in mind, but some points we have in mind are:
We can narrate all posts on the Forum with AI for the same price as a few posts a week with professional human narrators
Most engagement on posts happens in the first few days of publishing (definitely within the first week), which makes coordination to publish a human narration tricky
We expect AI narration to get better and better—already we think Type 3 Audio’s service is better than the NonLinear service (my subjective opinion, and largely because Type 3 is just using newer APIs)
All these factors make the threshold pretty high to keep paying for human narration. That said, we can definitely support people who narrate their own posts to override AI narration.
Our plan is to move to 100% AI soon, unless we get feedback that suggests users much prefer human narration.
(we may have some exceptions, like the best classic posts or the EA handbook)
At the moment I don’t think you can find human only narrations—but now I’m curious why you ask? (to be clear, we are evaluating how users feel about human vs AI and about a combined feed, so opinions are welcome)
Thanks for your thoughts. Our hypothesis based on interviews is that authors will appreciate the more nuanced positive feedback from reactions beyond “changed my mind”. That said, I think the easiest way to get to an answer is trying a set of reactions, seeing how readers and authors find them after a few weeks, and being open to revising.
I do think karma is a bit of a catch-all, and mostly I think that’s OK. We need a way to integrate a diversity of positive / negative sentiment into a metric, even if people have different reasons for their votes.
In terms of the tool tip, the other sites lack any explanation for their “upvote” or “like” buttons, and one of the reasons we have a whole explanation is to differentiate karma-votes from agree-votes, which other sites don’t have.
That said, It’s possible that reactions will help call out some of the behaviors we want to encourage on the Forum, like truth-seekingness and helpfulness. We’re also open to other suggestions for core reactions, but as mentioned in the post we do value simplicity.
Thanks, I think this is a good point. We’ll try out what a clear negative react at the post level could be to see if we can maintain this option for readers. FWIW, one reason we don’t have agree/disagree at the post level is that a post often contains a bunch of different points, which makes it hard to agree/disagree on the whole.
Fair points, we’ll definitely be iterating on this as we learn what works, and I should have been clear that this isn’t the one and only decision point!
The main reason for this is that in our interviews with an initial set of reactions that included negative attitudes, we found that authors felt fairly strongly compelled to respond to these reactions in some way—clarifying their writing, understanding the nature of the reaction, etc. With this experience in mind, we felt that it would be preferable for negative reactions to be articulated as comments with some explanation.
Reading the comments here so far, I think I’m more open to some very clear negative reactions to bring more balance particularly at the post level - (e.g. thumbs down or disagree at the post level).
But broadly, I’m personally more worried about the downsides of negative reactions to discussion coherence than the risks of reactions being positive-biased.
Thanks Michael, karma and author name do seem reasonable to add if we can easily keep episodes up to date from a technical perspective. Will put this on our list and work out how to prioritize it.
Interesting, thanks for your takes. One of the pros that we’ve been most excited about is sharing positive feedback beyond karma back with authors (some combination of your pros). The “serious” culture is super valuable, but also has the effect of scaring people away from posting their ideas, so we’re thinking about what the right balance is.
Anyway, thanks for your takes! We’ll probably post some ideas in the next week for more feedback.
You can give feature suggestions here any time.
Hi Kirsten, thanks for raising this issue. You’re right, we don’t have copyright permission. I’m going to be emailing authors of the specific set of posts we plan to narrate from before the license change.