Polis (Cluster) Poll: What are our opinions on EA, governance and FTX? (Now with visualisation)
Tl;dr
I am editing this as I go
Polis is a tool for understanding community opinion
It allows us to see clusters of similar opinions
It not a representative poll nor does it claim to be- it’s not saying how many people think X just that the cluster exists.
What are people’s views on community/governance/FTX in EA?
It creates a fun/interesting visualisation when enough people have voted. I think it will be about the most useful way to understand what different groups feel right now.
Add your own comments
It was mentioned on the 80k podcast (by Audrey Tang)
It is anonymous unless you log in
I will add the visualisations when we have enough votes. Polis is being a bit slow today so this may be delayed.
Link here: https://pol.is/6rk9hcav3w
Report here (this now works): https://pol.is/report/r7yrwpymn6yadbzwm3iyi
Longer intro
It is easy to have a false sense of what people think. But we don’t have to—we have tools that allow us to hear what people actually think. One is representative polling. This isn’t that. Here we can do cluster analysis (not a technical term) and look at what opinions are held in common. Do those who think Longtermisms is true tend to think that certain solutions are better than others? If so, why.
It shouldn’t be used to say X% of EAs think this. That’s not what this tool is for. Instead, it should suggest interesting questions to ask in future.
It would be very easy to use the loudest voices to decided what EAs really think. But that is unwise. Instead, if we want community input we should try and understand what groups form the community as a whole and understand what underlies the desires of each group.
Example question
Report
https://pol.is/report/r7yrwpymn6yadbzwm3iyi
Clusters and visualisation (Coming when Polis decides to work)
There are three clusters currently:
Group A largely think this is overrated and want clear suggestions
Group B wants more robust internal mechanisms and hold EA responsible for not spotting this
Group C is similar to group A but is more likely to hold those views. They also thinks that flirting at EAGs after the day time should be okay (ed: I misread this before and apologise, since it’s a big difference)
The majority views:
Note that the red bars are thing that everyone disagrees with.
- Doing EA Better by (17 Jan 2023 20:09 UTC; 257 points)
- Moving Toward More Concrete Proposals for Reform by (27 Jan 2023 0:36 UTC; 96 points)
- 's comment on Doing EA Better by (17 Jan 2023 21:39 UTC; 50 points)
- 's comment on A statement and an apology by (23 Feb 2023 9:09 UTC; 27 points)
- Polis: Why and How to Use it by (1 Feb 2023 14:03 UTC; 18 points)
- 's comment on Babel’s Quick takes by (7 Dec 2022 13:51 UTC; 17 points)
- Polis: Why and How to Use it by (LessWrong; 1 Feb 2023 14:03 UTC; 3 points)
In case the results end up getting used to argue “EA thinks X”, I want to add a couple of caveats regarding “our opinions”:
1. This note from Linch on another post:
2. I think it will be much more useful to focus on cold takes, when people are ready to share them over the coming weeks/months.
Polis is a better tool for understanding opinions than most other discussions we’ve had here.
Polis isn’t about size of opinions, but breadth and correlation. You don’t need respondents to be representative to get a sense of what different parts of the community thinks here.
I agree with the cold takes point, there is a comment like that in the poll.
Oh interesting, nice. Thanks for adding the ‘Longer intro’ section.
I added that while you were commenting, but have updated it since your comments. Great minds think alike etc etc.
This is a good point. I am interested to hear what you think of the idea of having an EA research participant panel that we can recruit from to understand consensus within community debates?
Thanks. I’m not entirely sure what you’re proposing but my guess is that we have better things to do with our time. Sorry for the unsophisticated answer!
On the object level, there now seem to be two roughly equal clusters. Cluster A are long-termist, consider EA epistemically humble,[1] don’t think we should blame ourselves for the FTX farrago and are unenthusiastic about democratisation. Cluster B take the opposite positions.
On the meta level:
It seems a bit concerning that the OP stresses the poll shouldn’t be used to say X% of EAs think Y, but the first item of the report is a list of statements where 60% of respondents either agreed or disagreed.[2] It seems that it would be easy to take these as consensus positions, when (a) the views of the 220 respondents may not be representative and (b) 60% isn’t much of a consensus. It will be interesting to see, for example, whether people assert that that this poll shows wide support for “EA leaders should be transparent about what they knew about SBF and when”.
Many of the “majority” statements seem anodyne. For example, it’s not particularly interesting to highlight (as the report does) that most people agree the top team at FTX made poor decisions.
The report doesn’t do a particularly good job at identifying “statements which make this group unique”. It includes statements which majorities in both groups agree. It highlights the statement, “I don’t believe in longtermism and I don’t feel like I can mention that to other EAs” rather than the more basic, “I think longtermism is true”. It doesn’t clearly identify the difference in views on democratisation: e.g. the disagreement on the statements “There should be a community-elected board in EA”, “‘EA should democratise’, without any detail on how this could happen, or would have prevented FTX, is just an applause light”[3] or “Democratic and transparent processes of decision making are better for both quality and oversight.”
The “areas of uncertainty” mostly identifies statements which in retrospect are unclear or inapplicable to most respondents, rather than any fundamental uncertainty. Possibly it would help to distuinguish “pass” from “unsure”, with a “pass” not being counted as an answer at all.
I can’t understand the graph at all. What are the axes? What are the rings? The graph doesn’t show two clusters. If anything, there’s one cluster in the centre, although that could conceivably be an artifact of the inner circle having a smaller area. How can participants be positioned “close” to statements they agreed, when agreement is 0/1?
OP says in a comment here that there are now too many statements, but the setup encourages respondents to submit statements. If anything, it seems surprising that 220 respondents only submitted 185 statements. Ideally, the poll would have many more respondents, so this seems to highlight a problem with the system.
The most relevant statement here reads, “I think EA is epistemically humble but external people don’t seem to think so.” It seems likely to me, based on other responses, that the disagreement stems from the first part, but of course a person who believed EA to be universally acknowledged as epistemically humble would also disagree. In retrospect, it would have been better to have “EA is epistemically humble” and “External people don’t consider EA to be epistemically humble” as separate statements.
For some reason “EA should not aim to be a community” is also appearing here although only 55% of respondents disagree.
Of course cluster B does have at least one concrete proposal, namely a community-elected board. It’s just that cluster A thinks that would be bad.
It looks to me like the clustering is basically ok, but the presentation breaks down when there are so many statements. Better to just look at the full list.
Would like to see more of this!
I’m not convinced that the opinions that will be aggregated by this poll will constitute a representative sample of the EA movement.
I am confident they won’t, nor is that what this is.
Fair enough :) I still think it’s worth amplifying this point.
Thanks for saying it.
There are many statements there not particularly related to FTX (some of which I added myself). Even if the sample is not representative, this grouping does feel like a real division that I also see in EA circles in real life.
[Edit: when this was written there were two groups, now there are three. Still, I maintain that there is some meaningful division here.]
I ended up answering “pass/unsure” to almost everything.
Lots of the time, it was because I thought the statement was ambiguous.
There was a space at the bottom where I could add a comment, but I wasn’t clear on whether the comment I would add would be attached to the statement I was voting on, or whether it would create a new standalone statement
Also expressing uncertainty is good. That’s a feature here.
It felt frustrating to me (and will likely complicate the findings) that “pass” had about five meanings (e.g., no opinion, too early to commit to an opinion, not applicable to me, too ambiguous, feelings too mixed for a response) and my pass response could be misinterpreted as having a meaning I didn’t intend. That’s just an inherent downside to the tool design, though.
It would create a new standalone statement.
This is the largest polis poll by response I’ve ever done. I think I might start deleting boring answers to make it easier to navigate:
Answers with consensus that seem unsurprising
Answers where everyone was uncertain
Answers which a similar answer is more approved
Please don’t delete answers. This is a fascinating snapshot of what people are thinking and I think there is merit for the poll to remain open and be reviewed in detail.
I wonder if there is any narrative coming out of the clusters/most agreed/disagreed/abstained questions/statements?
Well there are just too many answers now.
When you say “delete answers”, do you mean statements or votes?
When I sort all statements by ranking, %agree and %disagree seem to rank as expedted—by % descending, but %passed seems to jump all over the place. Is there a reason for this or is this a bug?
I don’t know. I don’t really have a gears level understanding of polis.
Thanks Nathan, this is an excellent initiative.
There are already 165 statements, more and more of them meta statements about the poll. Nathan, you created a monster.
(Please don’t take this as serious criticism)
I added some statements intended to identify, in a general manner, the respondent’s role and level of engagement. I think it would be helpful to know if certain clusters were associated with focus on meta level work vs. being in a role like AI research vs. those of us who work in non-EA roles.
(on phone) Thanks so much for doing this!I was going to post to request a survey today so really glad to this post. Polis seems cool and I have enjoyed using it.
I’ll take this as an opportunity to mention that I think we should normalise using consensus assessments (like something like polis or a more conventional survey) for nearly all community disagreements.
At the moment it’s usually impossible to determine what the EA community thinks on many issues. The lack of synthesis and consensus across complex debates makes it hard to update accurately.
I do also see some downside risks for example with the media so we should be careful when we do this, with the questions asked, and with how results are publicised.
Relatedly, I would like EA to have a participant panel of EA community members who can be hired for a representative sample.we would survey this group when we want to know what the community thinks and not just rely on assessing those who have the time and confidence to visit/comment/vote on the forum etc. I’d love to see that happen asap because it would be very useful at a time like now.
Thanks for running this, Nathan!
A few general comments for future runs:
People might benefit from more clarity around what makes a good statement to submit: e.g., respondents are expected to have different opinions/responses and those differences might be expected to cluster with existing statements in ways that would be helpful to understand.
Although I know Polis isnt intended as a crowdsourced poll per se, I wonder if there is value to running an “open” polis every quarter for that purpose. I got the sense that some community members saw a chance to anonymously get a sense of community sentiment on unrelated topics that were of concern to them (e.g., flirting at EAG). Obviously, there would have to be some norms and limits around this.
I think having a quick way for people to anonymously raise issues and get a measure of possible community sentiment is a good thing. Even understanding the sample bias problems, I think getting shallow feedback still often has enough value to justify five to seven minutes of the community’s collective time. The result could be helpful in informing someone that the topic is worth devoting several hours to a detailed forum post, or inclusion in a more formal / widespread poll.
Thanks for this interesting tool. I’d suggest that you suggest to submitters not to send compound statements in the section “What makes a good statement.” E.g., “It’s really bad that EAs don’t feel comfortable questioning longtermism. This points to deeper groupthink inside EA.” If it’s a premise-conclusion statement, you could also make clear that voters should pass unless they accept the premise.
I’m excited by the prospect of Polis, but it’s frustratingly limited. The system has no notion of whether people are agreeing with a statement because it’s convincing or bridging the gap, or because it’s banal.
In this case… I don’t think we’re really undergoing any factionalization about this? In that case, should we not just try talking more… that usually works pretty well with us.
Talking is a great idea in general, but it seems there are some opinions in this survey suggesting that there are barriers to talking openly?
I believe the forum allows commenting anonymously, though I wouldn’t know how to access that feature.
Psuedonyms would be a bit better, but it’ll do.
As far as I know, the supported way to comment anonymously is to make an anonymous account