Newer EAs are too junior to have good takes yet. It’s just that the growth rate has increased so there’s a higher proportion of them.
People who have better thoughts get hired at EA orgs and are too busy to post. There is anticorrelation between the amount of time people have to post on EA Forum and the quality of person.
Although we want more object-level discussion, everyone can weigh in on meta/community stuff, whereas they only know about their own cause areas. Therefore community content, especially shallow criticism, gets upvoted more. There could be a similar effect for posts by well-known EA figures.
Contests like the criticism contest decrease average quality, because the type of person who would enter a contest to win money on average has worse takes than the type of person who has genuine deep criticism. There were 232 posts for the criticism contest, and 158 for the Cause Exploration Prizes, which combined is more top-level posts than the entire forum in any month except August 2022.
EA Forum is turning into a place primarily optimized for people to feel welcome and talk about EA, rather than impact.
All of this is exacerbated as the most careful and rational thinkers flee somewhere else, expecting that they won’t get good quality engagement on EA Forum
So as I see it the main phenomenon is that there’s just much more being posted on the forum. I think there’s two factors behind that 1) community growth and 2) strong encouragement to post on the Forum. Eg there’s lots of encouragement to post on the forum from: the undergraduate introductory/onboarding fellowships, the AGI/etc ‘Fundamentals’ courses, the SERI/CERI/etc Summer Fellowships, or this or this (h/t John below).
The main phenomenon is that there is a lot more posted on the forum, mostly from newer/more junior people. It could well be the case that the average quality of posts has gone down. However, I’m not so sure that the quality of the best posts has gone down, and I’m not so sure that there are fewer of the best posts every month. Nevertheless, spotting the signal from the noise has become harder.
But then the forum serves several purposes. To take two of them: One (which is the one commenters here are most focussed on) is “signal”—producing really high-quality content—and its certainly got harder to find that. But another purpose is more instrumental—its for more junior people to demonstrate their writing/reasoning ability to potential employees. Or its to act as an incentive/endgoal for them to do some research—where the benefit is more that they see whether its a fit for them or not, but they wouldn’t actually do the work if it wasn’t structured towards writing something public.
So the main thing that those of us who are looking for “signal” need to do is find better/new ways to do so. The curated posts are a postive step in this direction, as are the weekly summaries and the monthly summaries.
We also seem to get a fair number of posts that make basically the same point as an earlier article, but the author presumably either didn’t read the earlier one or wanted to re-iterate it.
I think there are many people who have very high bars for how good something should be to post on the forum. Thus the forum becomes dominated by a few people (often people who aren’t aware of or care about forum norms) who have much lower bars to posting.
This is a plausible mechanism for explaining why content is of lower quality than one would otherwise expect, but it doesn’t explain differences in quality over time (and specifically quality decline), unless you add extra assumptions such that the proportion of people with low bars to posting has increased recently. (Cf. Ryan’s comment)
often people who aren’t aware of or care about forum norms
EA has grown a lot recently, so I think there are more people recently who aren’t aware of or care about the “high bar” norm. This is in part due to others explicitly saying the bar should be lower, which (as others here have noted) has a stronger effect on some than on others.
Edit: I don’t have time to do this right now, but I would be interested to plot the proportion of posts on the EA forum from people who have been on the forum for less than a year over time. I suspect that it would be trending upwards (but could be wrong). This would be a way to empirically verify part of my claim.
Eternal September is a slightly different hypothesis that those listed. It’s that if new people come into the community then there is an erosion of norms that make the community distinctive.
I’m interested in learning how plausible people find each of these mechanisms, so I created a short (anonymous) survey. I’ll release the results in a few days [ETA: see below]. Estimated completion time is ~90 seconds.
I think it’s fairly clear which of these are the main factors, and which are not. Explanations (3-5) and (7) do not account for the recent decline, because they have always been true. Also, (6) is a weak explanation, because the quality wasn’t substantially worse than an average post.
On the other hand, (1-2) +/- (8) fit perfectly with the fact that volume has increased over the last 18 months, over the same period as community-building has happened on a large scale. And I can’t think of any major contributors outside of (1-8), so I think the main causes are simply community dilution + a flood of newbies.
I wouldn’t be quick to dismiss (3-5) and (7) as factors we should pay attention to. These sorts of memetic pressures are present in many communities, and yet communities vary dramatically in quality. This is because things like (3-5) and (7) can be modulated by other facts about the community:
How intrinsically susceptible are people to clickbait?
Have they been taught things like politics is the mind-killer and the dangers of platforms where controversial ideas outcompete broadly good ones?
What is the variance in how busy people are?
To what degree do people feel like they can weigh in on meta? To what degree can they weigh in on cause areas that are not their own?
Are the people on EA Forum mostly trying for impact, or to feel like they’re part of a community (including instrumentally towards impact)?
So even if they cannot be solely reponsible for changes, they could have been necessary to produce any declines in quality we’ve observed, and be important for the future.
I agree that (4) could be modulated by the character of the community. The same is true for (3,5), except that, the direction is wrong. Old-timers are more likely to be professional EAs, and know more about the community, so their decreased prevalence should reduce problems from (3,5). And (7) seems more like an effect of the changing nature of the forum, rather than a cause of it.
Though the other factors could still partially explain why the level (as opposed to the trend) isn’t better, and arguably the level is what we’re ultimately interested in.
Re 3, ‘There is anticorrelation between the amount of time people have to post on EA Forum and the quality of person.’ - this makes me wince. A language point is that I think talking about how ‘good quality’ people are overall is unkind and leads to people feeling bad about themselves for not having such-and-such an attribute. An object level point is I don’t think there is an anticorrelation—I think being a busy EA org person does make it more likely that they’ll have valuable takes, but not being a busy-EA-org-person doesn’t make it less likely—there aren’t that many busy-EA-org-person jobs, and some people aren’t a good fit for busy jobs (eg because of their health or family commitments) but they still have interesting ideas.
Re 7: I’m literally working on a post with someone about how lots of people feel too intimidated to post on the Forum because of its perceived high standards! So I think though the Forum team are trying to make people feel welcome, it’s not true that it’s (yet) optimized for this, imo.
There’s a kind of general problem whereby any messaging or mechanism that’s designed to dissuade people from posting low-quality things will (a) just not work on some people—some people just have a lot of confidence in their not-very-good opinions, shrug, and (b) dissuade people who would post high-quality things, but who have impostor syndrome or are perfectionist or over self-critical. I think the number of people that the mechanism works as intended on—ie people who would have posted a low quality post but are now dissuaded from it—is probably pretty low. Since there are lots of people in EA with impostor syndrome/perfectionism/over-scrupulosity, I’m pretty in favour of the Forum having a ‘welcoming’ vibe over a We Are Very Serious and Important vibe.… because I’d rather have more good takes and more bad takes, than get rid of the bad takes and also get rid of good takes from impostors.
So napkin math suggests that the per-post cost of a contest post is something like 1% of the per-post cost of a RP publication. A typical RP publication is probably much higher quality. But maybe sometimes getting a lot of shallow explorations quickly is what’s desired. (Disclaimer: I haven’t been reading the forum much, didn’t read many contest posts, and don’t have an opinion about their quality. But I did notice the organizers of the ELK contest were “surprised by the number and quality of submissions”.)
A related point re: quality is that smaller prize pools presumably select for people with lower opportunity costs. If I’m a talented professional who commands a high hourly rate, I might do the expected value math on e.g. the criticism prize and decide it’s not worthwhile to enter.
It’s also not clear if the large number of entries will persist in the longer term. Not winning can be pretty demoralizing. Supposing a talented professional goes against their better judgement and puts a lot of time into their entry, then loses and has no idea why. Will they enter the next contest they see? Probably not. They’re liable to interpret lack of a prize as “the contest organizers didn’t think it was worth my time to make a submission”.
Hey just want to weigh in here that you can’t divide our FTE by our total publication count, since that doesn’t include a large amount of work we’ve produced that is not able to be made public or is not yet public but will be. Right now I think a majority of our output is not public right now for one reason or another, though we’re working on finding routes to make more of it public.
I do think your general point though that the per-post cost of a contest post is less / much less than an RP post is accurate though.
BTW, I hope it doesn’t seem like it was picking on you—it just occurred to me that I could do math for Rethink Priorities because your salaries are public. I have no reason to believe a cost-per-public-report estimate would be different for any other randomly chosen EA research organization in either direction. And of course most EA organizations correctly focus on making a positive impact rather than maximizing publication count.
I’d reframe this slightly, though I agree with all your key points. EA forum is finding a new comparative advantage. There are other platforms for deep, impact-focused research. Some of the best research has crystallized into founding efforts.
There will always be the need for an onboarding site and watering hole, and EA forum is filling that niche.
There are other platforms for deep, impact-focused research.
Could you name them? I’m not sure which ones are out there, other than LW and Alignment Forum for AI alignment research.
E.g. I’m not sure where else is a better place to post research on forecasting, research on EA community building, research on animal welfare, or new project proposals. There are private groups and slacks, but sometimes what you want is public or community engagement.
I was thinking about our biggest institutions, OpenPhil, 80k, that sort of thing—the work produced by their on-staff researchers. It sounds like you’re wanting a space that’s like the EA forum, but has a higher concentration of impact-focused research especially by independent researchers? Or maybe that you’d like to see the new work other orgs are doing get aggregated in one place?
A thing I wrote on social media a few months ago, in response to someone asking if an AI warning shot might happen:
[… I]t’s a realistic possibility, but I’d guess it won’t happen before AI destroys the world, and if it happens I’m guessing the reaction will be stupid/panicky enough to just make the situation worse.
(It’s also possible it happens before AI destroys the world, but six weeks before rather than six years before, when it’s too late to make any difference.)
A lot of EAs feel confident we’ll get a “warning shot” like this, and/or are mostly predicating their AI strategy around “warning-shot-ish things will happen and suddenly everyone will get serious and do much more sane things”. Which doesn’t sound like, eg, how the world reacted to COVID or 9/11, though it sounds a bit like how the world (eventually) reacted to nukes and maybe to the recent Ukraine invasion?
Someone then asked why I thought a warning shot might make things worse, and I said:
It might not buy time, or might buy orders of magnitude less time than matters; and/or some combination of:
- the places that are likely to have the strictest regulations are (maybe) the most safety-conscious parts of the world. So you may end up slowing down the safety-conscious researchers much more than the reckless ones.
- more generally, it’s surprising and neat that the frontrunner (DM) is currently one of the least allergic to thinking about AI risk. I don’t think it’s anywhere near sufficient, but if we reroll the dice we should by default expect a worse front-runner.
- regulations and/or safety research are misdirected, because people have bad models now and are therefore likely to have bad models when the warning shot happens, and warning shots don’t instantly fix bad underlying models.
The problem is complicated, and steering in the right direction requires that people spend time (often years) setting multiple parameters to the right values in a world-model. Warning shots might at best fix a single parameter, ‘level of fear’, not transmit the whole model. And even if people afterwards start thinking more seriously and thereby end up with better models down the road, their snap reaction to the warning shot may lock in sticky bad regulations, policies, norms, culture, etc., because they don’t already have the right models before the warning shot happens.
- people tend to make worse decisions (if it’s a complicated issue like this, not just ‘run from tiger’) when they’re panicking and scared and feeling super rushed. As AGI draws visibly close / more people get scared (if either of those things ever happen), I expect more person-hours spent on the problem, but I also expect more rationalization, rushed and motivated reasoning, friendships and alliances breaking under the emotional strain, uncreative and on-rails thinking, unstrategic flailing, race dynamics, etc.
- if regulations or public backlash do happen, these are likely to sour a lot of ML researchers on the whole idea of AI safety and/or sour them on xrisk/EA ideas/people. Politicians or the public suddenly caring or getting involved, can easily cause a counter-backlash that makes AI alignment progress even more slowly than it would have by default.
- software is not very regulatable, software we don’t understand well enough to define is even less regulatable, whiteboard ideas are less regulatable still, you can probably run an AGI on a not-expensive laptop eventually, etc.
So regulation is mostly relevant as a way to try to slow everything down indiscriminately, rather than as a way to specifically target AGI; and it would be hard to make it have a large effect on that front, even if this would have a net positive effect.
- a warning shot could convince everyone that AI is super powerful and important and we need to invest way more in it.
- (insert random change to the world I haven’t thought of, because events like these often have big random hard-to-predict effects)
Any given big random change will tend to be bad on average, because the end-state we want requires setting multiple parameters to pretty specific values and any randomizing effect will be more likely to break a parameter we already have in approximately the right place, than to coincidentally set a parameter to exactly the right value.
There are far more ways to set the world to the wrong state than the right one, so adding entropy will usually make things worse.
We may still need to make some high-variance choices like this, if we think we’re just so fucked that we need to reroll the dice and hope to have something good happen by coincidence. But this is very different from expecting the reaction to a warning shot to be a good one. (And even in a best-case scenario we’ll need to set almost all of the parameters via steering rather than via rerolling; rerolling can maybe get us one or even two values close-to-correct if we’re crazy lucky, but the other eight values will still need to be locked in by optimization, because relying on ten independent one-in-ten coincidences to happen is obviously silly.)
- oh, [redacted]‘s comments remind me of a special case of ‘worse actors replace the current ones’: AI is banned or nationalized and the UK or US government builds it instead. To my eye, this seems a lot likelier to go poorly than the status quo.
There are plenty of scenarios that I think make the world go a lot better, but I don’t think warning shots are one of them.
(One might help somewhat, if it happened; it’s mostly just hard to say, and we’ll need other major changes to happen first. Those other major changes are more the thing I’d suggest focusing on.)
How are you operationalizing “warning shot” and how low do you think the chances are that a warning shot operationalized that way happens and the world is not destroyed within e.g. the following three years?
I’m thinking of a ‘warning shot’ roughly as ‘an event where AI is widely perceived to have caused a very large amount of destruction’. Maybe loosely operationalized as ‘an event about as sudden as 9/11, and at least one-tenth as shocking, tragic, and deadly as 9/11’.
I don’t have stable or reliable probabilities here, and I expect that other MIRI people would give totally different numbers. But my current ass numbers are something like:
12% chance of a warning shot happening at some point.
6% chance of a warning shot happening more than 6 years before AGI destroys and/or saves the world.
10% chance of a warning shot happening 6 years or less before AGI destroys and/or saves the world.
My current unconditional odds on humanity surviving are very low (with most of my optimism coming from the fact that the future is just inherently hard to predict). Stating some other ass numbers:
Suppose that things go super well for alignment and timelines aren’t that short, such that we achieve AGI in 2050 and have a 10% chance of existential success. In that world, if we held as much as possible constant except that AGI comes in 2060 instead of 2050, then I’m guessing that would double our success odds to 20%.
If we invented AGI in 2050 and somehow impossibly had fifteen years to work with AGI systems before anyone would be able to destroy the world, and we knew as much, then I’d imagine our success odds maybe rising from 10% to 55%.
The default I expect instead is that the first AGI developer will have more than three months, and less than five years, before someone else destroys the world with AGI. (And on the mainline, I expect them to have ~zero chance of aligning their AGI, and I expect everyone else to have ~zero chance as well.)
If a warning shot had a large positive impact on our success probability, I’d expect it to look something like:
’20 or 30 or 40 years before we’d naturally reach AGI, a huge narrow-AI disaster unrelated to AGI risk occurs. This disaster is purely accidental (not terrorism or whatever). Its effect is mainly just to cause it to be in the Overton window that a wider variety of serious technical people can talk about scary AI outcomes at all, and maybe it slows timelines by five years or something. Also, somehow none of this causes discourse to become even dumber; e.g., people don’t start dismissing AGI risk because “the real risk is narrow AI systems like the one we just saw”, and there isn’t a big ML backlash to regulatory/safety efforts, and so on.′
I don’t expect anything at all like that to happen, not least because I suspect we may not have 20+ years left before AGI. But that’s a scenario where I could (maybe, optimistically) imagine real, modest improvements.
If I imagine that absent any warning shots, AGI is coming in 2050, and there’s a 1% chance of things going well in 2050, then:
If we add a warning shot in 2023, then I’d predict something like: 85% chance it has no major effect, 12% chance if makes the situation a lot worse, 3% chance it makes the situation a lot better. (I.e., an 80% chance that if it has a major effect, it makes things worse.)
This still strikes me as worth thinking about some, in part because these probabilities are super unreliable. But mostly I think EAs should set aside the idea of warning shots and think more about things we might be able to cause to happen, and things that have more effects like ‘shift the culture of ML specifically’ and/or ‘transmit lots of bits of information to technical people’, rather than ‘make the world as a whole panic more’.
I’m much more excited by scenarios like: ‘a new podcast comes out that has top-tier-excellent discussion of AI alignment stuff, it becomes super popular among ML researchers, and the culture, norms, and expectations of ML thereby shift such that water-cooler conversations about AGI catastrophe are more serious, substantive, informed, candid, and frequent’.
It’s rare for a big positive cultural shift like that to happen; but it does happen sometimes, and it can result in very fast changes to the Overton window. And since it’s a podcast containing many hours of content, there’s the potential to seed subsequent conversations with a lot of high-quality background thoughts.
To my eye, that seems more like the kind of change that might shift us from a current trajectory of “~definitely going to kill ourselves” to a new trajectory of “viable chance of an existential win”.
Whereas warning shots feel more unpredictable to me, and if they’re unhelpful, I expect the helpfulness to at best look like “we were almost on track to win, and then the warning shot nudged us just enough to secure a win”.
Well, it’s less effort insofar as these are very low-confidence, unstable ass numbers. I wouldn’t want to depend on a plan that assumes there will be no warning shots, or a plan that assumes there will be some.
So, could we be doing more to attract, train and upskill fundraising professionals? Is it something 80,000 Hours recommends in coaching calls? Should an EA-aligned fund establish or invest in a bootcamp-style course or program that emphasizes membership and smaller-gift fundraising because of it shares so many traits with community building?
GovAI now has a full-time researcher working on compute governance. Chinchilla’s Wild Implications suggests that access to data might also be a crucial leverage point for AI development. However, from what I can tell, there are no EAs working full time on how data protection regulations might help slow or direct AI progress. This seems like a pretty big gap in the field.
What’s going on here? I can see two possible answers:
Folks have suggested that compute is relatively to govern (eg). Someone might have looked into this and decided data is just too hard to control, and we’re better off putting our time into compute.
Someone might already be working on this that I just haven’t heard of.
How to upskill in AI Safety after AGISF, and how to help others upskill
For people who want to start working on AI Safety, it feels like the AGISF program is widely accepted as the start of the pipeline. It also feels like there are some things we want to see work on within AI Safety: conceptual work, engineering work, empirical work, etc.
However, I think there’s an upskilling gap that isn’t currently clearly signposted by the community: how to go from AGISF to being able to contribute! The main programmes I’m aware of for taking someone who knows a bit about alignment to being able to contribute to AI Safety are SERI MATS (trying to produce conceptual researchers), MLAB (trying to produce engineers), and internships at orgs like Conjecture, CHAI etc.
Without having done these, my impression is that they seem like the best options for upskilling past AGISF, but these are bottlenecked pretty hard by mentor time. So, I think there are plenty of people who could be doing useful work with sufficient upskilling, but who will have to do upskilling more independently. For these people, useful resources like How to pursue a career in technical AI Alignment by Charlie Rogers-Smith exist, but I think some clear signposting for people post-AGISF, pre-job would be useful.
So, after AGISF, what should an excited safety person be doing to upskill? Here are a few ideas
I think for almost all of these resources, the order of how good I would expect them to be is:
In person with mentorship > Online with mentorship > In person with no mentorship > Online with no mentorship >> solo. A caveat is that I’m not super sure about how high to rate mentorship compared with simply working through these resources with other enthusiastic people. It seems less important for some areas of engineering work than it does for conceptual work (like getting research taste), but I’m not sure about this.
Implications for AI Safety field-builders.
I think the gap between someone doing AGISF and doing impactful work has two implications for field builders in AI safety:
Firstly, I think there would be a lot of value in creating online versions of programmes which essentially go through some of these resources, similarly to how AGISF currently works. I think the biggest bottleneck for these would be mentor time, but I think lots of these could be used successfully without mentors, if there are others who are excited to work through the resources who also have some background in the area.
Secondly, I think that field builders working in local groups with enough members and resources (Oxford, Berkeley, London) could try to run versions of these programmes in person. If online versions of these programmes exist, this becomes even easier: essentially all it requires is scheduling when these groups need to meet!
Conclusion
If you have done AGISF and want to start upskilling, hopefully some of these resources will be useful!
If you want to help others upskill, I think running programmes centred around some of these resources would be a good idea!
What do we do as newcomers? How do we make sense of all of the information out there? How do I get involved if I’m not an academic, and I don’t have “career experience” to fall back on? I’ve just published a blog post about some of my impressions as a newcomer to EA. (Though I don’t know if I answer any of these questions very well, because I don’t know if I have the answers yet.)
Hi Danielle, welcome! I really enjoyed reading your blog post – it seems like you have a skill for communicating clearly, and I’d love to read more of your writing as you keep thinking more about EA :)
One thing you can do as you learn more is to try and crystallize what cruxes you have for your beliefs. Talking about cruxes is a common way EAs communicate, as it can help people avoid talking past each other and instead focus primarily on the things that would change each other’s minds.
My take is that it’s much more valuable to spend time early on thinking about your values and what you think the most important problems are, compared to finding an EA-relevant job quickly (because there are compelling cases for a lot of cause areas, and you might find that you change your mind about what path you want to pursue).
In terms of getting connected, you could try going to a meetup in your city (if there is one). Some people enjoy their city EA groups, others don’t as much. EAGs and EAGx’s can also be good places to meet people who share your interests. Yes, many groups use Facebook to coordinate, but I think there is less significant community discussion on FB than there used to be.
In the recent interview with Katja Grace referenced on ACX, she mentioned that many people may be opposed to slowing down AI progress because (I’m paraphrasing) they perceive AGI as a genie that will solve longevity and other problems and bring about the Cool Transhumanist Future, which they won’t see otherwise—due to their age and/or longer timelines. I have been hanging out in longevity/anti-aging spaces for a while and this perspective is exceedingly common there. People are very hyped about AI coming and curing aging, and dismiss any concerns about AI safety.
From this point of view, solving aging, or even just making a tangible progress towards LEV , will make those people less resistant to the notion of slowing/stopping AI progress. This is complementary to the often mentioned idea that longer life expectancy will cause people to care more about AGI (and other existential risks). This of course doesn’t imply that anti-aging is more important than direct work on AI alignment, but 1) it is likely more tractable and 2) not everyone can or want to work on AI alignment directly.
This talk by Jade Leung got me thinking—I’ve never seen a plan for what we do if AGI turns out misaligned.
The default assumption seems to be something like “well, there’s no point planning for that, because we’ll all be powerless and screwed”. This seems mistaken to me. It’s not clear that we’ll be so powerless that we have absolutely no ability to encourage a trajectory change, particularly in a slow takeoff scenario. Given that most people weight alleviating suffering higher than promoting pleasure, this is especially valuable work in expectation as it might help us change outcomes from ‘very, very bad world’ to ‘slightly negative’ world. This also seems pretty tractable—I’d expect ~10hrs thinking about this could help us come up with a very barebones playbook.
Why isn’t this being done? I think there are a few reasons:
Like suffering focused ethics, it’s depressing.
It seems particularly speculative—most of the ‘humanity becomes disempowered by AGI’ scenarios look pretty sci-fi. So serious academics don’t want to consider it.
People assume, mistakenly IMO, that we’re just totally screwed if AI is misaligned.
Can someone guide me towards studies, meta-reviews or collections of studies which quantifies the effectiveness of various interventions? I.e. “$10 000 spent on deworming medication in Sub Saharan countries saves on average 1 life” or “access to clean water wells in Sub Saharan countries increased educational performance by x points”. Thanks!
It seems like Mongolia is underrated as a case study in international development. Despite having some major factors pushing against it (landlocked, stuck between great powers, sparsely populated, recently colonized, recently communist), it does reasonably well on economic and political indicators.
Doing longtermist AI policy work feels a little like aiming heavy artillery with a blindfold on. We can’t see our target, we’ve no idea how hard to push the barrel in any one direction, we don’t know how long the fuse is, we can’t stop the cannonball once it’s in motion, and we could do some serious damage if we get things wrong.
Longtermist legal work seems particularly susceptible to the Cannonball Problem, for a few reasons:
Changes to hard law are difficult to reverse—legislatures rarely consider issues more than a couple times every ten years, and the judiciary takes even longer.
At the same time, legal measures which once looked good can quickly become ineffectual due to shifts in underlying political, social or economic circumstances.
Taken together, this means that bad laws have a long time to do a lot of harm, so we need to be careful when putting new rules on the books.
This is worsened by the fact that we don’t know what ideal longtermist governance looks like. In a world of transformative AI, it’s hard to tell if the rule of law will mean very much at all. If sovereign states aren’t powerful enough to act as leviathans, it’s hard to see why influential actors wouldn’t just revert to power politics.
Underlying all of this are huge, unanswered questions in political philosophy about where we want to end up. A lack of knowledge about our final destination makes it harder to come up with ways to get there.
I think this goes some way to explaining why longtermist lawyers only have a few concrete policy asks right now despite admirable efforts from LPP, GovAI and others.
Writing this from the “Writing on the EA Forum” workshop at EAGxBerlin, wanted to post some… post ideas:
(a) A quick report on how the conference went, and the particular vibes I encountered, expanding on my “like an academic conference with the annoying prestige layers removed” idea.
(b) Content creation as lowering barriers to entry for a given field, especially language barriers. It seems like an obvious point but on I’ve never seen written anywhere?
(c) Something about the impact of sharing useful trivia and life advice. I have a recent example about hearing damage/loss prevention, what’s the marginal value of these conversations?
(Thanks to Lizka for the nudge and excellent presentation)
I think there are currently too few infosec people and people trying to become billionaires.
Infosec: this seems really helpful for AI safety and biosecurity in a lot of worlds, and I’m guessing it’s just much less sexy / popular than technical research. Maybe I’m wrong about the number of people here, but from attendance at an EAGxSF event it didn’t seem like we would be saturated.
Entrepreneurship: I think the basic argument for making tens of billions of dollars still holds. Just because many longtermist orgs are well-funded now doesn’t mean they will be in the future (crypto and other risk), and there might be ways to spend hundreds of billions of dollars on other things. My understanding is that even at the crypto peak there were <50 EAs trying to become billionaires, and there are even fewer now, which seems like a mistake.
I’ve thought about both of these paths myself, and I think they’re not quite as good as technical alignment research for me, but I can’t rule out that I’m just being a coward.
(This is probably a bad idea that I haven’t thought through)
Suppose a startup raises $500k seed funding from VCs at $5M valuation, and the startup also commits $2M worth of illiquid stock to EA causes. Then EA could also provide in return another (say) $500k cash that is ear-marked for the founders and early employees’ personal bank accounts, and not towards startup growth. This ensures the founders are not exposed to too much financial anxiety if they fail.
I’m predicting a 10-25% probability that Russia will use a weapon of mass destruction (likely nuclear) before 2024. This is based on only a few hours of thinking about it with little background knowledge.
Russian pro-war propagandists are hinting at use of nuclear weapons, according to the latest BBC podcast Ukrainecast episode. [Ukrainecast] What will Putin do next? #ukrainecast
https://podcastaddict.com/episode/145068892 via @PodcastAddict
There’s a general sense that, in light of recent losses, something needs to change. My limited understanding sees 4 options:
Continue on the current course despite mounting criticism. Try to make the Ukrainians lives difficult by targeting their infrastructure, limit losses until winter, and try to reorganize during winter. This seems a pretty good option for now, even though I doubt Russia can really shore up its deeply set weaknesses. They can probably prepare to dig in, threaten and punish soldiers for fleeing. This wouldn’t go well for either party long-term, but Russia might bet on outlasting/undermining Western support. Probability: 40%?
Full-scale mobilisation of the population and the economy. This is risky for Putin: there’s supposedly a large anti-war sentiment in Russian culture, a legacy of the enormous losses during the 2nd World War. People don’t like to join a poorly-equipped poorly managed and losing army, even if it were a good cause.. This may be chosen, Putin may be misinformed and badly reading the public’s sentiment. I have no idea how this would develop internally. I doubt it will make a big difference in the course of the war, except by prolonging the war a bit. Maybe 25%? Maybe 50% if Putin underestimates public resistance.
Escalation by other means: I don’t know how many options Russia has. Chemical weapons, electro magnetic pulse, a single tactical nuclear strike on the battlefield for deterrence, multiple nuclear strikes for strategic reasons, population strike for deterrence. In the mind of Putin, I can see this as preferable: it leads to a potential military advantage, has limited risk for destabilising his internal power base. I don’t know how the international community would respond to this, nor how Putin thinks the international community would respond. In my (uninformed) view, only China can make a real difference here as the West already has stringent sanctions. I don’t know how China would respond to this. They wouldn’t like it, but I think the West won’t really punish China for its support in the short term. I’d say on this inside view, 10-25% seems reasonable. I’m setting the point estimate at 15%.
A distinction I’ve found useful is “object-level” vs “social reality”. They are both adjectives that describe types of conversation/ ideas.
Object-level discussions are about ideas and actions (e.g. AI timelines, the mechanics of launching a successful startup). Object-level ideas are technical, empirical, and often testable. Object-level refers to what ideas are important or make sense. It is focused on truth-seeking and presenting arguments clearly.
Social reality discussions are about people and organisations (e.g. Will MacAskill, Open Philanthropy). Social reality is more meta, more abstract, and less testable than object-level. Social reality refers to which people are influential/powerful (and what they think), how to network with people, how to persuade people.
Object-level: What’s the probability of extinction this century?
Social reality: What does Toby Ord think is the probability of extinction this century?
I have found it very helpful to start labelling whether I’m in object-level conversation mode vs social reality conversation mode. It helps me notice when I’m deferring without having thought about it (e.g. “well, Will MacAskill says [x]” instead of asking myself what I think about [x]), or when I fall into a mode of chit-chatting about the who’s-who of EA, instead of trying to truth-seek (of course, chit chatting sometimes is fine—I just want to be intentional about when I’m doing it).
And social reality isn’t necessarily bad, but it’s helpful to flag when a conversation enters “social reality mode.”
I do think it’s good for many/more/most conversations to centre around the object-level. I am personally trying to move my ratio more towards object-level.
(This was a core theme of an Atlas camp I attended, which I found extremely valuable. The above definitions are loosely based on a message from Jonas, but I didn’t run them by him before posting.)
where defensible view contains details you personally can defend, and delegated view includes other people’s views and whatever factor of delegation you feel is appropriate
I’m concerned about too much EA meta conversation — about worlds where most of EA dialogue is talking about EAs talking about EAs (lots of social reality and not enough object-level).
These sorts of convos are often very far removed from {concrete things that help the world}, and I worry about them taking away attention from more important ones.
I think it’s probably much better (for the world) for conversations to stay focused on the real world, object-level claims and arguments.
Part of me wants to flesh this thought out properly soon. But even this conversation is meta! And I’m trying to encourage/ focus more on object-level ideas. So do I write it? I’m not sure.
There’s a paradox I’m confused about, where if someone from a group I’m not in—let’s say Christians, came to me on a college campus and smiled at me and asked about my interests and connected all of them to Jesus and then I found out I’d been logged in a spreadsheet as “potential convert” or something and then found the questions they’d asked me in a blog post or “Christian evangelist top questions” I might very well feel extremely weird about that (though I think less than others, I kind of respect the hustle).
BUT, when I think about how one gets there, I think, ok:
You’re a christian, you care about saving other people from hell
You want to talk to people about this and get a community together + persuade people via arguments you think are in fact persuasive
Other people want to do the same, you discuss approaches
Other people have framings and types of questions that seem better to you than yours, so you switch
You’re talking to a lot of people and it’s hard to keep track of what each of them said and what they wanted out of a community or worldview, so you start writing it down
You don’t want people to get approached for the same conversations over and over again, so you share what you’ve written with your fellow Christian evangelists
It doesn’t seem useful to anyone to keep talking to people who don’t seem interested in Christianity, so you let your fellow evangelists know which folks are in that category
People who seem excited about Christianity would probably get a lot out of going to conferences or reading more about it, so you recommend conferences and books and try to make it as easy as possible for them to access those, without having annoying atheists who just want to cause trouble showing up.
This is probably too charitable, there is definitely a thing where you actively want to persuade people because you think your thing is important, and you might lose interest in people who aren’t excited about what you’re excited about, but those things also seem reasonable to me.
A process that seems bad:
Want to maximize number of EAs
Use framings, arguments and examples that you don’t think hold water but work at getting people to join your group [I don’t think EAs do this, I’m gesturing at the extreme other end]
Make people feel weird and bad for disagreeing with you, whether on purpose or not
Encourage people to repress their disagreements
Get energy and labor from people that they won’t endorse having given in a few years, or if they knew things you knew
3-5 seem like the worst parts here. 1 seems like a reasonable implication of their beliefs, though I do think we all have to cooperate to not destroy the commons.
2 is complicated—when people have different cruxes than you is it dishonest to talk about what should convince them based on their cruxes?
3 and 4 are bad, also hard to avoid.
5 seems really bad, and something I’d like to strongly avoid via things like transparency and some other percolating advice I might end up endorsing for people new to EA, like not letting your feet go faster than your brain, figuring out how much deference you endorse, seeing avoiding resentment as a crucial consideration in your life choices, staying grounded, etc.
I also think the processes can feel pretty similar from the inside (therefore danger alert!) but also look similar from the outside when they aren’t. I certainly have systematically underestimated the moral seriousness and earnestness of many an EA.
What’s the difference?
I think people are going to want to say something like “treating people as ends” but I don’t know where that obligation stops. I think I want to say something like “are you acting in the interests of the people you’re talking to”, but that doesn’t work either—I’m not! being an EA has a decent chance of being less pleasant than the other thing they were doing, and either way it’s not a crux. Ex: I endorse protecting the time and energy of other people by not telling everyone who I would talk to if I had a certain question or needed help in a certain way.
I do think it’s more about whether you’re doing things in such a way that if they knew why you were doing them, they’d mostly not be bothered (ie passing the red face test). But that doesn’t really solve the problem that digital sentience is a weird reason to do a lot of things, and there are lots of things I endorse it being inappropriate to be too explicit about.
[This is separate from the instrumental reasons to act differently because it weirds people out etc.]
....................................................................................................................... Later musings:
Presumably the strongest argument is that these feelings are tracking a bunch of the bad stuff that’s hard to point at:
people not actually understanding the arguments they’re making
people not having your best interests in mind
people being overconfident their thing is correct
people not being able to address your ideas / cruxes
I do think it’s more about whether you’re doing things in such a way that if they knew why you were doing them, they’d mostly not be bothered (ie passing the red face test). But that doesn’t really solve the problem that digital sentience is a weird reason to do a lot of things, and there are lots of things I endorse it being inappropriate to be too explicit about.
Of course this is a spectrum, and we shouldn’t put up a public website listing all our beliefs including the most controversial ones or something like that (no one in EA is very close to this extreme). But the implicit jump from “some things shouldn’t be explicit” to “digital sentience might weird some people out so there’s a decent chance we shouldn’t be that explicit about it” seems very non-obvious to me, given how central it is to a lot of longtermist’s worldviews and honestly I think it wouldn’t turn off many of the most promising people (in the long run; in the short run, it might get an initial “huh??” reaction).
I think steps 1 and 2 in your chain are also questionable, not just 3-5.
Want to maximize number of EAs
Why do we want to maximize number of EAs, this seems very non-obvious to me? Some people would add much more to the community than others via epistemics, culture, direct talent, etc. If we added enough of certain types of people to the community, especially too quickly, it could easily be net negative.
2. Use framings, arguments and examples that you don’t think hold water but work at getting people to join your group [I don’t think EAs do this, I’m gesturing at the extreme other end]
[...]
2 is complicated—when people have different cruxes than you is it dishonest to talk about what should convince them based on their cruxes?
I think sometimes/often talking about people’s cruxes rather than your own is good and fine. The issue is Goodharting via an optimal message to convert as many people to EA as quickly as possible, rather than messages that will lead to a healthy community over the long run.
Thanks so much for writing this! I think it could be a top-level post, I’m sure many others would find it very helpful.
My 2 cents:
2 is complicated—when people have different cruxes than you is it dishonest to talk about what should convince them based on their cruxes?
I think it’s definitely bad to “Use framings, arguments and examples that you don’t think hold water but work at getting people to join your group”. If I understand correctly it can cause point 5. Also “getting people to join your group” is rarely an instrumental goal, and “getting people to join your group for the wrong reasons” is probably not that useful in the long term.
Something that I think is very important that seems missing from this is that there’s a significant probability that we’re wrong about important things (i.e. EA as a question). We could be wrong about the impact of bednets, wrong about AI being the most important thing, wrong about population ethics, etc. I think it’s a huge difference from the “cult” mindset.
I think I want to say something like “are you acting in the interests of the people you’re talking to”, but that doesn’t work either—I’m not! being an EA has a decent chance of being less pleasant than the other thing they were doing, and either way it’s not a crux.
The way I think about this, on first approximation, is that I want people to work on maximising their values (and not their wellbeing). If they think altruism is not important and are solipsistic egoists and only value their own wellbeing, I don’t think EA can help them. If they value the wellbeing of others then EA can help them achieve their values better. From my personal perspective this is strongly related to the point on uncertainty: I don’t want to push other people to work on my values because from an outside view I don’t think my values are more important than their values, or more likely to be “correct”. I don’t know if it makes any sense, really curious to hear your thoughts, you have certainly thought about this more than I.
I think it’s definitely bad to “Use framings, arguments and examples that you don’t think hold water but work at getting people to join your group”. If I understand correctly it can cause point 5. Also “getting people to join your group” is rarely an instrumental goal, and “getting people to join your group for the wrong reasons” is probably not that useful in the long term.
Agree about the “not holding water”, I was trying to say that “addresses cruxes you don’t have” might look similar to this bad thing, but I’m not totally sure that’s true.
I disagree about getting people to join your group—that definitely seems like an instrumental goal, though definitely “get the relevant people to join your group” is more the thing—but different people might have different views on how relevant they need to be, or what their goal with the group is.
Something that I think is very important that seems missing from this is that there’s a significant probability that we’re wrong about important things (i.e. EA as a question).
I kind of agree here; I think there are things in EA I’m not particularly uncertain of, and while I’m open to being shown I’m wrong, I don’t want to pretend more uncertainty than I have.
The way I think about this, on first approximation, is that I want people to work on maximising their values (and not their wellbeing). If they think altruism is not important and are solipsistic egoists and only value their own wellbeing, I don’t think EA can help them. If value the wellbeing of others then EA can help them achieve their values better.
I’ve definitely heard that frame, but it honestly doesn’t resonate for me. I think some people are wrong about what values are right and arguing with me sometimes convinces them of that. I’ve definitely had my values changed by argumentation! Or at least values on some level of abstraction—not on the level of solipsism vs altruism, but there are many layers between that and “just an empirical question”.
I don’t want to push other people to work on my values because from an outside view I don’t think my values are more important than their values, or more likely to be “correct”
I incorporate an inside view on my values—if I didn’t think they were right, I’d do something else with my time!
I noticed this a while ago. I don’t see large numbers of low-quality low-karma posts as a big problem though (except that it has some reputation cost for people finding the Forum for the first time). What really worries me is the fraction of high-karma posts that neither original, rigorous, or useful. I suggested some server-side fixes for this.
PS: #3 has always been true, unless you’re claiming that more of their output is private these days.
Democratizing risk post update Earlier this week, a post was published criticizing democratizing risk. This post was deleted by the (anonymous) author. The forum moderation team did not ask them to delete it, nor are we aware of their reasons for doing so. We are investigating some likely Forum policy violations, however, and will clarify the situation as soon as possible.
EA forum content might be declining in quality. Here are some possible mechanisms:
Newer EAs have worse takes on average, because the current processes of recruitment and outreach produce a worse distribution than the old ones
Newer EAs are too junior to have good takes yet. It’s just that the growth rate has increased so there’s a higher proportion of them.
People who have better thoughts get hired at EA orgs and are too busy to post. There is anticorrelation between the amount of time people have to post on EA Forum and the quality of person.
Controversial content, rather than good content, gets the most engagement.
Although we want more object-level discussion, everyone can weigh in on meta/community stuff, whereas they only know about their own cause areas. Therefore community content, especially shallow criticism, gets upvoted more. There could be a similar effect for posts by well-known EA figures.
Contests like the criticism contest decrease average quality, because the type of person who would enter a contest to win money on average has worse takes than the type of person who has genuine deep criticism. There were 232 posts for the criticism contest, and 158 for the Cause Exploration Prizes, which combined is more top-level posts than the entire forum in any month except August 2022.
EA Forum is turning into a place primarily optimized for people to feel welcome and talk about EA, rather than impact.
All of this is exacerbated as the most careful and rational thinkers flee somewhere else, expecting that they won’t get good quality engagement on EA Forum
So as I see it the main phenomenon is that there’s just much more being posted on the forum. I think there’s two factors behind that 1) community growth and 2) strong encouragement to post on the Forum. Eg there’s lots of encouragement to post on the forum from: the undergraduate introductory/onboarding fellowships, the AGI/etc ‘Fundamentals’ courses, the SERI/CERI/etc Summer Fellowships, or this or this (h/t John below).
The main phenomenon is that there is a lot more posted on the forum, mostly from newer/more junior people. It could well be the case that the average quality of posts has gone down. However, I’m not so sure that the quality of the best posts has gone down, and I’m not so sure that there are fewer of the best posts every month. Nevertheless, spotting the signal from the noise has become harder.
But then the forum serves several purposes. To take two of them: One (which is the one commenters here are most focussed on) is “signal”—producing really high-quality content—and its certainly got harder to find that. But another purpose is more instrumental—its for more junior people to demonstrate their writing/reasoning ability to potential employees. Or its to act as an incentive/endgoal for them to do some research—where the benefit is more that they see whether its a fit for them or not, but they wouldn’t actually do the work if it wasn’t structured towards writing something public.
So the main thing that those of us who are looking for “signal” need to do is find better/new ways to do so. The curated posts are a postive step in this direction, as are the weekly summaries and the monthly summaries.
Are there examples of typical bad takes you’ve seen newer EAs post?
We also seem to get a fair number of posts that make basically the same point as an earlier article, but the author presumably either didn’t read the earlier one or wanted to re-iterate it.
I’ll add another mechanism:
I think there are many people who have very high bars for how good something should be to post on the forum. Thus the forum becomes dominated by a few people (often people who aren’t aware of or care about forum norms) who have much lower bars to posting.
This is a plausible mechanism for explaining why content is of lower quality than one would otherwise expect, but it doesn’t explain differences in quality over time (and specifically quality decline), unless you add extra assumptions such that the proportion of people with low bars to posting has increased recently. (Cf. Ryan’s comment)
You’re quite right, it was left too implicit.
EA has grown a lot recently, so I think there are more people recently who aren’t aware of or care about the “high bar” norm. This is in part due to others explicitly saying the bar should be lower, which (as others here have noted) has a stronger effect on some than on others.
Edit: I don’t have time to do this right now, but I would be interested to plot the proportion of posts on the EA forum from people who have been on the forum for less than a year over time. I suspect that it would be trending upwards (but could be wrong). This would be a way to empirically verify part of my claim.
Eternal September is a slightly different hypothesis that those listed. It’s that if new people come into the community then there is an erosion of norms that make the community distinctive.
I’m interested in learning how plausible people find each of these mechanisms, so I created a short (anonymous) survey. I’ll release the results in a few days [ETA: see below]. Estimated completion time is ~90 seconds.
The results are below. The data is here.
I think it’s fairly clear which of these are the main factors, and which are not. Explanations (3-5) and (7) do not account for the recent decline, because they have always been true. Also, (6) is a weak explanation, because the quality wasn’t substantially worse than an average post.
On the other hand, (1-2) +/- (8) fit perfectly with the fact that volume has increased over the last 18 months, over the same period as community-building has happened on a large scale. And I can’t think of any major contributors outside of (1-8), so I think the main causes are simply community dilution + a flood of newbies.
I wouldn’t be quick to dismiss (3-5) and (7) as factors we should pay attention to. These sorts of memetic pressures are present in many communities, and yet communities vary dramatically in quality. This is because things like (3-5) and (7) can be modulated by other facts about the community:
How intrinsically susceptible are people to clickbait?
Have they been taught things like politics is the mind-killer and the dangers of platforms where controversial ideas outcompete broadly good ones?
What is the variance in how busy people are?
To what degree do people feel like they can weigh in on meta? To what degree can they weigh in on cause areas that are not their own?
Are the people on EA Forum mostly trying for impact, or to feel like they’re part of a community (including instrumentally towards impact)?
So even if they cannot be solely reponsible for changes, they could have been necessary to produce any declines in quality we’ve observed, and be important for the future.
I agree that (4) could be modulated by the character of the community. The same is true for (3,5), except that, the direction is wrong. Old-timers are more likely to be professional EAs, and know more about the community, so their decreased prevalence should reduce problems from (3,5). And (7) seems more like an effect of the changing nature of the forum, rather than a cause of it.
Though the other factors could still partially explain why the level (as opposed to the trend) isn’t better, and arguably the level is what we’re ultimately interested in.
My comment got detached, woops
I broadly agree with 5 and 6.
Re 3, ‘There is anticorrelation between the amount of time people have to post on EA Forum and the quality of person.’ - this makes me wince. A language point is that I think talking about how ‘good quality’ people are overall is unkind and leads to people feeling bad about themselves for not having such-and-such an attribute. An object level point is I don’t think there is an anticorrelation—I think being a busy EA org person does make it more likely that they’ll have valuable takes, but not being a busy-EA-org-person doesn’t make it less likely—there aren’t that many busy-EA-org-person jobs, and some people aren’t a good fit for busy jobs (eg because of their health or family commitments) but they still have interesting ideas.
Re 7: I’m literally working on a post with someone about how lots of people feel too intimidated to post on the Forum because of its perceived high standards! So I think though the Forum team are trying to make people feel welcome, it’s not true that it’s (yet) optimized for this, imo.
There’s a kind of general problem whereby any messaging or mechanism that’s designed to dissuade people from posting low-quality things will (a) just not work on some people—some people just have a lot of confidence in their not-very-good opinions, shrug, and (b) dissuade people who would post high-quality things, but who have impostor syndrome or are perfectionist or over self-critical. I think the number of people that the mechanism works as intended on—ie people who would have posted a low quality post but are now dissuaded from it—is probably pretty low. Since there are lots of people in EA with impostor syndrome/perfectionism/over-scrupulosity, I’m pretty in favour of the Forum having a ‘welcoming’ vibe over a We Are Very Serious and Important vibe.… because I’d rather have more good takes and more bad takes, than get rid of the bad takes and also get rid of good takes from impostors.
Another possible mechanism is forum leadership encouraging people to be less intimidated and write more off-the-cuff posts—see e.g. this or this.
Side note: It seems like a small amount of prize money goes a long way.
E.g. Rethink Priorities makes their salaries public: they pay senior researchers $105,000 – $115,000 per year.
Their headcount near the end of 2021 was 24.75 full-time equivalents.
And their publications page lists 30 publications in 2021.
So napkin math suggests that the per-post cost of a contest post is something like 1% of the per-post cost of a RP publication. A typical RP publication is probably much higher quality. But maybe sometimes getting a lot of shallow explorations quickly is what’s desired. (Disclaimer: I haven’t been reading the forum much, didn’t read many contest posts, and don’t have an opinion about their quality. But I did notice the organizers of the ELK contest were “surprised by the number and quality of submissions”.)
A related point re: quality is that smaller prize pools presumably select for people with lower opportunity costs. If I’m a talented professional who commands a high hourly rate, I might do the expected value math on e.g. the criticism prize and decide it’s not worthwhile to enter.
It’s also not clear if the large number of entries will persist in the longer term. Not winning can be pretty demoralizing. Supposing a talented professional goes against their better judgement and puts a lot of time into their entry, then loses and has no idea why. Will they enter the next contest they see? Probably not. They’re liable to interpret lack of a prize as “the contest organizers didn’t think it was worth my time to make a submission”.
Hey just want to weigh in here that you can’t divide our FTE by our total publication count, since that doesn’t include a large amount of work we’ve produced that is not able to be made public or is not yet public but will be. Right now I think a majority of our output is not public right now for one reason or another, though we’re working on finding routes to make more of it public.
I do think your general point though that the per-post cost of a contest post is less / much less than an RP post is accurate though.
-Peter (Co-CEO of Rethink Priorities)
Thanks for the correction!
BTW, I hope it doesn’t seem like it was picking on you—it just occurred to me that I could do math for Rethink Priorities because your salaries are public. I have no reason to believe a cost-per-public-report estimate would be different for any other randomly chosen EA research organization in either direction. And of course most EA organizations correctly focus on making a positive impact rather than maximizing publication count.
Small formatting thought: making these numbered instead of bulleted will make it easier to have conversations about them
Done
I’d reframe this slightly, though I agree with all your key points. EA forum is finding a new comparative advantage. There are other platforms for deep, impact-focused research. Some of the best research has crystallized into founding efforts.
There will always be the need for an onboarding site and watering hole, and EA forum is filling that niche.
Could you name them? I’m not sure which ones are out there, other than LW and Alignment Forum for AI alignment research.
E.g. I’m not sure where else is a better place to post research on forecasting, research on EA community building, research on animal welfare, or new project proposals. There are private groups and slacks, but sometimes what you want is public or community engagement.
I was thinking about our biggest institutions, OpenPhil, 80k, that sort of thing—the work produced by their on-staff researchers. It sounds like you’re wanting a space that’s like the EA forum, but has a higher concentration of impact-focused research especially by independent researchers? Or maybe that you’d like to see the new work other orgs are doing get aggregated in one place?
A thing I wrote on social media a few months ago, in response to someone asking if an AI warning shot might happen:
Someone then asked why I thought a warning shot might make things worse, and I said:
There are plenty of scenarios that I think make the world go a lot better, but I don’t think warning shots are one of them.
(One might help somewhat, if it happened; it’s mostly just hard to say, and we’ll need other major changes to happen first. Those other major changes are more the thing I’d suggest focusing on.)
What are they?
(I don’t think anyone has written a scenarios that make the world go a lot better post/doc; it might be useful.)
How are you operationalizing “warning shot” and how low do you think the chances are that a warning shot operationalized that way happens and the world is not destroyed within e.g. the following three years?
I’m thinking of a ‘warning shot’ roughly as ‘an event where AI is widely perceived to have caused a very large amount of destruction’. Maybe loosely operationalized as ‘an event about as sudden as 9/11, and at least one-tenth as shocking, tragic, and deadly as 9/11’.
I don’t have stable or reliable probabilities here, and I expect that other MIRI people would give totally different numbers. But my current ass numbers are something like:
12% chance of a warning shot happening at some point.
6% chance of a warning shot happening more than 6 years before AGI destroys and/or saves the world.
10% chance of a warning shot happening 6 years or less before AGI destroys and/or saves the world.
My current unconditional odds on humanity surviving are very low (with most of my optimism coming from the fact that the future is just inherently hard to predict). Stating some other ass numbers:
Suppose that things go super well for alignment and timelines aren’t that short, such that we achieve AGI in 2050 and have a 10% chance of existential success. In that world, if we held as much as possible constant except that AGI comes in 2060 instead of 2050, then I’m guessing that would double our success odds to 20%.
If we invented AGI in 2050 and somehow impossibly had fifteen years to work with AGI systems before anyone would be able to destroy the world, and we knew as much, then I’d imagine our success odds maybe rising from 10% to 55%.
The default I expect instead is that the first AGI developer will have more than three months, and less than five years, before someone else destroys the world with AGI. (And on the mainline, I expect them to have ~zero chance of aligning their AGI, and I expect everyone else to have ~zero chance as well.)
If a warning shot had a large positive impact on our success probability, I’d expect it to look something like:
’20 or 30 or 40 years before we’d naturally reach AGI, a huge narrow-AI disaster unrelated to AGI risk occurs. This disaster is purely accidental (not terrorism or whatever). Its effect is mainly just to cause it to be in the Overton window that a wider variety of serious technical people can talk about scary AI outcomes at all, and maybe it slows timelines by five years or something. Also, somehow none of this causes discourse to become even dumber; e.g., people don’t start dismissing AGI risk because “the real risk is narrow AI systems like the one we just saw”, and there isn’t a big ML backlash to regulatory/safety efforts, and so on.′
I don’t expect anything at all like that to happen, not least because I suspect we may not have 20+ years left before AGI. But that’s a scenario where I could (maybe, optimistically) imagine real, modest improvements.
If I imagine that absent any warning shots, AGI is coming in 2050, and there’s a 1% chance of things going well in 2050, then:
If we add a warning shot in 2023, then I’d predict something like: 85% chance it has no major effect, 12% chance if makes the situation a lot worse, 3% chance it makes the situation a lot better. (I.e., an 80% chance that if it has a major effect, it makes things worse.)
This still strikes me as worth thinking about some, in part because these probabilities are super unreliable. But mostly I think EAs should set aside the idea of warning shots and think more about things we might be able to cause to happen, and things that have more effects like ‘shift the culture of ML specifically’ and/or ‘transmit lots of bits of information to technical people’, rather than ‘make the world as a whole panic more’.
I’m much more excited by scenarios like: ‘a new podcast comes out that has top-tier-excellent discussion of AI alignment stuff, it becomes super popular among ML researchers, and the culture, norms, and expectations of ML thereby shift such that water-cooler conversations about AGI catastrophe are more serious, substantive, informed, candid, and frequent’.
It’s rare for a big positive cultural shift like that to happen; but it does happen sometimes, and it can result in very fast changes to the Overton window. And since it’s a podcast containing many hours of content, there’s the potential to seed subsequent conversations with a lot of high-quality background thoughts.
To my eye, that seems more like the kind of change that might shift us from a current trajectory of “~definitely going to kill ourselves” to a new trajectory of “viable chance of an existential win”.
Whereas warning shots feel more unpredictable to me, and if they’re unhelpful, I expect the helpfulness to at best look like “we were almost on track to win, and then the warning shot nudged us just enough to secure a win”.
Very interesting analysis—thank you.
I thought this was great because it has concrete and specific details, times and probabilities. I think it’s really hard to write these out.
Well, it’s less effort insofar as these are very low-confidence, unstable ass numbers. I wouldn’t want to depend on a plan that assumes there will be no warning shots, or a plan that assumes there will be some.
Would it be possible for the EA forum to add footnote functionality? Thanks!
Hi! We do have footnotes.[1] Are you asking for something more specific (e.g. a different way of inserting them)?
We’re also currently facing a bug that makes it impossible to add bullet points or numbered lists in footnotes.
[Edit: I apparently had an old tab open — Linch has already answered. :) ]
(Here’s an example.)
Ah sweet, thank you! Didn’t know this existed, glad to see it and just used it :)
It’s been live for a while[1]!
Unless this isn’t what you’re referring to.
Ah sweet, thank you! Didn’t know this existed, glad to see it and just used it :)
GeWWC: Getting What We Can
I was particularly inspired by Kat Woods’ statement that fundraising is one of the highest leverage meta-skills you can develop. I work in fundraising, and agree. Holden Karnofsky categorized fundraising as a skill under the aptitude of “organization building, running and boosting” and he also acknowledged in Ep. 110 of the 80,000 Hours Podcast that the lack of a fundraising base as a grantee risk at Open Philanthropy.
So, could we be doing more to attract, train and upskill fundraising professionals? Is it something 80,000 Hours recommends in coaching calls? Should an EA-aligned fund establish or invest in a bootcamp-style course or program that emphasizes membership and smaller-gift fundraising because of it shares so many traits with community building?
Focusing more on data governance:
GovAI now has a full-time researcher working on compute governance. Chinchilla’s Wild Implications suggests that access to data might also be a crucial leverage point for AI development. However, from what I can tell, there are no EAs working full time on how data protection regulations might help slow or direct AI progress. This seems like a pretty big gap in the field.
What’s going on here? I can see two possible answers:
Folks have suggested that compute is relatively to govern (eg). Someone might have looked into this and decided data is just too hard to control, and we’re better off putting our time into compute.
Someone might already be working on this that I just haven’t heard of.
If anyone has an answer to this I’d love to know!
NB: One reason this might be tractable is that lots of non-EA folks are working on data protection already, and we could leverage their expertise.
How to upskill in AI Safety after AGISF, and how to help others upskill
For people who want to start working on AI Safety, it feels like the AGISF program is widely accepted as the start of the pipeline. It also feels like there are some things we want to see work on within AI Safety: conceptual work, engineering work, empirical work, etc.
However, I think there’s an upskilling gap that isn’t currently clearly signposted by the community: how to go from AGISF to being able to contribute! The main programmes I’m aware of for taking someone who knows a bit about alignment to being able to contribute to AI Safety are SERI MATS (trying to produce conceptual researchers), MLAB (trying to produce engineers), and internships at orgs like Conjecture, CHAI etc.
Without having done these, my impression is that they seem like the best options for upskilling past AGISF, but these are bottlenecked pretty hard by mentor time. So, I think there are plenty of people who could be doing useful work with sufficient upskilling, but who will have to do upskilling more independently. For these people, useful resources like How to pursue a career in technical AI Alignment by Charlie Rogers-Smith exist, but I think some clear signposting for people post-AGISF, pre-job would be useful.
So, after AGISF, what should an excited safety person be doing to upskill? Here are a few ideas
Engineering:
MLAB
MLSS
Levelling Up in AI Safety Research Engineering
Jacob Hilton’s Deep Learning Curriculum
Conceptual work:
SERI MATS
Concrete advice for Forming Inside Views on AI Safety
Practise distillations
Reading groups
Infra-Bayesianism problem set
How to use these resources?
I think for almost all of these resources, the order of how good I would expect them to be is:
In person with mentorship > Online with mentorship > In person with no mentorship > Online with no mentorship >> solo. A caveat is that I’m not super sure about how high to rate mentorship compared with simply working through these resources with other enthusiastic people. It seems less important for some areas of engineering work than it does for conceptual work (like getting research taste), but I’m not sure about this.
Implications for AI Safety field-builders.
I think the gap between someone doing AGISF and doing impactful work has two implications for field builders in AI safety:
Firstly, I think there would be a lot of value in creating online versions of programmes which essentially go through some of these resources, similarly to how AGISF currently works. I think the biggest bottleneck for these would be mentor time, but I think lots of these could be used successfully without mentors, if there are others who are excited to work through the resources who also have some background in the area.
Secondly, I think that field builders working in local groups with enough members and resources (Oxford, Berkeley, London) could try to run versions of these programmes in person. If online versions of these programmes exist, this becomes even easier: essentially all it requires is scheduling when these groups need to meet!
Conclusion
If you have done AGISF and want to start upskilling, hopefully some of these resources will be useful!
If you want to help others upskill, I think running programmes centred around some of these resources would be a good idea!
What do we do as newcomers? How do we make sense of all of the information out there? How do I get involved if I’m not an academic, and I don’t have “career experience” to fall back on? I’ve just published a blog post about some of my impressions as a newcomer to EA. (Though I don’t know if I answer any of these questions very well, because I don’t know if I have the answers yet.)
Hi Danielle, welcome! I really enjoyed reading your blog post – it seems like you have a skill for communicating clearly, and I’d love to read more of your writing as you keep thinking more about EA :)
One thing you can do as you learn more is to try and crystallize what cruxes you have for your beliefs. Talking about cruxes is a common way EAs communicate, as it can help people avoid talking past each other and instead focus primarily on the things that would change each other’s minds.
My take is that it’s much more valuable to spend time early on thinking about your values and what you think the most important problems are, compared to finding an EA-relevant job quickly (because there are compelling cases for a lot of cause areas, and you might find that you change your mind about what path you want to pursue).
In terms of getting connected, you could try going to a meetup in your city (if there is one). Some people enjoy their city EA groups, others don’t as much. EAGs and EAGx’s can also be good places to meet people who share your interests. Yes, many groups use Facebook to coordinate, but I think there is less significant community discussion on FB than there used to be.
Hope this helps and good luck!
In the recent interview with Katja Grace referenced on ACX, she mentioned that many people may be opposed to slowing down AI progress because (I’m paraphrasing) they perceive AGI as a genie that will solve longevity and other problems and bring about the Cool Transhumanist Future, which they won’t see otherwise—due to their age and/or longer timelines. I have been hanging out in longevity/anti-aging spaces for a while and this perspective is exceedingly common there. People are very hyped about AI coming and curing aging, and dismiss any concerns about AI safety.
From this point of view, solving aging, or even just making a tangible progress towards LEV , will make those people less resistant to the notion of slowing/stopping AI progress. This is complementary to the often mentioned idea that longer life expectancy will cause people to care more about AGI (and other existential risks). This of course doesn’t imply that anti-aging is more important than direct work on AI alignment, but 1) it is likely more tractable and 2) not everyone can or want to work on AI alignment directly.
No Plans for Misaligned AI:
This talk by Jade Leung got me thinking—I’ve never seen a plan for what we do if AGI turns out misaligned.
The default assumption seems to be something like “well, there’s no point planning for that, because we’ll all be powerless and screwed”. This seems mistaken to me. It’s not clear that we’ll be so powerless that we have absolutely no ability to encourage a trajectory change, particularly in a slow takeoff scenario. Given that most people weight alleviating suffering higher than promoting pleasure, this is especially valuable work in expectation as it might help us change outcomes from ‘very, very bad world’ to ‘slightly negative’ world. This also seems pretty tractable—I’d expect ~10hrs thinking about this could help us come up with a very barebones playbook.
Why isn’t this being done? I think there are a few reasons:
Like suffering focused ethics, it’s depressing.
It seems particularly speculative—most of the ‘humanity becomes disempowered by AGI’ scenarios look pretty sci-fi. So serious academics don’t want to consider it.
People assume, mistakenly IMO, that we’re just totally screwed if AI is misaligned.
Can someone guide me towards studies, meta-reviews or collections of studies which quantifies the effectiveness of various interventions?
I.e. “$10 000 spent on deworming medication in Sub Saharan countries saves on average 1 life” or “access to clean water wells in Sub Saharan countries increased educational performance by x points”.
Thanks!
It seems like Mongolia is underrated as a case study in international development. Despite having some major factors pushing against it (landlocked, stuck between great powers, sparsely populated, recently colonized, recently communist), it does reasonably well on economic and political indicators.
The Cannonball Problem:
Doing longtermist AI policy work feels a little like aiming heavy artillery with a blindfold on. We can’t see our target, we’ve no idea how hard to push the barrel in any one direction, we don’t know how long the fuse is, we can’t stop the cannonball once it’s in motion, and we could do some serious damage if we get things wrong.
Longtermist legal work seems particularly susceptible to the Cannonball Problem, for a few reasons:
Changes to hard law are difficult to reverse—legislatures rarely consider issues more than a couple times every ten years, and the judiciary takes even longer.
At the same time, legal measures which once looked good can quickly become ineffectual due to shifts in underlying political, social or economic circumstances.
Taken together, this means that bad laws have a long time to do a lot of harm, so we need to be careful when putting new rules on the books.
This is worsened by the fact that we don’t know what ideal longtermist governance looks like. In a world of transformative AI, it’s hard to tell if the rule of law will mean very much at all. If sovereign states aren’t powerful enough to act as leviathans, it’s hard to see why influential actors wouldn’t just revert to power politics.
Underlying all of this are huge, unanswered questions in political philosophy about where we want to end up. A lack of knowledge about our final destination makes it harder to come up with ways to get there.
I think this goes some way to explaining why longtermist lawyers only have a few concrete policy asks right now despite admirable efforts from LPP, GovAI and others.
Writing this from the “Writing on the EA Forum” workshop at EAGxBerlin, wanted to post some… post ideas:
(a) A quick report on how the conference went, and the particular vibes I encountered, expanding on my “like an academic conference with the annoying prestige layers removed” idea.
(b) Content creation as lowering barriers to entry for a given field, especially language barriers. It seems like an obvious point but on I’ve never seen written anywhere?
(c) Something about the impact of sharing useful trivia and life advice. I have a recent example about hearing damage/loss prevention, what’s the marginal value of these conversations?
(Thanks to Lizka for the nudge and excellent presentation)
I think there are currently too few infosec people and people trying to become billionaires.
Infosec: this seems really helpful for AI safety and biosecurity in a lot of worlds, and I’m guessing it’s just much less sexy / popular than technical research. Maybe I’m wrong about the number of people here, but from attendance at an EAGxSF event it didn’t seem like we would be saturated.
Entrepreneurship: I think the basic argument for making tens of billions of dollars still holds. Just because many longtermist orgs are well-funded now doesn’t mean they will be in the future (crypto and other risk), and there might be ways to spend hundreds of billions of dollars on other things. My understanding is that even at the crypto peak there were <50 EAs trying to become billionaires, and there are even fewer now, which seems like a mistake.
I’ve thought about both of these paths myself, and I think they’re not quite as good as technical alignment research for me, but I can’t rule out that I’m just being a coward.
Random idea: EA safety net for entrepreneurship
(This is probably a bad idea that I haven’t thought through)
Suppose a startup raises $500k seed funding from VCs at $5M valuation, and the startup also commits $2M worth of illiquid stock to EA causes. Then EA could also provide in return another (say) $500k cash that is ear-marked for the founders and early employees’ personal bank accounts, and not towards startup growth. This ensures the founders are not exposed to too much financial anxiety if they fail.
I’m predicting a 10-25% probability that Russia will use a weapon of mass destruction (likely nuclear) before 2024. This is based on only a few hours of thinking about it with little background knowledge.
Russian pro-war propagandists are hinting at use of nuclear weapons, according to the latest BBC podcast Ukrainecast episode. [Ukrainecast] What will Putin do next? #ukrainecast https://podcastaddict.com/episode/145068892 via @PodcastAddict
There’s a general sense that, in light of recent losses, something needs to change. My limited understanding sees 4 options:
Continue on the current course despite mounting criticism. Try to make the Ukrainians lives difficult by targeting their infrastructure, limit losses until winter, and try to reorganize during winter. This seems a pretty good option for now, even though I doubt Russia can really shore up its deeply set weaknesses. They can probably prepare to dig in, threaten and punish soldiers for fleeing. This wouldn’t go well for either party long-term, but Russia might bet on outlasting/undermining Western support. Probability: 40%?
Negotiation: I don’t think Putin wants this seriously, as even the status quo could be construed as a loss. Ukraine will have a strong bargaining position and demand a lot. Undesirable option. Maybe 10%? 20%? (Metaculus predicts 8% before 2023: https://www.metaculus.com/questions/10046/ukraine—russia-peace-talks-2022/)
Full-scale mobilisation of the population and the economy. This is risky for Putin: there’s supposedly a large anti-war sentiment in Russian culture, a legacy of the enormous losses during the 2nd World War. People don’t like to join a poorly-equipped poorly managed and losing army, even if it were a good cause.. This may be chosen, Putin may be misinformed and badly reading the public’s sentiment. I have no idea how this would develop internally. I doubt it will make a big difference in the course of the war, except by prolonging the war a bit. Maybe 25%? Maybe 50% if Putin underestimates public resistance.
Escalation by other means: I don’t know how many options Russia has. Chemical weapons, electro magnetic pulse, a single tactical nuclear strike on the battlefield for deterrence, multiple nuclear strikes for strategic reasons, population strike for deterrence. In the mind of Putin, I can see this as preferable: it leads to a potential military advantage, has limited risk for destabilising his internal power base. I don’t know how the international community would respond to this, nor how Putin thinks the international community would respond. In my (uninformed) view, only China can make a real difference here as the West already has stringent sanctions. I don’t know how China would respond to this. They wouldn’t like it, but I think the West won’t really punish China for its support in the short term. I’d say on this inside view, 10-25% seems reasonable. I’m setting the point estimate at 15%.
Any chance you’d wanna bet on this + find a suitable escrow (ideally crypto)? I could bet non-trivial sum on opposite side, imo odds less than 15%
A distinction I’ve found useful is “object-level” vs “social reality”. They are both adjectives that describe types of conversation/ ideas.
Object-level discussions are about ideas and actions (e.g. AI timelines, the mechanics of launching a successful startup). Object-level ideas are technical, empirical, and often testable. Object-level refers to what ideas are important or make sense. It is focused on truth-seeking and presenting arguments clearly.
Social reality discussions are about people and organisations (e.g. Will MacAskill, Open Philanthropy). Social reality is more meta, more abstract, and less testable than object-level. Social reality refers to which people are influential/powerful (and what they think), how to network with people, how to persuade people.
I have found it very helpful to start labelling whether I’m in object-level conversation mode vs social reality conversation mode. It helps me notice when I’m deferring without having thought about it (e.g. “well, Will MacAskill says [x]” instead of asking myself what I think about [x]), or when I fall into a mode of chit-chatting about the who’s-who of EA, instead of trying to truth-seek (of course, chit chatting sometimes is fine—I just want to be intentional about when I’m doing it).
And social reality isn’t necessarily bad, but it’s helpful to flag when a conversation enters “social reality mode.”
I do think it’s good for many/more/most conversations to centre around the object-level. I am personally trying to move my ratio more towards object-level.
(This was a core theme of an Atlas camp I attended, which I found extremely valuable. The above definitions are loosely based on a message from Jonas, but I didn’t run them by him before posting.)
I feel like defining a separate term for this may be better. Maybe:
Integrated view = defensible view + delegated view
where defensible view contains details you personally can defend, and delegated view includes other people’s views and whatever factor of delegation you feel is appropriate
This relates to a caveat in my recent post:
I’m concerned about too much EA meta conversation — about worlds where most of EA dialogue is talking about EAs talking about EAs (lots of social reality and not enough object-level).
These sorts of convos are often very far removed from {concrete things that help the world}, and I worry about them taking away attention from more important ones.
I think it’s probably much better (for the world) for conversations to stay focused on the real world, object-level claims and arguments.
Part of me wants to flesh this thought out properly soon. But even this conversation is meta! And I’m trying to encourage/ focus more on object-level ideas. So do I write it? I’m not sure.
Expected Value of galaxy-scale expansion assuming discount rate utilitarianism
Assume a spherical volume expanding from earth at speed ac, converting matter into morally valuable beings.
(where c = 3 * 10^8 m/s, a is small constant indicating sphere going slower than light)
volume of sphere at time t = 4⁄3 pi (ac t)^3
mass of sphere at time t = 4⁄3 pi (ac t)^3 rho
(where rho = density of universe = 10^-26 kg/m^3 approx)
total beings in sphere at time t = 4⁄3 pi (ac t)^3 rho M
(where M = mass of a being = 10^2 kg for a biological human for instance)
Utility of universe earned between t and t+dt = 4⁄3 pi (ac t)^3 rho M U e^(-bt) dt
(where b is discount rate of moral value of beings, U = 1 utilon per unit time)
Total utility earned = integrate [ (4/3 pi (ac t)^3 rho M U e^(-bt)) dt ]
Total utility earned = 4⁄3 pi (ac)^3 rho M U integrate [t^3 e^(-bt] dt = A I
A = 4⁄3 pi (ac)^3 rho M U = 113 a^3 U
I = integrate [t^3 e^(-bt] dt = e^(-bt) (b^3 t^3 + 3 b^2 t^2 + 6 b t + 6) / b^4
t varies 0 to r/ac
(where r is radius of universe = 4.4 * 10^26 m)
r/ac = (1.5 * 10^ 18 seconds) / a
Will finish in 2nd part
Poke holes in my systematizing outreach apologism
Re: Ick at systematizing outreach and human interactions
There’s a paradox I’m confused about, where if someone from a group I’m not in—let’s say Christians, came to me on a college campus and smiled at me and asked about my interests and connected all of them to Jesus and then I found out I’d been logged in a spreadsheet as “potential convert” or something and then found the questions they’d asked me in a blog post or “Christian evangelist top questions” I might very well feel extremely weird about that (though I think less than others, I kind of respect the hustle).
BUT, when I think about how one gets there, I think, ok:
You’re a christian, you care about saving other people from hell
You want to talk to people about this and get a community together + persuade people via arguments you think are in fact persuasive
Other people want to do the same, you discuss approaches
Other people have framings and types of questions that seem better to you than yours, so you switch
You’re talking to a lot of people and it’s hard to keep track of what each of them said and what they wanted out of a community or worldview, so you start writing it down
You don’t want people to get approached for the same conversations over and over again, so you share what you’ve written with your fellow Christian evangelists
It doesn’t seem useful to anyone to keep talking to people who don’t seem interested in Christianity, so you let your fellow evangelists know which folks are in that category
People who seem excited about Christianity would probably get a lot out of going to conferences or reading more about it, so you recommend conferences and books and try to make it as easy as possible for them to access those, without having annoying atheists who just want to cause trouble showing up.
This is probably too charitable, there is definitely a thing where you actively want to persuade people because you think your thing is important, and you might lose interest in people who aren’t excited about what you’re excited about, but those things also seem reasonable to me.
A process that seems bad:
Want to maximize number of EAs
Use framings, arguments and examples that you don’t think hold water but work at getting people to join your group [I don’t think EAs do this, I’m gesturing at the extreme other end]
Make people feel weird and bad for disagreeing with you, whether on purpose or not
Encourage people to repress their disagreements
Get energy and labor from people that they won’t endorse having given in a few years, or if they knew things you knew
3-5 seem like the worst parts here. 1 seems like a reasonable implication of their beliefs, though I do think we all have to cooperate to not destroy the commons.
2 is complicated—when people have different cruxes than you is it dishonest to talk about what should convince them based on their cruxes?
3 and 4 are bad, also hard to avoid.
5 seems really bad, and something I’d like to strongly avoid via things like transparency and some other percolating advice I might end up endorsing for people new to EA, like not letting your feet go faster than your brain, figuring out how much deference you endorse, seeing avoiding resentment as a crucial consideration in your life choices, staying grounded, etc.
I also think the processes can feel pretty similar from the inside (therefore danger alert!) but also look similar from the outside when they aren’t. I certainly have systematically underestimated the moral seriousness and earnestness of many an EA.
What’s the difference?
I think people are going to want to say something like “treating people as ends” but I don’t know where that obligation stops. I think I want to say something like “are you acting in the interests of the people you’re talking to”, but that doesn’t work either—I’m not! being an EA has a decent chance of being less pleasant than the other thing they were doing, and either way it’s not a crux. Ex: I endorse protecting the time and energy of other people by not telling everyone who I would talk to if I had a certain question or needed help in a certain way.
I do think it’s more about whether you’re doing things in such a way that if they knew why you were doing them, they’d mostly not be bothered (ie passing the red face test). But that doesn’t really solve the problem that digital sentience is a weird reason to do a lot of things, and there are lots of things I endorse it being inappropriate to be too explicit about.
[This is separate from the instrumental reasons to act differently because it weirds people out etc.]
.......................................................................................................................
Later musings:
Presumably the strongest argument is that these feelings are tracking a bunch of the bad stuff that’s hard to point at:
people not actually understanding the arguments they’re making
people not having your best interests in mind
people being overconfident their thing is correct
people not being able to address your ideas / cruxes
people having bad epistemics
Of course this is a spectrum, and we shouldn’t put up a public website listing all our beliefs including the most controversial ones or something like that (no one in EA is very close to this extreme). But the implicit jump from “some things shouldn’t be explicit” to “digital sentience might weird some people out so there’s a decent chance we shouldn’t be that explicit about it” seems very non-obvious to me, given how central it is to a lot of longtermist’s worldviews and honestly I think it wouldn’t turn off many of the most promising people (in the long run; in the short run, it might get an initial “huh??” reaction).
I think steps 1 and 2 in your chain are also questionable, not just 3-5.
Why do we want to maximize number of EAs, this seems very non-obvious to me? Some people would add much more to the community than others via epistemics, culture, direct talent, etc. If we added enough of certain types of people to the community, especially too quickly, it could easily be net negative.
I think sometimes/often talking about people’s cruxes rather than your own is good and fine. The issue is Goodharting via an optimal message to convert as many people to EA as quickly as possible, rather than messages that will lead to a healthy community over the long run.
Thanks so much for writing this! I think it could be a top-level post, I’m sure many others would find it very helpful.
My 2 cents:
I think it’s definitely bad to “Use framings, arguments and examples that you don’t think hold water but work at getting people to join your group”. If I understand correctly it can cause point 5. Also “getting people to join your group” is rarely an instrumental goal, and “getting people to join your group for the wrong reasons” is probably not that useful in the long term.
Something that I think is very important that seems missing from this is that there’s a significant probability that we’re wrong about important things (i.e. EA as a question).
We could be wrong about the impact of bednets, wrong about AI being the most important thing, wrong about population ethics, etc. I think it’s a huge difference from the “cult” mindset.
The way I think about this, on first approximation, is that I want people to work on maximising their values (and not their wellbeing). If they think altruism is not important and are solipsistic egoists and only value their own wellbeing, I don’t think EA can help them. If they value the wellbeing of others then EA can help them achieve their values better.
From my personal perspective this is strongly related to the point on uncertainty: I don’t want to push other people to work on my values because from an outside view I don’t think my values are more important than their values, or more likely to be “correct”.
I don’t know if it makes any sense, really curious to hear your thoughts, you have certainly thought about this more than I.
Thanks, Lorenzo!
Agree about the “not holding water”, I was trying to say that “addresses cruxes you don’t have” might look similar to this bad thing, but I’m not totally sure that’s true.
I disagree about getting people to join your group—that definitely seems like an instrumental goal, though definitely “get the relevant people to join your group” is more the thing—but different people might have different views on how relevant they need to be, or what their goal with the group is.
I kind of agree here; I think there are things in EA I’m not particularly uncertain of, and while I’m open to being shown I’m wrong, I don’t want to pretend more uncertainty than I have.
I’ve definitely heard that frame, but it honestly doesn’t resonate for me. I think some people are wrong about what values are right and arguing with me sometimes convinces them of that. I’ve definitely had my values changed by argumentation! Or at least values on some level of abstraction—not on the level of solipsism vs altruism, but there are many layers between that and “just an empirical question”.
I incorporate an inside view on my values—if I didn’t think they were right, I’d do something else with my time!
I noticed this a while ago. I don’t see large numbers of low-quality low-karma posts as a big problem though (except that it has some reputation cost for people finding the Forum for the first time). What really worries me is the fraction of high-karma posts that neither original, rigorous, or useful. I suggested some server-side fixes for this.
PS: #3 has always been true, unless you’re claiming that more of their output is private these days.
#4 has also been true for many years