Linch

Karma: 24,421

Linch 9 Sep 2023 7:33 UTC
127 points
34 ∶ 1
in reply to: Ebenezer Dukakis’s comment on: Sharing Information About Nonlinear
I think people are overcomplicating this. You should generally follow the law, but to shield against risks that you are being such a stickler in unreasonable ways (trying to avoid “3 felonies a day”), you can just imagine whether uninvolved peers hearing about your actions would think a situation is obviously okay. Some potential ways to think about such peer groups:
1. What laws people in the country you live in think are absolutely normal and commonplace to break.
  1. For example, bribing police officers is generally illegal, but iiuc in some countries approximately everybody bribes police officers at traffic stops
2. What laws people in your home country think is illegitimate and thus worth breaking
  1. For example some countries ban homosexuality, but your typical American would not consider it blameworthy to be gay.
3. What laws other EAs (not affiliated in any way with your organization) think is okay to break.
  1. So far, candidates people gave include ag-gag laws and taking stimulants for undiagnosed ADHD.
    FWIW I’m not necessarily convinced that the majority of EAs agree here; I’d like to see polls.
4. What laws your non-EA friends think is totally okay to break
  1. For example, most college-educated Millennials would not consider downloading papers on Sci-Hub to be blameworthy.
I think a relatively conservative organization should find it permissive to break the law if and only if every possible reasonable peer group considers it acceptable, and a relatively liberal but probably acceptable organization may consider it permissive to break laws if some reasonable peer group considers it acceptable.

In the most problematic examples in question, all the alleged law-breaking I’m aware of (driving without a license^[1], pressuring someone to transport recreational drugs across borders) are not likely to be considered readily acceptable by any reasonable peer group, including other EAs.
This is similarly true for other law-breaking scandals I’ve heard about in the past, including theft, fraud, workplace sexual harassment, etc.
1. ^
  Puerto Rico is in the US
What links here?
- bruce's comment on Sharing Information About Nonlinear by Ben Pace (9 Sep 2023 15:29 UTC; 62 points)

Linch 23 Jun 2021 1:18 UTC
116 points
0 ∶ 0
on: Linch’s Shortform
Red teaming papers as an EA training exercise?
I think a plausibly good training exercise for EAs wanting to be better at empirical/conceptual research is to deep dive into seminal papers/blog posts and attempt to identify all the empirical and conceptual errors in past work, especially writings by either a) other respected EAs or b) other stuff that we otherwise think of as especially important.
I’m not sure how knowledgeable you have to be to do this well, but I suspect it’s approachable for smart people who finish high school, and certainly by the time they finish undergrad^ with a decent science or social science degree.
I think this is good career building for various reasons:
- you can develop a healthy skepticism of the existing EA orthodoxy
  - I mean skepticism that’s grounded in specific beliefs about why things ought to be different, rather than just vague “weirdness heuristics” or feeling like the goals of EA conflict with other tribal goals.
    (I personally have not found high-level critiques of EA, and I have read many, to be particularly interesting or insightful, but this is just a personal take).
- you actually deeply understand at least one topic well enough to point out errors
- For many people and personalities, critiquing a specific paper/blog post may be a less hairy “entry point” into doing EA-research adjacent work than plausible alternatives like trying to form your own deep inside views on questions that are really open-ended and ambiguous like “come up with a novel solution in AI alignment” or “identify a new cause X”
- creates legible career capital (at least within EA)
- requires relatively little training/guidance from external mentors, meaning
  - our movement devotes less scarce mentorship resources into this
  - people with worse social skills/network/geographical situation don’t feel (as much) at a disadvantage for getting the relevant training
- you can start forming your own opinions/intuitions of both object-level and meta-level heuristics for what things are likely to be correct vs wrong.
- In some cases, the errors are actually quite big, and worth correcting (relevant parts of ) the entire EA movement on.
Main “cons” I can think of:
- I’m not aware of anybody successfully doing a really good critique for the sake of doing a really good critique. The most exciting things I’m aware of (publicly, zdgroff’s critique of Ng’s original paper on wild animal suffering, alexrjl’s critique of Giving Green. I also have private examples) mostly comes from people trying to deeply understand a thing for themselves, and then along the way spotting errors with existing work.
- It’s possible that doing deliberate “red-teaming” would make one predisposed to spot trivial issues rather than serious ones, or falsely identify issues where there aren’t any.
- Maybe critiques are a less important skill to develop than forming your own vision/research direction and executing on it, and telling people to train for this skill might actively hinder their ability to be bold & imaginative?
^ Of course, this depends on field. I think even relatively technical papers within EA are readable to a recent undergrad who cares enough, but this will not be true for eg (most) papers in physics or math.
What links here?

Linch 9 Dec 2022 1:26 UTC
111 points
35 ∶ 21
on: Why did CEA buy Wytham Abbey?
I’m confused why people keep insisting this is a “CEA” decision even after Owen Cotton-Barratt’s clarification (which I assume everyone commenting has read).

I see the process on deciding to purchase Wytham Abbey as:
1. Owen Cotton-Barratt made a proposal to spend ~$15M for a conference center
2. His funder(s) were willing to give him money.
3. Effective Ventures agreed to be a fiscal sponsor.
To the extent that anyone is responsible for this decision, it’s primarily (1) Owen, and (2) his funder(s). I don’t think (3) is much to blame here. Also, CEA the organization is distinct from EV, their fiscal sponsor.
I think if you think this is an ineffective use of limited resources, you absolutely should feel entitled to critique it! In many ways this is what our movement is about! But I think you should place the burden of blame on the actual decision-makers, and not vaguely associated institutions.
What links here?
- CEA Disambiguation by Jeff Kaufman (19 Dec 2022 13:27 UTC; 134 points)
- CEA Disambiguation by jefftk (LessWrong; 19 Dec 2022 13:20 UTC; 24 points)

Linch 3 Dec 2022 4:25 UTC
108 points
1 ∶ 0
on: Linch’s Shortform
tl;dr:
In the context of interpersonal harm:
1. I think we should be more willing than we currently are to ban or softban people.
2. I think we should not assume that CEA’s Community Health team “has everything covered”
3. I think more people should feel empowered to tell CEA CH about their concerns, even (especially?) if other people appear to not pay attention or do not think it’s a major concern.
4. I think the community is responsible for helping the CEA CH team with having a stronger mandate to deal with interpersonal harm, including some degree of acceptance of mistakes of overzealous moderation.
(all views my own) I want to publicly register what I’ve said privately for a while:

For people (usually but not always men) who we have considerable suspicion that they’ve been responsible for significant direct harm within the community, we should be significantly more willing than we currently are to take on more actions and the associated tradeoffs of limiting their ability to cause more harm in the community.
Some of these actions may look pretty informal/unofficial (gossip, explicitly warning newcomers against specific people, keep an unofficial eye out for some people during parties, etc). However, in the context of a highly distributed community with many people (including newcomers) that’s also embedded within a professional network, we should be willing to take more explicit and formal actions as well.
This means I broadly think we should increase our willingness to a) ban potentially harmful people from events, b) reduce grants we make to people in ways that increase harmful people’s power, c) warn organizational leaders about hiring people in positions of power/contact with potentially vulnerable people.
I expect taking this seriously to involve taking on nontrivial costs. However, I think this is probably worth it.
I’m not sure why my opinion here is different from others^[1]’, however I will try to share some generators of my opinion, in case it’s helpful:

A. We should aim to be a community that’s empowered to do the most good. This likely entails appropriately navigating the tradeoff of both attempting to reducing the harms of a) contributors feeling or being unwelcome due to sexual harassment or other harms and b) contributors feeling or being unwelcome due to false accusations or overly zealous response.
B. I think some of this is fundamentally a sensitivity vs specificity tradeoff. If we have a detection system that’s too tuned to reduce the risk of false positives (wrong accusations being acted on), we will overlook too many false negatives (people being too slow to be banned/censured, or not at all), and vice versa.
Consider the first section of “Difficult Tradeoffs”
Avoid false negatives: take action if there’s reason to think someone is causing problems Avoid false positives: don’t unfairly harm someone’s reputation / ability to participate in EA
In the world we live in, I’ve yet to hear of a single incidence where, in full context, I strongly suspect CEA CH (or for that matter, other prominent EA organizations) was overzealous in recommending bans due to interpersonal harm. If our institutions are designed to only reduce first-order harm (both from direct interpersonal harm and from accusations), I’d expect to see people err in both directions.
Given the (apparent) lack of false positives, I broadly expect we accept too high a rate of false negatives. More precisely, I do not think CEA CH’s current work on interpersonal harm will lead to a conclusion like “We’ve evaluated all the evidence available for the accusations against X. We currently think there’s only a ~45% chance that X has actually committed such harms, but given the magnitude of the potential harm, and our inability to get further clarity with more investigation, we’ve pre-emptively decided to ban X from all EA Globals pending further evidence.”

Instead, I get the impression that substantially more certainty is deemed necessary to take action. This differentially advantages conservatism, and increases the probability and allowance of predatory behavior.
C. I expect an environment with more enforcement is more pleasant than an environment with less enforcement.
I expect an environment where there’s a default expectation of enforcement for interpersonal harm is more pleasant for both men and women. Most directly in reducing the first-order harm itself, but secondarily an environment where people are less “on edge” for potential violence is generally more pleasant. As a man, I at least will find it more pleasant to interact with women in a professional context if I’m not worried that they’re worried I’ll harm them. I expect this to be true for most men, and the loud worries online about men being worried about false accusations to be heavily exaggerated and selection-skewed^[2].
Additionally, I note that I expect someone who exhibit traits like reduced empathy, willingness to push past boundaries, sociopathy, etc, to also exhibit similar traits in other domains. So someone who is harmful in (e.g.) sexual matters is likely to also be harmful in friendly and professional matters. For example, in the more prominent cases I’m aware of where people accused of sexual assault were eventually banned, they also appeared to have done other harmful activities like a systematic history of deliberate deception, being very nasty to men, cheating on rent, harassing people online, etc. So I expect more bans to broadly be better for our community.
D. I expect people who are involved in EA for longer to be systematically biased in both which harms we see, and which things are the relevant warning signals.
The negative framing here is “normalization of deviance”. The more neutral framing here is that people (including women) who have been around EA for longer a) may be systematically less likely to be targeted (as they have more institutional power and cachet) and b) are selection-biased to be less likely to be harmed within our community (since the people who have received the most harm are more likely to have bounced off).
E. I broadly trust the judgement of CEA CH in general, and Julia Wise in particular.

EDIT 2023/02: I tentatively withhold my endorsement until this allegation is cleared up.
I think their judgement is broadly reasonable, and they act well within the constraints that they’ve been given. If I did not trust them (e.g. if I was worried that they’d pursue political vendettas in the guise of harm-reduction), I’d be significantly more worried about given them more leeway to make mistakes with banning people.^[3]
F. Nonetheless, the CEA CH team is just one group of individuals, and does a lot of work that’s not just on interpersonal harm. We should expect them to a) only have a limited amount of information to act on, and b) for the rest of EA to need to pick up some of the slack where they’ve left off.
For a), I think an appropriate action is for people to be significantly more willing to report issues to them, as well as make sure new members know about the existence of the CEA CH team and Julia Wise’s work within it. For b), my understanding is that CEA CH sees themself as having what I call a “limited purview”: e.g. they only have the authority to ban people from official CEA and maybe CEA-sponsored events, and not e.g. events hosted by local groups. So I think EA community-builders in a group organizing capacity should probably make it one of their priorities to be aware of the potential broken stairs in their community, and be willing to take decisive actions to reduce interpersonal harms.
Remember: EA is not a legal system. Our objective is to do the most good, not to wait to be absolutely certain of harm before taking steps to further limit harm.
One thing my post does not cover is opportunity cost. I mostly framed things as changing the decision-boundary. However, in practice I can see how having more bans is more costly in time and maybe money than the status quo. I don’t have good calculations here, however my intuition is strongly in the direction that having a safer and more cohesive is worth the relevant opportunity costs.
1. ^
  fwiw my guess is that the average person in EA leadership wishes the CEA CH team does more (is currently insufficiently punitive), rather than wish that they did less (is currently overzealous). I expect there’s significant variance in this opinion however.
2. ^
  This is a potential crux.
3. ^
  I can imagine this being a crux for people who oppose greater action. If so, I’d like to a) see this argument explicitly being presented and debated, and b) see people propose alternatives for reducing interpersonal harm that routes around CEA CH.
What links here?
- Linch's comment on EA, Sexual Harassment, and Abuse by whistleblower9 (6 Feb 2023 21:05 UTC; 18 points)

Linch 1 Sep 2022 4:46 UTC
106 points
0 ∶ 0
on: Open EA Global
Not opinionating on the general point, but:
In her early 20s, Kelsey was taking leave from college for mental health reasons and babysitting her friends’ kid for room and board. If either of us had been in the student group, we would have been the least promising of the lot
IIRC, Kelsey was in fact the president of the Stanford EA student group, and I do not think she would’ve been voted “least likely to succeed” by the other members.
What links here?
- Linch's comment on Open EA Global by Scott Alexander (4 Sep 2022 16:27 UTC; 40 points)

Linch 7 Feb 2023 3:29 UTC
104 points
27 ∶ 6
on: [Atlas Fellowship] Why do 100 high-schoolers need $50k each from Open Philanthropy?
Hi, I think on balance I appreciate this post. This is a hard thing for me to say, as the post has likely caused nontrivial costs to some people rather close to me, and has broken some norms that I view as both subtle and important. But on balance I think our movement will do better with more critical thinkers, and more people with critical pushback when there is apparent divergence between stated memes and revealed goals.
I think this is better both culturally, and also is directly necessary to combat actual harm if there is also actual large-scale wrongdoing that agreeable people have been acculturated to not point out. I think it will be bad for the composition and future of our movement if we push away young people who are idealistic and disagreeable, which I think is the default outcome if posts like this only receive critical pushback.
So thank you for this post. I hope you stay and continue being critical.
What links here?
- The number of burner accounts is too damn high by quinn (7 Feb 2023 7:04 UTC; 99 points)

Linch 11 Jul 2022 19:48 UTC
103 points
0 ∶ 0
on: My EA Failure Story
Reading between the lines, you’re a funny writer, are self-aware, were successful enough at work to be promoted multiple times, and have a partner and a supportive family. This is more than what most people can hope for. At some level I think you should be proud of what you’ve accomplished, not just what you tried and failed to do.
Depression really sucks, and it’s unfortunate that this is entangled with trying hard to achieve ambitious EA goals and not succeeding. At the same time, I think the EA community would’ve done right by its members if most of our “failure stories” looked like yours, albeit I’d prefer perhaps more community support in the longer term.

Linch 11 Jul 2022 19:39 UTC
100 points
1 ∶ 0
on: EA for dumb people?
I’m going to be boring/annoying here and say some things that I think are fairly likely to be correct but may be undersaid in the other comments:
- EAs on average are noticeably smarter than most of the general population
- Intelligence is an important component for doing good in the world.
- The EA community is also set up in a way that amplifies this, relative to much of how the rest of the world operates.
- ~~Most people on average are reasonably well-calibrated about how smart they are.~~
  - ~~(To be clear exceptions certainly exist)~~ EDIT: This is false, see Max Daniel’s comment.
- If you’re less smart than average for EAs (or less driven, or less altruistic, or less hardworking, or have less of a social safety net), than on average I’d expect you to be less good at having a positive impact than others.
- But this is in relative terms, in absolute terms I think it’s certainly possible to have a large impact still.
- Our community is not (currently) set up well to accommodate the contributions of many people who don’t check certain boxes, so I expect there to be more of an uphill battle for many such people.
  - I don’t think this should dissuade you from the project of (effectively) doing good, but I understand and emphasize if this makes you frustrated.
What links here?

Linch 11 Nov 2022 6:43 UTC
95 points
32 ∶ 0
in reply to: Pablo’s comment on: Some comments on recent FTX-related events
It’s also listed under GiveWell’s Our Mistakes page.

Linch 27 Aug 2022 20:55 UTC
91 points
0 ∶ 0
on: Climate Change & Longtermism: new book-length report
Here are my high-level thoughts around the comments so far of this report:
- This is a detailed report, where a lot of work has been put in, by one of EA’s foremost scholars on the intersection of climate change and other global priorities.
- So it’d potentially be quite valuable for people with either substantial domain expertise or solid generalist judgement to weigh in here on object-level issues, critiques, and cruxes, to help collective decision-making.
- Unfortunately, all of the comments here are overly meta. Out of the ~60 comments so far on this thread, 0.5 of the comments on this thread approach anything like technical criticism, cruxes, or even engagement.
  - The 0.5 in question is this comment by Karthik.
  - (EDIT: I think Noah’s comment here qualifies)
  - Compare, for example, the following comments to one of RP’s cultured meat reports.
- After saying that, I will hypocritically continue to follow the streak of being meta while not having read the full report.
- I think I’m confused about the quality of the review process so far. Both the number and quality of the reviewers John contacted for this book seemed high. However, I couldn’t figure out what the methodology for seeking reviews is here.
  - To be clear, I’m aware that this is an isolated demand for rigor. My impression is that very few other EA research organizations have a very formal and legible process for prepublication reviews.
  - However, I think for a report of this scope, it might be valuable to have a fairly good researcher sit down and review the report very carefully, in a lot of detail.
- If this is not already done, for people who are not satisfied (or at least queasy) with the report, it might be helpful to commission 1-3 months of research critiquing and redteaming this report in a lot of detail.
- I think John is understandably frustrated by the lack of technical engagement and the overly metaness (and possibly personal acrimoniousness? ) of the comments so far.
- I think John has acted badly in accusing A.C.Skraeling of being a sock puppet account.
- I think every loosely organized emerging research field has “duds,” or low-quality work that in a poorly filtered ecosystem will suck up a lot of the oxygen.
  - For both politeness and practical time-constraints reasons, people tend to ignore the bad work and only engage with the good/mediocre work.
  - Larks is one of the few people who point out this explicitly (for AI alignment). I think his summary of the situation (including why people do this, and downsides) is a worthwhile read, will recommend!
  - My impression is that academia and other relatively formal systems deal with this via more explicit notions of prestige, conference/paper rejections, etc.
    However, this comes with its own dysfunctions and risks ossification.
- Taking a step back, I’m a bit confused about why longtermist researchers continue to spend such significant intellectual resources on studying climate in this fashion. My guess is that anyone who was convinced by earlier research by John, 80k, etc, plus the relevant research and communication into other, riskier, domains, are unlikely to change their minds by critiques of this deeper report. And anybody who wasn’t convinced by the earlier reports are unlikely to be convinced by this one.
  - So I don’t really know the value-add of more intellectual attention into this.
- I looked into climate change for myself for several weeks in 2019 (back when I was substantially worse at both research and judgement). I think I was fairly satisfied that it should not be one of EA’s top priorities, even ignoring neglectedness. My reasoning then was fairly simple.
  - My main reason for thinking that climate change was a significant existential risk (or at least a GCR) was that I had the vague impression that this is a position in academia.
  - My main reason for not believing that was because a) weak inside views about historical resilience to temperature variability and b) other smart and thoughtful EAs didn’t believe this.
  - Thus, the appropriate way to evaluate the evidence is to try to look at the climate change literature, in a way that does not reference past EA work (so as to not be clouded as much by biases/information cascades).
  - I looked at it for a while, and concluded that the literature mostly did not say that climate change was a significant existential risk (of course it was not framed in those terms).
    I thought that climate risk experts should be predisposed to overestimate rather than estimate risks, so there is additional evidence beyond just their conclusions.
  - Given that my reasons for thinking climate change was a significant existential risk weren’t very strong or robust to begin with, I didn’t need that many bits to be convinced that the risks are low.
  - I think my research quality/rigor at the time was pretty meh. Still I don’t think it should swing the overall conclusion.
  - I’m more worried about motivated reasoning/selection effects. Motivated reasoning because I felt like part of the reason for doing that research was convincing myself and others, rather than because I fully impassionately wanted to know the truth. Selection effects because I went into it using an EA methodology and ways of thinking about the world, and of course I’m selected to be one of the people who think in that way.
- Since then, I had 3 updates re: how high climate risk should intuitively “feel”, in my head, in different directions:
  - Jan-March 2020 Covid was a large update for me against a) trusting the object-level of a lot of expert modeling and b) trusting that experts will uniformly overestimate rather than underestimate risk (as opposed to “don’t panic” justifications and conservatism bias). This caused me to worry more about climate stuff.
  - John/Johannes looked into and updated in believing in lower overall climate risk, with reasonable-sounding object-level reasons. This caused me to worry less about climate stuff.
    I did not look into or investigate their reasoning, and generally have not prioritized looking into climate much since 2019.
  - Compared to myself in 2019, I placed relatively more trust in the relative judgement of EAs over that of external people. This caused me to worry less about climate stuff.
  - I think the first update is the largest.
- My understanding is that a lot of the purported value of engaging on the object-level details of climate, and of WWOTF’s aims in general, is to draw in the climate change people as allies.
  - I’m a bit confused about this reasoning. I think:
    the arguments for working on AI risk/biorisk over climate risks seems fairly simple to me, and resilient to a practical range of empirical disagreements within climate.
    to the extent the arguments are bad or poorly communicated, I strongly suspect that they come in almost entirely because AI risk/biorisk arguments are insufficiently grounded, rather than because the climate arguments are insufficiently sophisticated.
    from a persuasion perspective, I think this is maybe a bad move anyway. Maybe better to be “yes, and” than to explicitly and immediately tell people that the thing they’ve devoted their lives to is >100x less important.
- Given that it might not be realistic to find (and may not be very valuable) a senior-ish EA researcher to critique and red-team John’s report, I’d be excited to have several junior EAs try this.
  - Roughly, a summer-long project for someone of the calibre of CERI/SERI might be to go line by line in the report (or pick a few chapters) to find the relevant potential mistakes/disagreements/cruxes (from an LT perspective).
  - This helps train critical thinking abilities and thoughtfulness.
  - To be clear, this is premised on domain-general critical thinking abilities and thoughtfulness being among the most valuable things for junior EAs to train.
    I’m genuinely confused about how much we want less thoughtful people as allies. I think an increasingly common position in longtermist EA nowadays is that the things that matter is AI risk and biorisk (sometimes just AI risk), and what we most/only need are people who are either extremely technically competent (for solving the technical problems) or who are very good at politics (for policy/persuasion).
    Whereas I feel like I sort of represent the older guard of people who still think that thoughtfulness is a huge asset.
    But I acknowledge that this is something I have a (large?) bias to believing is true (because I’m relatively worse at being very technical or social, compared to being thoughtful).
What links here?
- Beware Isolated Demands for Rigor (Alexander, 2014) by Will Aldred (18 Jul 2022 11:48 UTC; 44 points)

Linch 20 Sep 2023 23:26 UTC
87 points
24 ∶ 1
in reply to: richard_ngo’s comment on: richard_ngo’s Shortform
I think it would be helpful for you to mention and highlight your conflict-of-interest here.
I remember becoming much more positive about ads after starting work at Google. After I left, I slowly became more cynical about them again, and now I’m back down to ~2018 levels.
EDIT: I don’t think this comment should get more than say 10-20 karma. I think it was a quick suggestion/correction that Richard ended up following, not too insightful or useful.

Linch 3 Feb 2024 19:16 UTC
85 points
15 ∶ 0
on: Linch’s Shortform
Going forwards, LTFF is likely to be a bit more stringent (~15-20%?^[1] Not committing to the exact number) about approving mechanistic interpretability grants than in grants in other subareas of empirical AI Safety, particularly from junior applicants. Some assorted reasons (note that not all fund managers necessarily agree with each of them):
- Relatively speaking, a high fraction of resources and support for mechanistic interpretability comes from other sources in the community other than LTFF; we view support for mech interp as less neglected within the community.
- Outside of the existing community, mechanistic interpretability has become an increasingly “hot” field in mainstream academic ML; we think good work is fairly likely to come from non-AIS motivated people in the near future. Thus overall neglectedness is lower.
- While we are excited about recent progress in mech interp (including some from LTFF grantees!), some of us are suspicious that even success stories in interpretability are that large a fraction of the success story for AGI Safety.
- Some of us are worried about field-distorting effects of mech interp being oversold to junior researchers and other newcomers as necessary or sufficient for safe AGI.
- A high percentage of our technical AIS applications are about mechanistic interpretability, and we want to encourage a diversity of attempts and research to tackle alignment and safety problems.
We wanted to encourage people interested in working on technical AI safety to apply to us with proposals for projects in areas of empirical AI safety other than interpretability. To be clear, we are still excited about receiving mechanistic interpretability applications in the future, including from junior applicants. Even with a higher bar for approval, we are still excited about funding great grants.
We tentatively plan on publishing a more detailed explanation about the reasoning later, as well as suggestions or a Request for Proposals for other promising research directions. However, these things often take longer than we expect/intend (and may not end up happening), so I wanted to give potential applicants a heads-up.
(x-posted from LW)
1. ^
  Operationalized as “assuming similar levels of funding in 2024 as in 2023, I expect that about 80-85% of the mech interp projects we funded in 2023 will be above the 2024 bar.
What links here?
- Digest #175 [normal] by tobytrem (7 Feb 2024 14:11 UTC; 6 points)

Linch 12 May 2022 19:24 UTC
79 points
0 ∶ 0
on: Bad Omens in Current Community Building
Thanks for this post. If true, it does describe a pretty serious concern.
One issue I’ve always had with the “highly engaged EA” metric is that it’s only a measure for alignment,* but the people who are most impactful within EA have both high alignment and high competence. If your recruitment selects only on alignment this suggests we’re at best neutral to competence and at worst (as this post describes) actively selecting against competence.
(I do think the elite university setting mitigates this harm somewhat, e.g. 25th percentile MIT students still aren’t stupid in absolute terms).
That said, I think the student group organizers I recently talked to are usually extremely aware of this distinction. (I’ve talked to a subset of student group organizers from Stanford, MIT, Harvard (though less granularity), UPenn (only one) and Columbia, in case this is helpful). And they tend to operationalize their targets more in terms of people who do good EA research, jobs, and exciting entrepreneurship projects, rather than in terms of just engagement/identification. Though I could be wrong about what they care about in general (as opposed to just when talking with me).
The pet theory I have for why people focus on “Highly Engaged EAs” as the metric most openly talked about as opposed to (e.g.) “Highly Engaged and Very Competent EAs” is that it looks more fair/egalitarian (at least in theory, anybody can become highly engaged, but very few people can become as smart as X or as good at operations as Y).
Some quick questions:

1. Can you define what you mean in your article by “Pascal’s Mugging?” I think you’re not using the formal definition but as a metonym for something else (which is totally fine, but I’m not sure what idea or cluster of ideas you’re exactly pointing to that’s more specific than “vaguely shady”).
2. Your critiques seem to mostly come from the LessWrong/rationality cluster. Do you have a sense of whether these critiques are shared (though expressed in different cultural languages) from people with good epistemics but from other cultural clusters?
*and of course, like all metrics it’s an imperfect measure for alignment, and can be gamed.
What links here?
- We need more discussion and clarity on how university groups create value by Oscar Galvin (28 Jun 2022 15:56 UTC; 35 points)

Linch 7 Jun 2021 21:51 UTC
74 points
1 ∶ 0
on: My current impressions on career choice for longtermists
I think this is a really well-written piece, and personally I’ve shared it with my interns and in general tentatively think I am more inclined to share this with my close contacts than most 80k articles for “generic longtermist EA career advice” (though obviously many 80k articles /podcasts have very useful details about specific questions).
2 things that I’m specifically confused about:
1. As Max_Daniel noted, an underlying theme in this post is that “being successful at conventional metrics” is an important desiderata, but this doesn’t reflect the experiences of longtermist EAs I personally know. For example, anecdotally, >60% of longtermists with top-N PhDs regret completing their program, and >80% of longtermists with MDs regret it.
  
  (Possible ways that my anecdata is consistent with your claims:
  - These people are often in the “Conceptual and empirical research on core longtermist topics” aptitudes camp, and success at conventional metrics is a weaker signal here than in other domains you listed.
  - Your notions of “success”/excellence are a much higher bar than completing a PhD at a top-N school.
  - My friends are wrong to think that getting a PhD/MD was a mistake.
2. You mention that a crazy amount of total hours is necessary to become world-class excellent at things. I agree with the sentiment that a) fit/talent is very important and b) college and other “normal/default” practices acclimate people to wrongly believing that success is easier and hard work is less critical than is true. But when I think about things that matter for longtermist EAs (rather than success at well-defined prestige ladders in fields with established paradigms), I think a lot of outlier success comes from extremely (arguably unsustainably) intense periods with relatively small calendar time or total hours invested. eg
  - A lot of success in early cryptocurrency trading (pre-2019, say) comes from people a) very talented b) willing to see the opportunity and c) be willing to make radical life changes to realize the once-in-a-decade event and immediately jump on it.
  - This seems to have happened a bunch during the pandemic. Eg amateur short-term forecasts, Youyang Gu’s modeling, patio11′s work with VaccinateCA, etc, all seemed to have been broadly better in most cases than similar work by established experts.
    Obviously there are important exceptions like the mRNA vaccines and a lot of the testing/sequencing work.
  - My impression (from outside the field) is that a lot of the most important work in AI Safety is done by people who are fairly junior, and without a lot of experience in the field.
  - I imagine “crunch time” for EAs in longtermist causes to look a lot more like “be a generically competent person who has your shit together + is willing to drop everything to work on the hard things that aren’t really your specialty but somebody has to do it and nobody else well” than “prepare for 10-20 years working very hard for the exact thing you prepared for, and then emergency times will look pretty close to your specialty so you’re well-placed.”
    Perhaps a crux here is that either your mental image of “crunch time” looks more like the latter scenario?
    or maybe more that “crunch time” is just much less important relatively speaking?
  - A caveat here is that I do agree that a) excellence is very important and b) many EAs (myself included) are perhaps not working hard enough to achieve true excellence.
  - I also agree that a) general excellence and b) hard work specifically is somewhat transferable (eg many of the successful crypto people were great finance traders or great programmers before crypto trading, at least one of which requires insane hours, patio11 was a world-class software evangelizer before his covid work). But I think the importance of being world-class here is “just” building a) the general skillset of becoming world-class and b) the mental fortitude, flexibility, etc of willingness to sacrifice other things when the stakes are high enough, rather than either the direct benefits of your expertise or the network advantages of being around other prestigious/high-status/etc. people.
    One way in which our models cash out to different actions:
    If my intuition/heuristic is correct, this points to sometimes doing crunch-time work in less important eras as being the right way to prepare, rather than steadily climbing towards excellence in very competitive domains.
    Being in “crunch mode” all the time may be actively bad, to the extent that it makes you miss out on great opportunities because you’re too zoned into your specific work.
    On the other hand, if we assume most of the benefits of excellence comes from the networking, etc, benefits of steady excellence, this points much more towards “spend 5-20 years becoming world-class at something that society thinks of as hard and important.”
An obvious caveat to these points is that you have much more experience with excellence than I do, and your “closing thoughts on advice” aside, I’m mostly willing to defer to you if you think my heuristics/intuitions here are completely off.

Linch 22 Oct 2020 6:56 UTC
73 points
0 ∶ 0
on: Progress Open Thread: October // Student Summit 2020
I started working at Rethink Priorities. I’m pretty happy about it!
I tried applying to EA jobs about 3 years ago, and didn’t have much luck.
I don’t think this is causal, but I think my emotional/cognitive relationship to applying to EA jobs have changed. 3 years ago, I was more like “I want to have a job at an EA org because that was a path I heard that can have more impact, so I really want to have an EA job so I can have more impact” and was definitely much more in “job application mode.” This time around, it felt much more like “okay there are some stuff I want to do. It seems like these orgs will let me progress my goals better.” All-in-all, I think this was probably a much healthier relationship to have to my work and EA aspirations.

Linch 28 Aug 2023 9:44 UTC
69 points
21 ∶ 2
in reply to: Lizka’s comment on: Eliezer Yudkowsky Is Frequently, Confidently, Egregiously Wrong
I agree that much of the language is inflammatory, and this is blameworthy. I disagree that the connection to EA and doing good better is unclear, conditional upon the writer being substantively correct. And historically, the personal blogpost/frontpage distinction has not been contingent on correctness. (But I understand you’re operating under pretty difficult tradeoffs, need to move fast, etc, so wording might not be exact).
What links here?
- Grumpy Squid's comment on Grumpy Squid’s Quick takes by Grumpy Squid (5 Sep 2023 21:35 UTC; 24 points)
- Linch's comment on Grumpy Squid’s Quick takes by Grumpy Squid (6 Sep 2023 0:24 UTC; 5 points)

Linch 7 Jul 2023 2:51 UTC
68 points
10 ∶ 0
on: Linch’s Shortform
Has anybody modeled or written about the potential in the future to directly translate capital into intellectual work, namely by paying for compute so that automated scientists can solve EA-relevant intellectual problems (eg technical alignment)? And the relevant implications to the “spend now vs spend later” debate?

I’ve heard this talked about in casual conversations, but never seriously discussed formally, and I haven’t seen models.

To me, this is one of the strongest arguments against spending a lot of money on longtermist/x-risk projects now. I normally am on the side of “we should spend larger sums now rather than hoard it.” But if we believe capital can one day be translated to intellectual labor at substantially cheaper rates than we can currently buy from messy human researchers now, then it’d be irrational to spend $$s on human labor instead of conserving the capital.
Note that this does not apply if:
- we are considering intellectual labor that needs to be done now rather than later
  - work that needs serial time can’t be automated quickly
    eg physical experiments
    eg building up political/coalitional support
    eg work needed to set up the initial conditions for automated intellectual labor to make it not automatically deceptively misaligned
  - projects that are needed to save the world before automated intellectual labor
    all the capital laying about is useless if we die from nuclear war or an errant virus first
  - possibly some types of cause prioritization
    for example trying to identify which projects are in the above two categories
  - Maybe field-building?
    But it’s not clear to me how much you need to do field-building if our end-game is getting machines rather than human ML scientists to do the hard work.
- you are not optimistic that there’ll be a significant period of time after intellectual labor is automated/automatable and before humans no longer control history
- you are not optimistic that we can trust machines to do alignment work, or other forms of significant morally meaningful intellectual labor
- you think worlds where we can get meaningful intellectual work just by spending capital are “saved by default” so saving EA money for them is irrelevant
  - eg because the labs, and governments, should be willing to pay much larger sums than EAs have access to do technical alignment if the work is as easy and as legible as “just pay machines to do alignment”
- you think the world is almost certainly doomed anyway, so we might as well make the world better for poor people and/or chickens in the meantime.
What links here?
- Linch's comment on There should be more AI safety orgs by mariushobbhahn (22 Sep 2023 5:22 UTC; 32 points)

Linch 13 May 2022 19:33 UTC
64 points
0 ∶ 0
on: EA and the current funding situation
I know that they very much are not doing it for the gratitude, but I still want to express my thanks to Sam and the other core members of Alameda and FTX, as well as others earning-to-give, for making the current funding situation possible. I think earning-to-give has been downplayed in recent years in EA, and it seems like a thankless job to work crazy hours and take crazy risks for many years, doing frequently weird, uninteresting, or frustrating work, just to donate it in service of making the future better.
My own organization and I’m sure that of many others in this movement have and will benefit greatly from this influx of funding. But as the post says, this is a huge responsibility and we should not bear it lightly.
I hope that the large donors in EA will continue to be responsible stewards of the resources they’ve gathered. I hope too, that the rest of us in the EA community will become strong enough to execute well on the most important priorities, to build a safe and flourishing future for our grandchildren’s grandchildren.
What links here?
- Linch's comment on Some unfun lessons I learned as a junior grantmaker by Linch (23 May 2022 19:05 UTC; 20 points)

Linch 3 Jun 2021 7:10 UTC
63 points
0 ∶ 0
on: Help me find the crux between EA/XR and Progress Studies
- How does XR weigh costs and benefits?
  - I don’t think there’s a uniform “existential risk reduction movement” answer to what % of X-risk reduction people are willing to trade for large harms today, any more than there’s a uniformly “effective altruism movement” answer for what the limits of altruism ought to be. For me personally, all the realistic tradeoffs I make day to day point towards reducing X-risk being of overwhelming importance, aside from model uncertainty.
    In practice, this means I try to spend my time thinking about longtermism stuff, with mixed success, aside from a) stuff that builds my own skills or other ways that I can be more effective, b) small cooperative actions that I think are unusually good to values systems otherwise fairly similar to my own, c) entertainment, and d) bullshit.
    I think probability of existential risk is high enough that I’m not in practice too worried about Pascal’s mugging issues. In the moments I doubt whether EA longtermism is stuff I ought to be working on, I’m much more worried about issues in the class of “randomly or systematically deluding myself about the probabilities of specific outcomes I’m worried about.” Rather than issues akin to “my entire decision procedure is wrong because expected value calculations are dumb.”
- Does XR consider tech progress default-good or default-bad?
  - Basically my view is similar to what JP said. From an XR perspective, the sign of faster tech is pretty hard to tell. There are some weak arguments in favor of it being systematically good (eg that fast technological progress reduces natural risks over the long run, while not being systematically one direction or another for man-made risks). But this is pretty weak if we think the base rate of natural risks is very low. But of course I don’t just have an XR perspective, and from other perspectives (as JP mentions) the benefits of technological and economic progress are clearer.
    As an aside, I think trying to shoehorn in “belief in progress” from an xrisk perspective is kinda dubious. Analogously, I think it’d be dumb to sell kids on the value of learning music because of purported benefits to transfer learning to mathematics standardized tests. Sure there may or may not be some cognitive benefits to learning music, but that’s not why I got into music, music is hardly the most efficient way to learn mathematics, and it’d be dubious if you primarily got into music to boost your math test scores.
- What would moral/social progress actually look like?
  - I think a lot of the types of progress that some subset of EAs (including myself) are interested in is “specific technical or organizational solutions to risks that we think of as around the horizon.” In terms of high-level moral/social progress, this is something we don’t fully understand and still trying to have better clarity of.
    “it comes across as if tech progress is on indefinite hold until we somehow become better people and thus have sufficiently reduced XR” I think this is assigning more impact/agency/credit/blame to either XR or EAs than is reasonable. The rest of the world is big, and many people want to (often for selfish or narrowly altruistic reasons) indirectly work on economic progress to better their own lives, or that of people close to them. Some people like yourself also want to more broadly increase economic progress for more global reasons.
- What does XR think about the large numbers of people who don’t appreciate progress, or actively oppose it?
  - In practice I don’t think much about them.
  - I think they’re probably wrong about this but they’re probably wrong about many other things too (as am I to be clear!), and I personally feel like (possibly overly arrogantly) that I understand their position well enough to reject them, or at least have made enough attempts at communication that further attempts aren’t very useful.
  - Note that this is just a personal take, reasonable people can disagree about how valuable this type of engagement is either in general or in this specific case, and I can also imagine (tho I think it’s unlikely) that people much more charitable and/or perceptive and/or diplomatic than myself can gain a lot value from those conversations.
- Re the “highway of progress”/road trip metaphor:
  - I like the idea of a highway of progress.
  - A key distinction to me is whether you think of the highway as more of a road trip or more of a move (less metaphorically, whether the purpose is more to enjoy the journey or to go from Point A to Point B).
  - To extend and hopefully not butcher the metaphor too much, my stereotype of progress studies people is that they treat the highway of progress as pretty much like a road trip. That is, you have limited time and it’s really important to have a balance of a) an appropriate fast/steady pace along this highway of progress, along with b) enjoying yourself while you’re at it. Sure, it’s important to avoid unlikely catastrophes like driving off a cliff, but those things are unusual and the faster you go the more sights you see, etc. So the usual balance of considerations is something like consuming now (stopping and smelling the roses, in this metaphor) vs doing more stuff so we and our near descendants can enjoy more stuff later (traveling faster to see more sights, in this metaphor).
  - My own view** of the highway of progress is that we’re actually trying to go somewhere, perhaps with the intention of settling there forever. We’re leaving our unpleasant homeland to travel, along a dangerous road, to somewhere glorious and wonderful, perhaps a location we previously identified as potentially great (like New York), perhaps a nice suburb somewhere along the road.
    (There are nicer and less nice versions of this metaphor. Perhaps we’re refugees from a wartorn country. Perhaps we’re new college graduates convinced that we can find a better life outside of our provincial hometowns).
  - So to many longtermist EAs, the destination matters much more than the journey.
  - In this regard, I think the story/framing where “humanity is trying to take an exciting but dangerous journey to lands known and unknown” (ie, trying to reach utopia) makes some sense as a story where exquisite care should be made.
    Ultimately you have a lifetime to go from point A to point B, but you want to be very careful not to make rash irrevocable moves that means your journey ends prematurely.
    existential risks can come from death (extinction risk) or from other ways that prematurely ends the journey
    in the travel metaphor, you can be stuck in a suboptimal town and fool yourself into this being a great place to live
    in our world, this can be dystopias or astronomical suffering, or just trapped in bad values
    Notable that there were many past (admittedly non-existential) failures from people attempting to reach utopia
  - if you see the highway of progress as a road trip:
    haste is really important (limited vacation time!)
    you’d be a dumbass to be so careful that you barely visit anywhere.
    journey matters more than destination
    not reaching final destination totally fine.
    safety is important but not critical
  - if you see the highway as a mode of one-way transit
    fine to take your time
    have a lifetime to get there
    when you get there much less important than getting there at all.
    obviously you’d still want to get there eventually, but since the trade is something on the order of 1% of astronomical waste every 10 million years delay, not too important how fast you are.
- I haven’t heard of a satisfying explanation from Progress Studies folks about why urgency of economic growth is so important, at least for people who a) think there are percentage point probabilities of existential risk and b) agree with zero intrinsic discount rate.
  - Possible explanations:
    There aren’t percentage points of existential risk,
    alternatively, the net probability that” dedicated effort from humanity averting existential risks” can actually reduce xrisks is <<1%.
    As a special case of the above, probability of all of humanity dying in the next ~1000 years approach 1 (cf Tyler Cowen, H/T Applied Divinity Studies)
    This is an argument for working on medium-term economic progress over working on existential risk not because the risk is too low, but it’s too (unfixably) high.
    the zero intrinsic discount rate is morally or empirically mistaken
    temporary stagnation either inevitably or with high probability lead to permanent stagnation (so it is itself an existential risk).
    Some combination of values pluralism + comparative advantage arguments, such that it makes some sense for some individuals to work on progress studies even if it is overall less overwhelmingly important than xrisk.
    I find this very plausible but my general anecdotal feeling from you and others in this circle is that you’re usually making much stronger claims.
    Something else
  - Clarifying which position individuals believe may be helpful here.
** to be clear, you don’t have to have this view to be a longtermist or an EA, but I do think this is much more common among the modal longtermist EA than the modal Progress studies fan.
What links here?
- EA Forum Prize: Winners for May-July 2021 by Aaron Gertler (20 Oct 2021 11:17 UTC; 38 points)

Linch 7 Sep 2021 9:16 UTC
61 points
0 ∶ 0
in reply to: Abby Hoskin’s comment on: A Primer on the Symmetry Theory of Valence
Hi, all. Talk is cheap, and EA Forum karma may be insufficiently nuanced to convey substantive disagreements.

I’ve taken the liberty to sketch out several forecasting questions that might reflect underlying differences in opinion. Interested parties may wish to forecast on them (which the EA Forum should allow you to do directly, at least on desktop) and then make bets accordingly.
Feel free to also counterpropose (and make!) other questions if you think the existing question operationalizations are not sufficient (I’m far from knowledgeable in this field!).
By EOY 2026, will a single neuroscience paper by Mike Johnson or Andrés Emilsson have more than 100 citations?
By EOY 2031, will there be more than 3000 citations by academic researchers of articles by QRI or direct affiliates on the Symmetry Theory of Valence?
Will QRI exist and have a >250k/year budget in 2031?
In 2031, will Luke Muelhauser say statements to the effect that the Symmetry Theory of Valence is substantively true?