This is part of a weekly series—you can see the full collection here. The first post includes some details on purpose and methodology.
If you’d like to receive these summaries via email, you can subscribe here.
Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for ‘Effective Altruism Forum Podcast’.
Top / Curated Readings
Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all.
The EA Survey is an annual survey of the effective altruism community, and is now live at this link until Dec 1st. The results inform decisions for a number of different orgs and help to understand how the community is changing over time. This year’s survey is substantially shorter than 2021, and should take <10 minutes. If you’re reading this, you fit the target audience and your response is valuable. Sharing widely with other EAs is also appreciated.
The second of the two posts aiming to make discussion of how we grow the EA movement more concrete. EA is on a BAU trajectory, built up of small intentional decisions (eg. rough growth rate, name & branding), nudges from community leaders, and natural social dynamics. This post analyzes different domains where we could change trajectory, and key considerations for assessing these—without advocating any particular approach.
The domains identified group into:
How much EA should stay as a single entity and identity vs. splitting branding / outreach / identity into philosophical or cause-specific areas.
How EA mixes social and professional, and how much it interacts with outside communities and professional networks.
Growth rate, level of demands of members, and where growth is targeted (eg. do we continue the focus on top universities in high-income countries, or diversify?)
Level of centralization of funding and decision making.
These could affect decisions on location of hubs, onboarding content, target audience of projects or events, branding, funding choices and much more.
Considerations on many uncertain questions such as TAI timelines, costs vs. benefits of insulation, who EA needs to attract etc. could influence what course changes are best. The author suggests next steps including to think through EA success cases, analyze the tractability of different changes, run small-scale experiments, and begin arguing for and against concrete changes.
From Dec 1st, all new content posted to the Forum will be available under a CC BY 4.0 license. This means you can share, adapt and distribute the material for any purpose—including commercial ones. However you must credit the original author in a way that does not imply they endorse your changes, and cannot restrict others from using it.
The very repugnant conclusion is the argument that enough lives ‘barely worth living’, even if they’re bundled with a set of really negative lives, are better than a much smaller set of super duper awesome lives. This is seen as repugnant because trivial benefits outweigh extreme happiness and pain purely due to quantity.
Minimalist axiologies are those which say ‘The less X, the better’. If we take X to be suffering, this avoids the conclusion above, as the suffering is always enough to outweigh the barely worth it lives. The author also talks about variants that eg. have sharp or gradient cut-offs on what suffering is ‘bad enough’ that no happiness can outweigh it. There are still some variants of the repugnant conclusion that apply in this case, but the author argues they are the least repugnant ones.
Givewell uses a discount rate of 4% for developmental effects (eg. increases in consumption) in all their cost-effectiveness analyses (CEAs). The author argues for instead using a probability distribution over possible discount rates. This would notably change recommendations eg. making deworming comparatively more attractive, because the multi-decadal effects of deworming means a large amount of the impact is in scenarios with low discount rates.
GiveWell’s discount rate of 4% includes a 1.4% contribution from ‘temporal uncertainty’ arising from the possibility of major events radically changing the world. The author argues given the AI timelines of experts in the area (eg. Ajeya’s 50% credence of TAI by 2040, or AI Impacts median expert timeline of 2059), that the contribution of temporal uncertainty should be much higher.
The AI risk community is estimated to be spending at most 3% of its capital per year. The authors have created a tool for calculating the optimal spending schedule, and find that most scenarios give results in the 5-35% range. The model’s inputs can be varied by the user and include your AI timeline, discount rate of AGI success by year, starting money and influence, rate of diminishing returns on each, and more.
Author’s abstract: “This essay argues that the evidence supporting GiveWell’s top cause area – Seasonal Malaria Chemoprevention, or SMC – is much weaker than it appears at first glance and would benefit from high-quality replication. Specifically, GiveWell’s assertion that every $5,000 spent on SMC saves a life is a stronger claim than the literature warrants on three grounds: 1) the effect size is small and imprecisely estimated; 2) co-interventions delivered simultaneously pose a threat to external validity; and 3) the research lacks the quality markers of the replication/credibility revolution. I conclude by arguing that any replication of SMC should meet the standards of rigor and transparency set by GiveDirectly, whose evaluations clearly demonstrate contemporary best practices in open science.”
Last spring, ~$1M USD was spent by EA-aligned political action committee Protect Our Future (POF) to support the nomination of NC State Senator Valerie Foushee.
The author notes there was limited information on the rationale for this funding, and why POF supported Valerie over other candidates. Ads run also focused mainly on Valerie’s personal background. These factors could make neutral observers question the motives of funders, particularly given political spending in support of business interests is common in the US.
The author suggests more transparency in future efforts (eg. clear criteria and rationale for candidates supported) could minimize the risk of reputational damage to EA.
Linkpost for Our World in Data article on the flu. The risk of death from influenza has declined over time, but it still causes between 294,000 and 518,000 deaths each year.
~28M UK adults won’t be able to get a covid booster this winter, even if they’re willing to pay for it, because the NHS is including only the most vulnerable in its rollout of the bivalent booster. By creating a website for people to register interest in paying for a vaccine, we could convince Pfizer or similar to make them privately available (as they will be in the US from Jan 2023, and as flu vaccines are in the UK). If you’re interested in getting involved, post in the comments.
The MIT Media Lab has a new fellowship for PhD and Masters applicants funded by the FTX Future Fund, for “ambitiously benevolent projects that could not be accomplished anywhere else”. Applications are due 1st Dec, and cover tuition, health insurance, $45Kp/y stipend and $25Kp/y research costs.
The African EA Summit will be a ~30-person retreat primarily for members of the EA community in Africa who are interested in movement-building for EA or related cause areas, or for those working on projects relevant to the African context.
It runs 1 − 5th December in Cape Town. Apply by 6th November. Room, board, and travel expenses covered by CEA.
Redwood Research is running a coordinated research effort on mechanistic interpretability of transformer models. Researchers will design, run, and analyze experiments to shed light on how models accomplish specific tasks. Housing, travel, and time will be paid.
Apply by Nov 8th to join, dates flexible but primarily December / January. Applicants should be comfortable with basic Python and Linear Algebra, but don’t need any background in interpretability research. Those with experience in empirical science from any discipline are particularly encouraged.
Probably Good is a career guidance organization aiming to make impact-centered career guidance more accessible to more people. They have multiple open roles, including Operations Lead, Growth Manager, Career Advisor, Community Manager, and Web Developer. All roles are fully remote (any timezone) and most have the option of either full-time or part-time.
The Center for AI Safety is looking for benchmark ideas that motivate or advance research that reduces existential risks from AI. Prizes are 50-100K per winning idea, with a total prize pool of 100-500K depending on submission quality.
Founders Pledge has launched a new fund designed to offer high-impact giving opportunities for both longtermists and non-longtermists who care about catastrophic risks. It has a special focus on international stability, and also tackles nuclear war, AI risk, bio risk, and risks from emerging tech.
by Esben Kran, Sabrina Zaki, RichardAnnilo, Joe Hardie, Apart Research
Author’s tl;dr: Participate online or in-person in London, Aarhus, and Tallinn on the weekend 11th to 13th November in a fun and intense AI safety research hackathon focused on interpretability research. We invite mid-career professionals to join but it is open for everyone (also no-coders) and we will create starter code templates to help you kickstart your team’s projects. Join here.
The EA Survey is an annual survey of the effective altruism community, and is now live at this link until Dec 1st. The results inform decisions for a number of different orgs and help to understand how the community is changing over time. This year’s survey is substantially shorter than 2021, and should take <10 minutes. If you’re reading this, you fit the target audience and your response is valuable. Sharing widely with other EAs is also appreciated.
Lord Martin Rees is one of the world’s most eminent scientists, and an established advocate for existential risk reduction. He’s published several books on the topic, including ‘Our Final Century’ published in 2003 and ‘If Science is to Save Us’ published just recently. His Ted Talks on these have millions of views. He was also involved in the Pugwash conferences of the 1970s, following on from the first existential risk reduction efforts in the 1950s by scientists who tabooed the use of nuclear weapons.
Given all this, we should turn to him more often as a respected advocate for reducing existential risk, and include reference to his work in introductory articles, books, and similar.
The first of two posts aiming to make the discussion on how we should grow EA more concrete. This one focuses on ways the EA movement could fail ie. have substantially less impact than it would otherwise:
Reputation failures eg. media paints us in bad light, internal dynamics like elitism put people off, major projects goes badly, politicization, infighting, scandals
Resource failures eg. running out of money or talent, inadequate infrastructure
Rigor failures eg. funding and status-seeking behavior, diluted truth-seeking norms, fracture into siloed cause areas, excessive deference, nepotism, echo chambers, poor feedback loops, isolated leadership, lack of diversity of thought
Reservation failures ie. leaving impact on the table by being too risk-averse. Eg. projects too small, deny impactful ones on reputation grounds, overlook short AI timelines, wait too long to engage powerful entities like the UN, spending averse, bureaucracy
More work to identify relative likelihoods, severities, warning signs and mitigations of these risks would be valuable.
The second of the two posts aiming to make discussion of how we grow the EA movement more concrete. EA is on a BAU trajectory, built up of small intentional decisions (eg. rough growth rate, name & branding), nudges from community leaders, and natural social dynamics. This post analyzes different domains where we could change trajectory, and key considerations for assessing these—without advocating any particular approach.
The domains identified group into:
How much EA should stay as a single entity and identity vs. splitting branding / outreach / identity into philosophical or cause-specific areas.
How EA mixes social and professional, and how much it interacts with outside communities and professional networks.
Growth rate, level of demands of members, and where growth is targeted (eg. do we continue the focus on top universities in high-income countries, or diversify?)
Level of centralization of funding and decision making.
These could affect decisions on location of hubs, onboarding content, target audience of projects or events, branding, funding choices and much more.
Considerations on many uncertain questions such as TAI timelines, costs vs. benefits of insulation, who EA needs to attract etc. could influence what course changes are best. The author suggests next steps including to think through EA success cases, analyze the tractability of different changes, run small-scale experiments, and begin arguing for and against concrete changes.
Vox’s Future Perfect includes a new top 50 list of “The scientists, thinkers, scholars, writers, and activists building a more perfect future”—with multiple EAs included.
by jessica_mccurdy, Jesse Rothman, Joris P, Jake McKinnon
CEA’s University Group Accelerator Program (UGAP) had limited capacity, so they experimented with a virtual two week starter program open to a larger group of schools. 81 universities participated, of which 56 were also accepted to UGAP.
Feedback was decently positive, with mentor meetings and resource guides called out as most helpful. The organizers took several learnings from the experience:
Part of the idea was to test fit for UGAP with the starter program, but it ended up encouraging groups who weren’t ready to start (ie. organizers needed to learn more about EA first). In future they’d make the entry bar the same for each program.
Their advice is well-targeted to those starting new groups, but less useful for groups in advanced stages. In future they’re planning to target just the former.
Onboarding new team members during program delivery is difficult and can lead to suboptimal experiences for the group organizers.
The author has just published their new book on s-risks, Avoiding the Worst: How to Prevent a Moral Catastrophe. You can find it on Amazon or read the PDF version. It covers what s-risks are, why we should focus on them, and how we can reduce them.
New episode of the author’s podcast The Most Interesting People I Know centers on the relationship between EA and left-wing politics, in an interview with Habiba Islam (an 80K careers advisor).
The EA forum has grown rapidly, and September hit records in every metric (users, views, posts etc.) To help handle the larger amount of content, the forum team has:
Implemented a new search UI (including allowing excluding terms and filtering on exact phrases)
From Dec 1st, all new content posted to the Forum will be available under a CC BY 4.0 license. This means you can share, adapt and distribute the material for any purpose—including commercial ones. However you must credit the original author in a way that does not imply they endorse your changes, and cannot restrict others from using it.
If you work in operations at an EA-aligned organization, apply to the slack space here. It’s used for the sharing of operations-related insights and advice in areas including HR, recruitment, accounting, event planning, personal assistance and office management.
PAs tend to fall into two categories—personal assistants (few hours, often remote, primarily person life admin) and executive assistants (often full time, help with professional work, mix of admin and generalist). For both these cases, Pineapple Operation has published this guide which covers:
Value (Pineapple Ops estimates 10-100% of the value of the person they’re supporting)
Pay (often £30k to £77k in EA)
Required attributes (support mindset, organised, good communication etc.)
They also provide advice on how to find a role (or find a PA), legal considerations with setting that up, and practices that can help make it successful. Most of the latter advice centers around clear and efficient communication, and understanding of each other’s working styles.
Author’s summary (lightly edited): “Response to an Oct 2022 MIT Technology Review editorial entitled “Inside effective altruism, where the far future counts a lot more than the present.” I believe it was written in good faith, however by my count there are 12 false and 22 misleading claims. Each is explained with sources in annotations to the original text.”
The author is starting a series of magazine-style profiles of people doing interesting EA work, called “Humans of Effective Altruism.” They’re looking for recommendations (including self-recs) of people doing high impact work, with interesting life stories, who are willing to be profiled. The results will be targeted at both EAs and non-EAs.
Concrete questions can spark more interest in EA principles than abstract arguments. The author has found students in their college classes respond well to questions like “If you had all of Superman’s superpowers for 24 hours, and you wanted to do the most good in the world during that one day, what would you do?” The facilitator can then nudge the discussion towards topics like scope sensitivity or comparing moral frameworks, as these come up via participants differing answers.
Community retreats are aimed at people making friends—so vibes matter. Some things that help:
Light & flexible schedule, with few but high quality events
Creating comfort—minimizing latecomers / orienting them if you can’t, checking in, sharing attendee info pre-retreat, vulnerability promoting activities eg. hamming circles.
Facilitate group bonding via eg. physical games, everyone participates activities.
Encourage intentionality via eg. communicating retreat goals, the main organizer being very present (someone else handles Ops needs).
Get an easily accessed venue that can accommodate different social spaces (eg. 1-1s), 24⁄7 access to a comfy shared space, and amenities like snacks, toiletries, chargers. Doesn’t have to be fancy—participants can share rooms or do chores.
The author argues that even if it’s possible to create both ‘good’ and ‘bad’ AGIs, and a good one is the first created, we’re still unlikely to be able to use it to prevent or defend against the bad ones. This is because the good AGI is likely to be more limited than the bad one eg.
1. Offense is inherently different than defense (eg. disabling an electric grid could be easier than protecting it).
2. Humans might not trust a good AGI eg. we might limit it’s self improvement or access to resources, or slow it down so we can interpret it better. A bad AGI doesn’t have these limits.
3. The good AGI may be hamstrung by human norms, laws, overton windows, mass agreement etc. - while again the bad AGI can skip past this.
They also address possible things that could help with these issues and why they think these are unlikely to make a difference.
Aims to make it easier for people to field-switch by presenting a list of barebones prerequisites required to understand, write code and ideally contribute useful research to mechanistic interpretability. This includes some math (basics of linear algebra, probability, calculus), Python basics, and a rough grounding in ML with focus on transformers.
Summary of a 40-page document arguing that AGI makes it possible to perfectly preserve nuanced specifications of a wide variety of values or goals far into the future, and develop AGI-based institutions to pursue these for millions or trillions of years. These could be good or bad lock-ins.
The three arguments given are: 1. Better preservation of information (eg. whole brain emulation to allow future AGIs to ask what humans today would have thought).
2. AIs can pursue a set of values single-mindedly vs. being distracted by biological goals like humans.
3. Institutions could ‘reboot’ AIs that showed value drift or halt progress if decisions became ambiguous due to the changing environment.
aisafety.community—A living document of AI safety communities
by zeshen, plex
Volunteers are maintaining this live database of AI safety communities. Please update missing communities or improve descriptions as you use it. They’re also looking for more volunteer maintainers.
by VipulNaik A trigger-based rapid checklist is a memorized checklist that you execute rapidly in response to a specific trigger. Eg. The trigger could be turning on your laptop in the morning, and the checklist could include items like ‘check weather’ and ‘catch up on email’. The author has found this useful for ensuring they’re on top of messages / email, recording things regularly, and finishing the day in a ‘clean state’ (laptop closed, backups done, no open tabs etc.)
In the USA, there are more child pedestrian fatalities on Halloween than any other day. But there are also more children walking around on Halloween, so given that base rate it might actually be one of the safest days.
Ahead of the publication of the white paper on AI governance expected later this year,
@CommonsSTC, (a cross-party group of UK MPs) has launched a new inquiry to “examine the effectiveness of AI governance and the UK Government’s proposals.” The call for evidence is open until 25 November. (tweet)
Forecasting
Infer are a group doing forecasts for stakeholders in US government on topics like critical science and tech issues, and high risk geopolitical events. They’re looking for more forecasters. (website)
EA & LW Forums Weekly Summary (24 − 30th Oct 22′)
Supported by Rethink Priorities
This is part of a weekly series—you can see the full collection here. The first post includes some details on purpose and methodology.
If you’d like to receive these summaries via email, you can subscribe here.
Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for ‘Effective Altruism Forum Podcast’.
Top / Curated Readings
Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all.
Announcing EA Survey 2022
by David_Moss, WillemSleegers
The EA Survey is an annual survey of the effective altruism community, and is now live at this link until Dec 1st. The results inform decisions for a number of different orgs and help to understand how the community is changing over time. This year’s survey is substantially shorter than 2021, and should take <10 minutes. If you’re reading this, you fit the target audience and your response is valuable. Sharing widely with other EAs is also appreciated.
EA movement course corrections and where you might disagree
by michel
The second of the two posts aiming to make discussion of how we grow the EA movement more concrete. EA is on a BAU trajectory, built up of small intentional decisions (eg. rough growth rate, name & branding), nudges from community leaders, and natural social dynamics. This post analyzes different domains where we could change trajectory, and key considerations for assessing these—without advocating any particular approach.
The domains identified group into:
How much EA should stay as a single entity and identity vs. splitting branding / outreach / identity into philosophical or cause-specific areas.
How EA mixes social and professional, and how much it interacts with outside communities and professional networks.
Growth rate, level of demands of members, and where growth is targeted (eg. do we continue the focus on top universities in high-income countries, or diversify?)
Level of centralization of funding and decision making.
These could affect decisions on location of hubs, onboarding content, target audience of projects or events, branding, funding choices and much more.
Considerations on many uncertain questions such as TAI timelines, costs vs. benefits of insulation, who EA needs to attract etc. could influence what course changes are best. The author suggests next steps including to think through EA success cases, analyze the tractability of different changes, run small-scale experiments, and begin arguing for and against concrete changes.
New Forum License: Creative Commons
by Ben_West
From Dec 1st, all new content posted to the Forum will be available under a CC BY 4.0 license. This means you can share, adapt and distribute the material for any purpose—including commercial ones. However you must credit the original author in a way that does not imply they endorse your changes, and cannot restrict others from using it.
EA Forum
Philosophy and Methodologies
Minimalist extended very repugnant conclusions are the least repugnant
by Teo Ajantaival
The very repugnant conclusion is the argument that enough lives ‘barely worth living’, even if they’re bundled with a set of really negative lives, are better than a much smaller set of super duper awesome lives. This is seen as repugnant because trivial benefits outweigh extreme happiness and pain purely due to quantity.
Minimalist axiologies are those which say ‘The less X, the better’. If we take X to be suffering, this avoids the conclusion above, as the suffering is always enough to outweigh the barely worth it lives. The author also talks about variants that eg. have sharp or gradient cut-offs on what suffering is ‘bad enough’ that no happiness can outweigh it. There are still some variants of the repugnant conclusion that apply in this case, but the author argues they are the least repugnant ones.
GiveWell Misuses Discount Rates
by Oscar Delaney
Givewell uses a discount rate of 4% for developmental effects (eg. increases in consumption) in all their cost-effectiveness analyses (CEAs). The author argues for instead using a probability distribution over possible discount rates. This would notably change recommendations eg. making deworming comparatively more attractive, because the multi-decadal effects of deworming means a large amount of the impact is in scenarios with low discount rates.
GiveWell should use shorter TAI timelines
by Oscar Delaney
GiveWell’s discount rate of 4% includes a 1.4% contribution from ‘temporal uncertainty’ arising from the possibility of major events radically changing the world. The author argues given the AI timelines of experts in the area (eg. Ajeya’s 50% credence of TAI by 2040, or AI Impacts median expert timeline of 2059), that the contribution of temporal uncertainty should be much higher.
Object Level Interventions / Reviews
The optimal timing of spending on AGI safety work; why we should probably be spending more now
by Tristan Cook, Guillaume Corlouer
The AI risk community is estimated to be spending at most 3% of its capital per year. The authors have created a tool for calculating the optimal spending schedule, and find that most scenarios give results in the 5-35% range. The model’s inputs can be varied by the user and include your AI timeline, discount rate of AGI success by year, starting money and influence, rate of diminishing returns on each, and more.
GiveWell should fund an SMC replication
by Seth Ariel Green
Author’s abstract: “This essay argues that the evidence supporting GiveWell’s top cause area – Seasonal Malaria Chemoprevention, or SMC – is much weaker than it appears at first glance and would benefit from high-quality replication. Specifically, GiveWell’s assertion that every $5,000 spent on SMC saves a life is a stronger claim than the literature warrants on three grounds: 1) the effect size is small and imprecisely estimated; 2) co-interventions delivered simultaneously pose a threat to external validity; and 3) the research lacks the quality markers of the replication/credibility revolution. I conclude by arguing that any replication of SMC should meet the standards of rigor and transparency set by GiveDirectly, whose evaluations clearly demonstrate contemporary best practices in open science.”
EA-Aligned Political Activity in a US Congressional Primary: Concerns and Proposed Changes
by Carolina_EA
Last spring, ~$1M USD was spent by EA-aligned political action committee Protect Our Future (POF) to support the nomination of NC State Senator Valerie Foushee.
The author notes there was limited information on the rationale for this funding, and why POF supported Valerie over other candidates. Ads run also focused mainly on Valerie’s personal background. These factors could make neutral observers question the motives of funders, particularly given political spending in support of business interests is common in the US.
The author suggests more transparency in future efforts (eg. clear criteria and rationale for candidates supported) could minimize the risk of reputational damage to EA.
How many people die from the flu? (OWID)
by Lizka
Linkpost for Our World in Data article on the flu. The risk of death from influenza has declined over time, but it still causes between 294,000 and 518,000 deaths each year.
A Potential Cheap and High Impact Way to Reduce Covid in the UK this Winter
by Lawrence Newport
~28M UK adults won’t be able to get a covid booster this winter, even if they’re willing to pay for it, because the NHS is including only the most vulnerable in its rollout of the bivalent booster. By creating a website for people to register interest in paying for a vaccine, we could convince Pfizer or similar to make them privately available (as they will be in the US from Jan 2023, and as flu vaccines are in the UK). If you’re interested in getting involved, post in the comments.
Opportunities
Consider Applying to the Future Fellowship at MIT
by Jeff Kaufman
The MIT Media Lab has a new fellowship for PhD and Masters applicants funded by the FTX Future Fund, for “ambitiously benevolent projects that could not be accomplished anywhere else”. Applications are due 1st Dec, and cover tuition, health insurance, $45Kp/y stipend and $25Kp/y research costs.
The African Movement-building Summit
by jwpieters
The African EA Summit will be a ~30-person retreat primarily for members of the EA community in Africa who are interested in movement-building for EA or related cause areas, or for those working on projects relevant to the African context.
It runs 1 − 5th December in Cape Town. Apply by 6th November. Room, board, and travel expenses covered by CEA.
Apply to the Redwood Research Mechanistic Interpretability Experiment (REMIX), a research program in Berkeley
by Max Nadeau, Xander Davies, Buck, Nate Thomas
Redwood Research is running a coordinated research effort on mechanistic interpretability of transformer models. Researchers will design, run, and analyze experiments to shed light on how models accomplish specific tasks. Housing, travel, and time will be paid.
Apply by Nov 8th to join, dates flexible but primarily December / January. Applicants should be comfortable with basic Python and Linear Algebra, but don’t need any background in interpretability research. Those with experience in empirical science from any discipline are particularly encouraged.
We’re hiring! Probably Good is expanding our team
by Probably Good
Probably Good is a career guidance organization aiming to make impact-centered career guidance more accessible to more people. They have multiple open roles, including Operations Lead, Growth Manager, Career Advisor, Community Manager, and Web Developer. All roles are fully remote (any timezone) and most have the option of either full-time or part-time.
Prizes for ML Safety Benchmark Ideas
by Joshc, Dan Hendrycks
The Center for AI Safety is looking for benchmark ideas that motivate or advance research that reduces existential risks from AI. Prizes are 50-100K per winning idea, with a total prize pool of 100-500K depending on submission quality.
Announcing the Founders Pledge Global Catastrophic Risks Fund
by christian.r
Founders Pledge has launched a new fund designed to offer high-impact giving opportunities for both longtermists and non-longtermists who care about catastrophic risks. It has a special focus on international stability, and also tackles nuclear war, AI risk, bio risk, and risks from emerging tech.
Join the interpretability research hackathon
by Esben Kran, Sabrina Zaki, RichardAnnilo, Joe Hardie, Apart Research
Author’s tl;dr: Participate online or in-person in London, Aarhus, and Tallinn on the weekend 11th to 13th November in a fun and intense AI safety research hackathon focused on interpretability research. We invite mid-career professionals to join but it is open for everyone (also no-coders) and we will create starter code templates to help you kickstart your team’s projects. Join here.
Community & Media
Announcing EA Survey 2022
by David_Moss, WillemSleegers
The EA Survey is an annual survey of the effective altruism community, and is now live at this link until Dec 1st. The results inform decisions for a number of different orgs and help to understand how the community is changing over time. This year’s survey is substantially shorter than 2021, and should take <10 minutes. If you’re reading this, you fit the target audience and your response is valuable. Sharing widely with other EAs is also appreciated.
Lord Martin Rees: an appreciation
by HaydnBelfield
Lord Martin Rees is one of the world’s most eminent scientists, and an established advocate for existential risk reduction. He’s published several books on the topic, including ‘Our Final Century’ published in 2003 and ‘If Science is to Save Us’ published just recently. His Ted Talks on these have millions of views. He was also involved in the Pugwash conferences of the 1970s, following on from the first existential risk reduction efforts in the 1950s by scientists who tabooed the use of nuclear weapons.
Given all this, we should turn to him more often as a respected advocate for reducing existential risk, and include reference to his work in introductory articles, books, and similar.
Ways in which EA could fail
by michel
The first of two posts aiming to make the discussion on how we should grow EA more concrete. This one focuses on ways the EA movement could fail ie. have substantially less impact than it would otherwise:
Reputation failures eg. media paints us in bad light, internal dynamics like elitism put people off, major projects goes badly, politicization, infighting, scandals
Resource failures eg. running out of money or talent, inadequate infrastructure
Rigor failures eg. funding and status-seeking behavior, diluted truth-seeking norms, fracture into siloed cause areas, excessive deference, nepotism, echo chambers, poor feedback loops, isolated leadership, lack of diversity of thought
Reservation failures ie. leaving impact on the table by being too risk-averse. Eg. projects too small, deny impactful ones on reputation grounds, overlook short AI timelines, wait too long to engage powerful entities like the UN, spending averse, bureaucracy
More work to identify relative likelihoods, severities, warning signs and mitigations of these risks would be valuable.
EA movement course corrections and where you might disagree
by michel
The second of the two posts aiming to make discussion of how we grow the EA movement more concrete. EA is on a BAU trajectory, built up of small intentional decisions (eg. rough growth rate, name & branding), nudges from community leaders, and natural social dynamics. This post analyzes different domains where we could change trajectory, and key considerations for assessing these—without advocating any particular approach.
The domains identified group into:
How much EA should stay as a single entity and identity vs. splitting branding / outreach / identity into philosophical or cause-specific areas.
How EA mixes social and professional, and how much it interacts with outside communities and professional networks.
Growth rate, level of demands of members, and where growth is targeted (eg. do we continue the focus on top universities in high-income countries, or diversify?)
Level of centralization of funding and decision making.
These could affect decisions on location of hubs, onboarding content, target audience of projects or events, branding, funding choices and much more.
Considerations on many uncertain questions such as TAI timelines, costs vs. benefits of insulation, who EA needs to attract etc. could influence what course changes are best. The author suggests next steps including to think through EA success cases, analyze the tractability of different changes, run small-scale experiments, and begin arguing for and against concrete changes.
The Future Perfect 50
by Catherine Low
Vox’s Future Perfect includes a new top 50 list of “The scientists, thinkers, scholars, writers, and activists building a more perfect future”—with multiple EAs included.
UGAP Starter Program Retrospective
by jessica_mccurdy, Jesse Rothman, Joris P, Jake McKinnon
CEA’s University Group Accelerator Program (UGAP) had limited capacity, so they experimented with a virtual two week starter program open to a larger group of schools. 81 universities participated, of which 56 were also accepted to UGAP.
Feedback was decently positive, with mentor meetings and resource guides called out as most helpful. The organizers took several learnings from the experience:
Part of the idea was to test fit for UGAP with the starter program, but it ended up encouraging groups who weren’t ready to start (ie. organizers needed to learn more about EA first). In future they’d make the entry bar the same for each program.
Their advice is well-targeted to those starting new groups, but less useful for groups in advanced stages. In future they’re planning to target just the former.
Onboarding new team members during program delivery is difficult and can lead to suboptimal experiences for the group organizers.
Progress Open Thread: EAGxVirtual 2022
by Alex Berezhnoi
Open thread for EAGxVirtual’s ~900 attendees to post if they learn something useful, change their plans, have a great meeting etc.
New book on s-risks
by Tobias_Baumann
The author has just published their new book on s-risks, Avoiding the Worst: How to Prevent a Moral Catastrophe. You can find it on Amazon or read the PDF version. It covers what s-risks are, why we should focus on them, and how we can reduce them.
Podcast: The Left and Effective Altruism with Habiba Islam
by Garrison
New episode of the author’s podcast The Most Interesting People I Know centers on the relationship between EA and left-wing politics, in an interview with Habiba Islam (an 80K careers advisor).
Search, subforums, and other Forum updates (October 2022)
by Lizka
The EA forum has grown rapidly, and September hit records in every metric (users, views, posts etc.) To help handle the larger amount of content, the forum team has:
Implemented a new search UI (including allowing excluding terms and filtering on exact phrases)
Launched bioethics and software engineering subforums (as pilots)
Improved site performance and reliability
New Forum License: Creative Commons
by Ben_West
From Dec 1st, all new content posted to the Forum will be available under a CC BY 4.0 license. This means you can share, adapt and distribute the material for any purpose—including commercial ones. However you must credit the original author in a way that does not imply they endorse your changes, and cannot restrict others from using it.
EA Operations Slack
by AmritSidhu-Brar, sawyer
If you work in operations at an EA-aligned organization, apply to the slack space here. It’s used for the sharing of operations-related insights and advice in areas including HR, recruitment, accounting, event planning, personal assistance and office management.
PAs in EA: A Brief Guide & FAQ
by Vaidehi Agarwalla
PAs tend to fall into two categories—personal assistants (few hours, often remote, primarily person life admin) and executive assistants (often full time, help with professional work, mix of admin and generalist). For both these cases, Pineapple Operation has published this guide which covers:
Typical tasks (admin, light research, ‘ugh’ tasks, inbox / calendar management, motivation, ad-hoc generalist tasks)
Value (Pineapple Ops estimates 10-100% of the value of the person they’re supporting)
Pay (often £30k to £77k in EA)
Required attributes (support mindset, organised, good communication etc.)
They also provide advice on how to find a role (or find a PA), legal considerations with setting that up, and practices that can help make it successful. Most of the latter advice centers around clear and efficient communication, and understanding of each other’s working styles.
[Linkpost] A Response to Rebecca Ackermann’s “Inside Effective Altruism” Editorial
by stecas
Author’s summary (lightly edited): “Response to an Oct 2022 MIT Technology Review editorial entitled “Inside effective altruism, where the far future counts a lot more than the present.” I believe it was written in good faith, however by my count there are 12 false and 22 misleading claims. Each is explained with sources in annotations to the original text.”
Recommend Me EAs To Write About
by Stephen Thomas
The author is starting a series of magazine-style profiles of people doing interesting EA work, called “Humans of Effective Altruism.” They’re looking for recommendations (including self-recs) of people doing high impact work, with interesting life stories, who are willing to be profiled. The results will be targeted at both EAs and non-EAs.
Teaching EA through superhero thought experiments
by Geoffrey Miller
Concrete questions can spark more interest in EA principles than abstract arguments. The author has found students in their college classes respond well to questions like “If you had all of Superman’s superpowers for 24 hours, and you wanted to do the most good in the world during that one day, what would you do?” The facilitator can then nudge the discussion towards topics like scope sensitivity or comparing moral frameworks, as these come up via participants differing answers.
On retreats: nail the ‘vibes’ and venue
by Vaidehi Agarwalla, Amber Dawn
Community retreats are aimed at people making friends—so vibes matter. Some things that help:
Light & flexible schedule, with few but high quality events
Creating comfort—minimizing latecomers / orienting them if you can’t, checking in, sharing attendee info pre-retreat, vulnerability promoting activities eg. hamming circles.
Facilitate group bonding via eg. physical games, everyone participates activities.
Encourage intentionality via eg. communicating retreat goals, the main organizer being very present (someone else handles Ops needs).
Get an easily accessed venue that can accommodate different social spaces (eg. 1-1s), 24⁄7 access to a comfy shared space, and amenities like snacks, toiletries, chargers. Doesn’t have to be fancy—participants can share rooms or do chores.
LW Forum
AI Related
What does it take to defend the world against out-of-control AGIs?
by Steven Byrnes
The author argues that even if it’s possible to create both ‘good’ and ‘bad’ AGIs, and a good one is the first created, we’re still unlikely to be able to use it to prevent or defend against the bad ones. This is because the good AGI is likely to be more limited than the bad one eg.
1. Offense is inherently different than defense (eg. disabling an electric grid could be easier than protecting it).
2. Humans might not trust a good AGI eg. we might limit it’s self improvement or access to resources, or slow it down so we can interpret it better. A bad AGI doesn’t have these limits.
3. The good AGI may be hamstrung by human norms, laws, overton windows, mass agreement etc. - while again the bad AGI can skip past this.
They also address possible things that could help with these issues and why they think these are unlikely to make a difference.
A Barebones Guide to Mechanistic Interpretability Prerequisites
by Neel Nanda
Aims to make it easier for people to field-switch by presenting a list of barebones prerequisites required to understand, write code and ideally contribute useful research to mechanistic interpretability. This includes some math (basics of linear algebra, probability, calculus), Python basics, and a rough grounding in ML with focus on transformers.
Resources that (I think) new alignment researchers should know about
by Akash
Including links to other resources, how to start researching, programs and funding opportunities, and useful exercises to try.
AGI and Lock-In
by Lukas_Finnveden, Jess_Riedel, CarlShulman
Summary of a 40-page document arguing that AGI makes it possible to perfectly preserve nuanced specifications of a wide variety of values or goals far into the future, and develop AGI-based institutions to pursue these for millions or trillions of years. These could be good or bad lock-ins.
The three arguments given are:
1. Better preservation of information (eg. whole brain emulation to allow future AGIs to ask what humans today would have thought).
2. AIs can pursue a set of values single-mindedly vs. being distracted by biological goals like humans.
3. Institutions could ‘reboot’ AIs that showed value drift or halt progress if decisions became ambiguous due to the changing environment.
aisafety.community—A living document of AI safety communities
by zeshen, plex
Volunteers are maintaining this live database of AI safety communities. Please update missing communities or improve descriptions as you use it. They’re also looking for more volunteer maintainers.
Rationality Related
Mnestics
by Jarred Filmer
Humans forget important things regularly. So write down what you care about, and read it each morning before you plan your day.
Trigger-based rapid checklists
by VipulNaik
A trigger-based rapid checklist is a memorized checklist that you execute rapidly in response to a specific trigger. Eg. The trigger could be turning on your laptop in the morning, and the checklist could include items like ‘check weather’ and ‘catch up on email’. The author has found this useful for ensuring they’re on top of messages / email, recording things regularly, and finishing the day in a ‘clean state’ (laptop closed, backups done, no open tabs etc.)
Other
How Risky Is Trick-or-Treating?
by jefftk
In the USA, there are more child pedestrian fatalities on Halloween than any other day. But there are also more children walking around on Halloween, so given that base rate it might actually be one of the safest days.
Didn’t Summarize
A Walkthrough of A Mathematical Framework for Transformer Circuits by Neel Nanda
Beyond Kolmogorov and Shannon by Alexander Gietelink Oldenziel, Adam Shai
Some Lessons Learned from Studying Indirect Object Identification in GPT-2 small by KevinRoWang, Alexandre Variengien, Arthur Conmy, Buck, jsteinhardt
This Week on Twitter
AI
Ahead of the publication of the white paper on AI governance expected later this year,
@CommonsSTC, (a cross-party group of UK MPs) has launched a new inquiry to “examine the effectiveness of AI governance and the UK Government’s proposals.” The call for evidence is open until 25 November. (tweet)
Forecasting
Infer are a group doing forecasts for stakeholders in US government on topics like critical science and tech issues, and high risk geopolitical events. They’re looking for more forecasters. (website)
National Security
Russia terminated the grain deal.Putin has said there is ‘no need’ for nuclear weapons in Ukraine. (article summarizing the week’s developments in the Russia / Ukraine war)