I was listening/chatting to the British philosopher, Jonathan Wolff on how to value life. We went through some expected value and cost-benefit theory as applied to philosophy and healthcare spending.
We then came across the challenge of potentially incorporating “society preferences”. For instance, in the UK it seems—from surveys and outreach as well as policy practice—that much more money is spent on pre-term babies and also rare disease healthcare spend then woul dbe suggested by the more “normal” quality adjusted life year (QALY) calculation for eg diabetes treatment.
We didn’t discuss this that much further, but it seems challenging to me that people/society does seem to value some times of health/life differently and to what extent we need to think about that when allocating healthcare or other scare but beneficial sources.
It does seem to me something EA or society has to account for? Or not? And then also how? As revealed preference is tricky here?
I’ve been thinking about AI safety again, and this is what I’m thinking:
The main argument of Stuart Russell’s book focuses on reward modeling as a way to align AI systems with human preferences. But reward modeling seems more like an AI capabilities technology than an AI safety one. If it’s really difficult to write a reward function for a given task Y, then it seems unlikely that AI developers would deploy a system that does it in an unaligned way according to a misspecified reward function. Instead, reward modeling makes it feasible to design an AI system to do the task at all.
Even with reward modeling, though, AI systems are still going to have similar drives due to instrumental convergence: self-preservation, goal preservation, resource acquisition, etc., even if they have goals that were well specified by their developers. Although maybe corrigibility and not doing bad things can be built into the systems’ goals using reward modeling.
The ways I could see reward modeling technology failing to prevent AI catastrophes (other than misuse) are:
An AI system is created using reward modeling, but the learned reward function still fails in a catastrophic, unexpected way. This is similar to how humans often take actions that unintentionally cause harm, such as habitat destruction, because they’re not thinking about the harms that occur.
Possible solution: create a model garden for open source reward models that developers can use when training new systems with reward modeling. This way, developers start from a stronger baseline with better safety guarantees than they would have if they were developing reward modeling systems from scratch/with only their proprietary training data.
A developer cuts corners while creating an AI system (perhaps due to economic pressure) and doesn’t give the system a robust enough learned reward function, and the system fails catastrophically.
Lots of ink has been spilled about arms race dynamics 😛
Possible solution: Make sure reward models can be run efficiently. For example, if reward modeling is done using a neural network that outputs a reward value, make sure it can be done well even with slimmer neural networks (fewer parameters, lower bit depth, etc.).
The main argument of Stuart Russell’s book focuses on reward modeling as a way to align AI systems with human preferences.
Hmm, I remember him talking more about IRL and CIRL and less about reward modeling. But it’s been a little while since I read it, could be wrong.
If it’s really difficult to write a reward function for a given task Y, then it seems unlikely that AI developers would deploy a system that does it in an unaligned way according to a misspecified reward function. Instead, reward modeling makes it feasible to design an AI system to do the task at all.
Maybe there’s an analogy where someone would say “If it’s really difficult to prevent accidental release of pathogens from your lab, then it seems unlikely that bio researchers would do research on pathogens whose accidental release would be catastrophic”. Unfortunately there’s a horrifying many-decades-long track record of accidental release of pathogens from even BSL-4 labs, and it’s not like this kind of research has stopped. Instead it’s like, the bad thing doesn’t happen every time, and/or things seem to be working for a while before the bad thing happens, and that’s good enough for the bio researchers to keep trying.
So as I talk about here, I think there are going to be a lot of proposals to modify an AI to be safe that do not in fact work, but do seem ahead-of-time like they might work, and which do in fact work for a while as training progresses. I mean, when x-risk-naysayers like Yann LeCun or Jeff Hawkins are asked how to avoid out-of-control AGIs, they can spout off a list of like 5-10 ideas that would not in fact work, but sound like they would. These are smart people and a lot of other smart people believe them too. Also, even something as dumb as “maximize the amount of money in my bank account” would plausibly work for a while and do superhumanly-helpful things for the programmers, before it starts doing superhumanly-bad things for the programmers.
Even with reward modeling, though, AI systems are still going to have similar drives due to instrumental convergence: self-preservation, goal preservation, resource acquisition, etc., even if they have goals that were well specified by their developers. Although maybe corrigibility and not doing bad things can be built into the systems’ goals using reward modeling.
Yup, if you don’t get corrigibility then you failed.
Regardless of overarching opinions you may or may not have about the unilateralist’s curse, I think Petrov Day is a uniquely bad time to lambast the foibles of being a well-intentioned unilateralist.
I worry that people are updating in exactly the wrong way from the events, possibly to fit preconceived ideas of what’s correct.
Your independent impression about X is essentially what you’d believe about X if you weren’t updating your beliefs in light of peer disagreement—i.e., if you weren’t taking into account your knowledge about what other people believe and how trustworthy their judgement seems on this topic relative to yours. Your independent impression can take into account the reasons those people have for their beliefs (inasmuch as you know those reasons), but not the mere fact that they believe what they believe.
Armed with this concept, I try to stick to the following epistemic/discussion norms, and think it’s good for other people to do so as well:
Trying to keep track of my own independent impressions separately from my all-things-considered beliefs (which also takes into account peer disagreement)
Trying to be clear about whether I’m reporting my independent impression or my all-things-considered belief
Feeling comfortable reporting my own independent impression, even when I know it differs from the impressions of people with more expertise in a topic
In contrast, when I actually make decisions, I try to make them based on my all-things-considered beliefs.
For example, my independent impression is that it’s plausible that a stable, global authoritarian regime, or some other unrecoverable dystopia, is more likely than extinction, and that we should prioritise those risks more than we currently do. But I think that this opinion is probably uncommon among people who’ve thought a lot about existential risks. And that makes me somewhat less confident in this opinion and somewhat less likely to actually act on it. But I think it’s still useful for me to keep track of my independent impression and report it sometimes, or else the community might end up with overly certain and overly homogenous beliefs.
This term and concept and these suggested norms aren’t at all original to me—see in particular Naming beliefs and several of the posts tagged Epistemic humility (especially this one). But I wanted a clear, concise description of this specific set of terms and norms so that I could link to it whenever I say I’m reporting my independent impression, ask someone for theirs, or ask someone whether an opinion they’ve given is their independent impression or their all-things-considered belief.
Thanks, I appreciate having something to link to! My independent impression is that it would be even easier to link to and easier to find as a top-level post.
I just re-read this comment by Claire Zabel, which is also good and is probably where I originally encountered the “impressions” vs “beliefs” distinction.
(Though I still think that this shortform serves a somewhat distinct purpose, in that it jumps right to discussing that distinction, uses terms I think are a bit clearer—albeit clunkier—than just “impressions” vs “beliefs”, and explicitly proposes some discussion norms that Claire doesn’t quite explicitly propose.)
Content warning: missing persons, violence against women, racism.
Amid the media coverage of the Gabby Petito case in the United States, there’s been some discussion of how missing persons cases for women and girls of color are more neglected than those for missing White women. Some statistics:
Black girls and women go missing at high rates, but that isn’t reflected in news coverage of missing persons cases. In 2020, of the 268,884 girls and women who were reported missing, 90,333, or nearly 34% of them, were Black, according to the National Crime Information Center. Meanwhile, Black girls and women account for only about 15% of the U.S. female population, according to census data. In contrast, white girls and women — which includes those who identify as Hispanic — made up 59% of the missing, while accounting for 75% of the overall female population.
[...]
In [Wyoming], more than 400 Indigenous girls and women went missing between 2011 and the fall of 2020, according to a state report. Indigenous people made up 21% of homicide victims in Wyoming between 2000 and 2020, despite being less than 3% of the state’s population. The disparity can be seen in the media: Only 18% of Indigenous female victims received coverage. However, among white victims, 51% were in the news.
To be clear, Rivers explains, it’s not about asking for more attention or being in “competition” with white people — it’s about other groups getting the same attention as white victims and having their lives honored in the same ways.
Like many young people in the EA community, I often find myself paralyzed by career planning and am quick to second-guess my current path, developing an unhealthy obsession for keeping doors open in case I realize that I really should have done this other thing.
Many posts have been written recently about the pitfalls of planning your career as if you were some generic template to be molded by 80,000 Hours [reference Holden’s aptitudes post, etc.]. I’m still trying to process these ideas and think that the distinction between local and global optimization may help me (and hopefully others) with career planning.
Global optimization involves finding the best among all possible solutions. By its nature, EA is focused on global optimization, identifying the world’s most pressing problems and what we can do to solve them. This technique works well at the community level: we can simultaneously explore and exploit, transfer money between cause areas and strategies, and plan across long timescales. But global optimization is not as appropriate in career planning. Instead, perhaps it is better to think about career choice in terms of local optimization, finding the best solution in a limited set. Local optimization is more action-oriented, better at developing aptitudes, and less time-intensive.
The differences between global and local optimization are perhaps similar to the differences between sequence-based and cluster-based thinking [reference Holden’s post]. Like sequence-based thinking, which asks and answers questions with linear, expected-value style reasoning, global optimization is too vulnerable to subtle changes in parameters. Perhaps I’ve enrolled in a public health program but find AI safety and animal suffering equally compelling cause areas. Suppose I’m too focused on global optimization. In that case, a single new report by the Open Philantrophy Project suggesting shorter timelines for transformative AI might lead me to drop out of my program and begin anew as a software engineer. But perhaps the next day, I find out that clean meat is not as inevitable as I once thought, so I leave my job and begin studying bioengineering.
Global optimization makes us more likely to vacillate between potential paths excessively. The problem, though, is that we need some stability in our goals to make progress and develop the aptitudes necessary for impact in any field. Adding onto this the psychological stress of constant skepticism about one’s trajectory, it seems that global optimization can be a bad strategy for career planning. The alternative, local optimization, would have us look around our most immediate surroundings and do our best within that environment. Local optimization seems like a better strategy if we think that “good correlates with good,” and aptitudes are likely to transfer if we later become convinced that no, really, I should have done this other thing.
I think the difficult thing for us is to find the right balance between these two optimization techniques. We don’t want to fall into value traps or otherwise miss the forest for the trees, focusing too much on our most immediate options without considering more drastic changes. But too much global optimization can be similarly dangerous.
Redwood Research is looking for people to help us find flaws in our injury-detecting model. We’ll pay $20/hour for this, for up to 2 hours; after that, if you’ve found interesting stuff, we’ll pay you for more of this work. We might stop doing this (or stop recruiting additional people) in a few days.
If you’re interested, please email adam@rdwrs.com so he can add you to a Slack channel with other people who are working on this. This might be a fun task for people who like being creative, being tricky, and figuring out how language models understand language.
You can try out the interface here. The task is to find things that the model classifies as non-injurious that are actually injurious according to our definition. Full instructions here.
Correct me if I’m wrong, but I think in Christianity, there’s a lot of respect and positive affect for the “ordinary believer”. Christians who identify as “ordinary Christians” feel good about themselves for that fact. You don’t have to be among the brightest stars of the community in order to feel like you belong.
I think in EA, we’re extremely kind, but we somehow have less of this. Like, unless you have 2 PhD’s by the age of 25 and you’re able to hold your own in a conversation about AI-alignment theory with the top researchers in the world… you sadly have to “settle” for menial labour with impact hardly worth talking about. I’m overstating it, of course, but am I wrong?
I’m not saying ambition is bad. I think shooting for the stars is a great way to learn your limits. But I also notice a lot of people suffering under intellectual pressure, and I think we could collectively be more effective (and just feel better) if we had more… room for “ordinary folk dignity”?
My experience as a non-PhD who dropped out of EA things for two years before returning is that I felt welcome and accepted when I started showing up in EA spaces again. And now that I’ve been at CEA for three years, I still spend a lot of my time talking to and helping out people who are just getting started and don’t have any great credentials or accomplishments; I hope that I’m not putting pressure on them when I do this.
That said, every person’s experience is unique, and some people have certainly felt this kind of pressure, whether self-imposed as a result of perceived community norms or thrust upon them by people who were rude or dismissive at some point. And that’s clearly awful — people shouldn’t be made to feel this way in general, and it’s especially galling to hear about it sometimes happening within EA.
My impression is that few of these rude or dismissive people are themselves highly invested in the community, but my impression may be skewed by the relationships I’ve built with various highly invested people in the job I now have.
Lots of people with pretty normal backgrounds have clearly had enormous impact (too many examples to list!). And within the EA spaces I frequent, there’s a lot of interest and excitement about people sharing their stories of joining the movement, even if those people don’t have any special credentials. The most prominent example of this might be Giving What We Can.
I don’t understand the “menial labor” point; the most common jobs for people in the broader EA community are very white-collar (programmers, lawyers, teachers...) What did you mean by that?
Personally, the way I view “ordinary folk dignity” in EA is through something I call “the airplane test”. If I sat next to someone on an airplane and saw them reading Doing Good Better, and they seemed excited about EA when I talked to them, I’d be very happy to have met them, even if they didn’t have any special ambitions beyond finding a good charity and making occasional donations. There aren’t many people in the world who share our unusual collection of values; every new person is precious.
And to respond to your question about what I meant by “menial labour”. I was being poetic. I just mean that I feel like EA places a lot of focus on the very most high-status jobs, and I’ve heard friends despairing for having to “settle” for anything less. I sense that this type of writing might not be the norm for EA shortform, but I wasn’t sure.
Nono, I’m not trying to point to a problem of EAs trying to make others feel unwelcome or dumb. I think EA is extremely kind, and almost universally tries hard to make people feel welcome. I’m just pointing to the existence of an unusually strong intellectual pressure, perhaps combined with lots of focus on world-saving heroes and talk about “what should talented people do?”
I think ambition is good, but I think we can find ways of encouraging ambition while also mitigating at least some of the debilitating intelligence-dysphoria many in our community suffer from.
I’m writing this in reaction to talking to three of my friends who suffer under the intellectual pressure they feel. (Note that the following are all about the intellectual pressure they get from EA, and not just in general due to academic life.)
Friend1: “EA makes me feel real dumb XD i think i feel out of place by being less intelligent”
_
Friend2: “I’m not worried that I’m not smart, but I am worried that I am not smart enough to meet a certain threshold that is required for me to do the things I want to do. … I think I have very low odds of achieving things I deeply want to achieve. I think that is at least partially responsible for me being as extremely uncomfortable about my intelligence as I am, and not being able to snap out of it.”
_
Me: “Do you ever refrain from trying to contribute intellectually because you worry about taking up more attention than it’s worth?”
Friend3: “hmm, not really for that reason. because I’m afraid my contribution will be wrong or make me look stupid. wrong in a way that reflects negatively on me—stupid errors, revealing intellectual or character weakness.
_
Some of this is a natural and unavoidable result of the large focus EA places on intellectual labour, but I think it’s worse than it needs to be. I think some effort to instil some “ordinary EA dignity” into our culture wouldn’t hurt. I might have a skewed sample, however.
I think an EA career fair would be a good idea. It could have EA orgs as well as non-EA orgs that are relevant to EAs (for gaining career capital or earning to give)
Naïve question: What’s the deal with the cheapest CO2 offset prices?
It seems, though, that the current price of credible offsets is much lower than the social cost of carbon, and possibly so low that just buying offsets starts to look competitive with GiveWell top charities.
I’m not an expert on this. (I run an offsetting program for a small organization, but that takes about 4h/year. Otherwise I don’t think about this much.) I’m also not anywhere near advocating that we should sink tons of money into offsets. But this observation strikes me as unintuitive enough that I suspect I’m missing something.
Cost of offsetting: I’ve generally seen the UN FCCC’s carbon offset market presented as credible, if not the biggest or most scalable. They make it pretty easy direct money to specific projects, and most of them pass the smell test as being both verifiable and additive. One project that popped up last year involved converting the operations of a platinum mining company in Bihar from burning coal to another burning another fossil fuel in a slightly-lower-emissions way. That’s easy to verify, and there was a clear argument for why it wouldn’t make economic sense for them to transition without the offset money.
Current prices on that market are around $1.75/tonne, and at other times of year recently, I’ve seen numbers dip as low as ~$0.33/tonne.
Value of offsetting: My impression is that the social cost of carbon is somewhere in the $30–300/tonne range, suggesting that cheap credible offsets are plausibly pretty high-leverage, but I don’t have good framework for thinking about this kind of impact.
I found this easier to visualize in light of this line from a recent Future Perfect piece:
[...] Bressler found that adding 4,434 metric tons of carbon dioxide into the atmosphere would result in one heat-related death this century.
Combining that with the numbers above yields a price-per-life-saved in the $1k-10k range, which is within the same order of magnitude as charities like AMF, IIRC.
So? What am I missing? Is there something fishy with that use of of dollars per life saved? Are these cheap UN-monitored offsets actually bullshit? Or is it actually just very impactful to buy this kind of offset while they’re still cheap (...even if it’s still not the most effective way to spend money, or even the most effective way to spend money on climate impacts)?
One project that popped up last year involved converting the operations of a platinum mining company in Bihar from burning coal to another burning another fossil fuel in a slightly-lower-emissions way. That’s easy to verify, and there was a clear argument for why it wouldn’t make economic sense for them to transition without the offset money
I am also confused about the general question, but I found this intervention interesting to think about. It seems like the legitimacy of this comes down to the elasticity of demand for coal in India (basically, if someone buys 1 ton less of coal, will someone else buy that same ton for a lower price, or will coal producers make one ton less?). I couldn’t find any data on elasticity of demand for coal in India, but this paper estimates it for China as 0.3 to 0.7, which is maybe an OK proxy? And I don’t know if it’s reasonable to model the elasticity of demand of coal and of the other fossil fuel as the same (eg, it would be terrible if not buying 1 ton of coal reduces the total coal by 0.2 tons produced, but buying 1 ton of oil increases total oil produced by 0.8 tons).
Overall it feels non-obvious to me whether it’s legit, though I lean towards “probably, but about half as effective as a naive calculation suggests”
(See the linked doc for the most up-to-date version of this.)
The scope of this doc is fairly broad and nebulous. This is not The Definitive Collection of collections of resources on these topics—it’s just the relevant things that I (Michael Aird) happen to have made or know of.
Here are some things that are “internal” or that I don’t have permission to share, but that I might be able to make shareable versions of, share after asking permission, or talk to people about, if someone is interested::
Summary and commentary on the book Managing to Change the World—someone at RP
“Management / org structure things RP could consider”—me
“Aspects of CLR’s SRF which RP’s internship could draw on”—me
My Anki cards on Getting Things Done—me
“RSP Personal Review: Meta doc”—someone who was at FHI
Performance review templates collected at the bottom of the FHI doc “Performance feedback—Meta doc”—someone who was at FHI
Quick notes on management—someone at RP
Finally, here are my notes on relevant books (though each of these links isn’t really a “collection”):
This is a doc I made, and I suggest reading the doc rather than shortform version (assuming you want to read this at all). But here it is copied out anyway:
What is this doc, and why did I make it?
AI governance is a large, complex, important area that intersects with a vast array of other fields. Unfortunately, it’s only fairly recently that this area started receiving substantial attention, especially from specialists with a focus on existential risks and/or the long-term future. And as far as I’m aware there aren’t yet any canonical, high-quality textbooks or online courses on the topic.[1] It seems to me that this means this is an area where well-curated and well-structured reading lists, syllabi, or similar can be especially useful, helping to fill the role that textbooks otherwise could.[2]
Fortunately, when I started looking for relevant reading lists and syllabi, I was surprised by how many there were. So I decided to try to collect them all in one place. I also tried to put them in very roughly descending order of how useful I’d guess they’d be to a randomly chosen EA-aligned person interested in learning about AI governance.
I think this might help myself, my colleagues, and others who are trying to “get up to speed”, for the reasons given in the following footnote.[3]
I might later turn this doc into a proper post on the EA Forum.
This is more relevant to technical AI safety than to AI governance, but some categories are pretty relevant to AI governance, especially “AI strategy and policy”, “Forecasting”, and “Field building”
Tangential critique: I personally think that it’s problematic and misleading that both The Precipice and this syllabus use the heading “unaligned artificial intelligence” while seeming to imply that this covers all key aspects of AI risk, since I think this obscures some risk pathways.
My post Crucial questions for longtermists includes a structured list of questions related to the “Value of, and best approaches to, work related to AI”, and this associated doc contains readings related to each of those questions
I haven’t updated this much since 2020
Questions listed there include:
Is it possible to build an artificial general intelligence (AGI) and/or transformative AI (TAI) system? Is humanity likely to do so?
What form(s) is TAI likely to take? What are the implications of that? (E.g., AGI agents vs comprehensive AI services)
What will the timeline of AI developments be?
How much should longtermists’ prioritise AI?
What forms might an AI catastrophe take? How likely is each?
What are the best approaches to reducing AI risk or increasing AI benefits?
I think many/most of these books were chosen for “seem[ing] likely to have passages relevant to the question of how well policy-makers will deal with AGI”
Many/most of these aren’t available as audiobooks; Luke turned them into audiobooks himself
You could also use research agendas related to AI governance as reading lists, by following the sources they cite on various topics. Relevant agendas include:
Longtermist AI policy projects for economists—Risto Uuk (this doc was originally just made for Risto’s own use, so the ideas shouldn’t be taken as high-confidence recommendations to anyone else)
I think this is much more focused on technical AI safety than AI governance
Some Rethink Priorities staff may soon make a long, tiered reading list tailored to the AI governance project ideas we may work on. If it seems to me that this would be useful to other people, I might add a link to a version of it here.
I think there was also a short reading list associated with the EA In-Depth Fellowship
A related category to reading lists is newsletters that provide summaries and commentary of a bunch of research outputs. E.g.:
Rohin’s
Jack Clarke’s
CSET’s
…
Mauricio Baker suggested that I or people reading this doc might also be interested in “syllabi aimed at aspiring AI technical safety researchers, such as this one: Technical AI Safety Reading List. I have a vague sense that engaging with some of this content has been helpful for my having a better broad sense of what’s going on with AI safety, which seems helpful for governance.”
Some parts of Krakovna’s AI safety resources and Maini’s AI Reading List may be quite useful for AI governance people, though I think they’re more relevant for technical AI safety people
My thanks to everyone who made these lists, as well as to Mauricio Baker for pointing me to some of the lists.
Footnotes
[1] Though there are various presumably high-quality textbooks or courses with some relevance, some high-quality non-textbook books on the topic, some in-person courses that might be high-quality (I haven’t participated in them), and some things that fill somewhat similar roles (like EA seminar series, reading groups, or fellowships).
This collection should make it easier to find additional reading lists, syllabi, etc., and thus easier to find additional readings that have been evaluated as especially worth reading in general, especially worth reading on a given topic, and/or especially good as introductory resources.
This collection should make it easier to find and focus on reading lists, syllabi, etc. that are better and/or more relevant to one’s specific needs.
To help with this, please comment on this doc if you have opinions about anything listed.
Even before or without engaging with the actual items included in a given reading list, syllabus, or similar, engaging with the structure and commentary in that document itself could help one understand what the important components, divisions, concepts, etc. within AI governance are. And this collection should help people find more, better, and/or more relevant such documents.
I think it is a cool idea for people to take a giving pledge on the same day. For example, you and your friend both decide to pledge 10% to charity on the same day. It would be even more fun if you did it with strangers. Call it “giving twins” or “giving siblings”.
Imagine that you met a couple strangers and they pledged with you. Imagine that after pledging you all just decided to be friends, or at least a type of support group for one another. Like “Hey, you and I took the further pledge together on New Year’s Day last year. When I’m in your city let’s go have a pint or maybe we can email each other about our career plans to discuss ways we can help people more!”
Which leads me to my final bit here: would anyone be interested in being my giving sibling? Haha I am interested in taking the further pledge in 2022, and it would be fun to have the same ‘giving birthday’ as other people so I could befriend them, meet people in the community, and get a couple lifelong friends.
Giving what we can could even take this idea and run with it. They could assign you a giving sibling if you entered into the sibling program, this could help increase the feeling that we are a community.
That could be the case, but I think the emphasis is more on the idea that you have the same “birthdate” to be considered a giving sibling.
Like on February 15 you and a friend took the Giving Pledge together and then that date was the same day you became siblings. Then you celebrate that day every year or form a bond around this shared experience.
Hi, I am working on a non profit to help animals in another country by creating a sanctuary for them. I already know how to setup a corporation, convert it to a non profit, and operate it with a board of directors. For this project, I will be opening a US non-profit and using funds to help animals in other countries.
I am looking for additional guides on how to establish a non-profit organization in another country and wanted to know if there is anything different about running a US non profit that does work in another country and if the process is any different when operating only in the US.
I wrote a very rough draft of an idea I had. It was just a stream of consciousness and I didn’t really edit it. I’m not sure what the standards are like on the EA Forum: I would like to invest time in developing this further, as well as other posts I could possibly write, but as I mentioned at the bottom of this post, I’m at a critical juncture in my career and need to invest my time and energy elsewhere.
If I can produce forum posts with potentially some interesting ideas, but that are of a relatively low standard, is is better for me to post these without worrying about developing them, or waiting until I have more time to invest in developing better forum posts/writing less and editing more?
Epistemic status: medium-low (I have some conviction in this idea, but I’m posting this without much rework in an attempt to overcome perfectionism, partially inspired by Neel Nanda. This likely means that some of the things in this post will be wrong, but I hope it could be useful nonetheless).
Summary: we should systematically proceduralise activities more. Learning how to proceduralise tasks requires a lot of up-front investment. This could be reduced by centralised, shared ‘checklists’ for different tasks which are designed to increase effectiveness in particular tasks.
I’ve always had a preference for proceduralising (approaching tasks in a systematised manner), but my exposure to EA and the rationalist community has inundated me with high quality information about how to do effective research, learn effectively, prioritise effectively and make better decisions etc.
The bottleneck holding back my progress is now the implementation of ideas that I have high conviction in. To effectively implement these ideas and integrate them into my system 2 would require a lot of deliberate practice and highly engaged, focussed efforts.
I want to somehow outsource or expedite this process, and this seems like an important issue that is likely to be affecting other EAs as well. One potential solution is the more deliberate development of effective checklists, ideally some kind of centralised database.
Checklist have been advocated by Charlie Munger (Warren Buffet’s business partner) as an effective way of making better decisions and overcoming biases. They help ensure that we prime our brain to consider different important aspects of a problem that we might not naturally consider (discussed more thoroughly in https://www.amazon.de/-/en/Atul-Gawande/dp/0312430000).
My mental model of highly skilled workers in a complex field is essentially people who have internalised complex procedures internally, but that with some effort these mental processes could be proceduralised to a greater degree. Checklists do not have to involve mindlessly going through a recipe book: they could involve open ended questions designed to elicit deeper thinking.
Implementation
I’m still in the early phases of developing this idea, but I’ve started to implement it as part of my workflow. I use OneNote and have sections entitled ‘Working Best Practices’, ‘Research Best Practices’ etc. As an example, here’s something taken the page ‘when reading a research paper’ (I stole the questions from EA-related sources, mostly from: https://effectivethesis.org/resources/)
Short:
Main question
Methods used
Data required
Results
Long:
Can the approach used in a study be applied to other contexts, countries or time periods?
What assumptions are implicit?
Are all the assumptions sensible?
To what extent might the results be sensitive to the assumptions?
How can they be relaxed?
Are there any unnecessary assumptions?
Is the approach used the most appropriate one?
Have new techniques been developed since the paper was first written?
Have all relevant statistical tests been carried out?
Are the results consistent with expectations, or earlier work?
Are the surrogate or constructed variables the most appropriate for the task and can anything be said about likely biases?
Are there any implications of the study which have not been fully drawn out by the author?
Can these be exploited in your work?
With a checklist, rather than having to learn all of the questions to ask yourself when reading a paper, you just have to remember where to find these questions, which hopefully frees up time to focus on other things.
I imagine a centralised EA checklist, with different sections differentiated by clear cues. A section would have an inbox, where people add in ideas with low epistemic standards, to increase creativity. There would then be two additional aspects to the section: a short version and a long version. The short version would highlight the best 4 parts of a checklist to follow, the long version would highlight the best 20, with voting to help ensure that the best ones rise to the top. Comments and feedback would also be useful as a way of developing our understanding of the pitfalls of certain procedures.
This approach might work well for activities where the order of actions is not important. However in many cases it is. Perhaps then a better solution would be for users to submit either individual items or whole procedures. Users would vote on individual procedures, which would be collated into a second inbox. From here, users could bundle procedures together to create ‘packages’, which could then be voted on.
These are early stage ideas and could almost certainly be improved upon. I would be happy to help develop this but I have limited programming experience so would struggle to implement as-hoc features. If anyone was interested in helping me implement this that would be great! I do think this has the potential to multiply EAs impact.
Caveats and downside risk: it’s possible that these checklists would act as a mental crutch and undermine the long-term development of people using them: learning the procedures of a task and internalising this kind of knowledge may facilitate greater creativity and better work in the long-run, and developing these checklists could end up being a distracting diversion for EAs and not a good use of time.
I think it’s worthwhile proceeding despite these risks: if they end up being true, I hope that this would be noticed by EAs, and users could be encouraged to learn the checklists rather than blindly apply them.
Another risk is that this could result in EAs approaching problems in a similar way, which could stymie productivity in the long run. This could be mitigated by having another section with more creative ways of approaching problems. But I think the gains to optimisation are so large that they’re much more important than the trade off between creativity and proceduralisation. The ‘efficient frontier’ comes to mind. If we reach a point where EAs are optimising consistently to a high degree, then we can start having more conversations about potential trade-offs.
Implementation
This is something I would be interested in taking on at some point, but I’m at a critical juncture in my career right now and need to devote my time and energy to figuring out my next steps (likely applying for Economics PhDs).
I haven’t yet looked into how this could be effectively implemented: it seems likely that there are online platforms that could incorporate this, but I don’t have much expertise in the field.
If anyone is interested in collaborating on making this happen, please comment ‘interested’.
In answer to your question, I think that it’s generally better to create a top-level post than a Shortform post, as long as you’re comfortable doing so. Shortform serves a useful purpose, but top-level posts have a better chance of getting useful engagement.
I think this would make sense as a top-level post, and would encourage you to try sharing it in that context!
My thoughts on the checklist compilation idea: It’s hard to make a given intellectual resource very popular, but if you can pull off this one, I think it could be really useful. Many EA orgs have their own private procedural checklists; just taking those and bringing them out into the public could be valuable on its own. (For example, I have a checklist for creating email newsletters in Mailchimp that I think could be useful for others who do similar things.)
I think a plausibly good training exercise for EAs wanting to be better at empirical/conceptual research is to deep dive into seminal papers/blog posts and attempt to identify all the empirical and conceptual errors in past work, especially writings by either a) other respected EAs or b) other stuff that we otherwise think of as especially important.
I’m not sure how knowledgeable you have to be to do this well, but I suspect it’s approachable for smart people who finish high school, and certainly by the time they finish undergrad^ with a decent science or social science degree.
I think this is good career building for various reasons:
you can develop a healthy skepticism of the existing EA orthodoxy
I mean skepticism that’s grounded in specific beliefs about why things ought to be different, rather than just vague “weirdness heuristics” or feeling like the goals of EA conflict with other tribal goals.
(I personally have not found high-level critiques of EA, and I have read many, to be particularly interesting or insightful, but this is just a personal take).
you actually deeply understand at least one topic well enough to point out errors
For many people and personalities, critiquing a specific paper/blog post may be a less hairy “entry point” into doing EA-research adjacent work than plausible alternatives like trying to form your own deep inside views on questions that are really open-ended and ambiguous like “come up with a novel solution in AI alignment” or “identify a new cause X”
creates legible career capital (at least within EA)
requires relatively little training/guidance from external mentors, meaning
our movement devotes less scarce mentorship resources into this
people with worse social skills/network/geographical situation don’t feel (as much) at a disadvantage for getting the relevant training
you can start forming your own opinions/intuitions of both object-level and meta-level heuristics for what things are likely to be correct vs wrong.
In some cases, the errors are actually quite big, and worth correcting (relevant parts of ) the entire EA movement on.
Main “cons” I can think of:
I’m not aware of anybody successfully doing a really good critique for the sake of doing a really good critique. The most exciting things I’m aware of (publicly, zdgroff’s critique of Ng’s original paper on wild animal suffering, alexrjl’s critique of Giving Green. I also have private examples) mostly comes from people trying to deeply understand a thing for themselves, and then along the way spotting errors with existing work.
It’s possible that doing deliberate “red-teaming” would make one predisposed to spot trivial issues rather than serious ones, or falsely identify issues where there aren’t any.
Maybe critiques are a less important skill to develop than forming your own vision/research direction and executing on it, and telling people to train for this skill might actively hinder their ability to be bold & imaginative?
^ Of course, this depends on field. I think even relatively technical papers within EA are readable to a recent undergrad who cares enough, but this will not be true for eg (most) papers in physics or math.
This is another example of a Shortform that could be an excellent top-level post (especially as it’s on-theme with the motivated reasoning post that was just published). I’d love to see see this spend a week on the front page and perhaps convince some readers to try doing some red-teaming for themselves. Would you consider creating a post?
One additional risk: if done poorly, harsh criticism of someone else’s blog post from several years ago could be pretty unpleasant and make the EA community seem less friendly.
I’m actually super excited about this idea though—let’s set some courtesy norms around contacting the author privately before red-teaming their paper and then get going!
One additional risk: if done poorly, harsh criticism of someone else’s blog post from several years ago could be pretty unpleasant and make the EA community seem less friendly.
I think I agree this is a concern. But just so we’re on the same page here, what’s your threat model? Are you more worried about
The EA community feeling less pleasant and friendly to existing established EAs, so we’ll have more retention issues with people disengaging?
The EA community feeling less pleasant and friendly to newcomers, so we have more issues with recruitment and people getting excited to join novel projects?
Criticism makes being open about your work less pleasant, and open Red Teaming about EA projects makes EA move even further in the direction of being less open than we used to be. See also Responsible Transparency Consumption.
I’m actually super excited about this idea though—let’s set some courtesy norms around contacting the author privately before red-teaming their paper and then get going!
Thanks for the excitement! I agree that contacting someone ahead of time might be good (so at least someone doesn’t learn about their project being red teamed until social media blows up), but I feel like it might not mitigate most of the potential unpleasantness/harshness. Like I don’t see a good cultural way to both incentivize Red Teaming and allow a face-saving way to refuse to let your papers be Red Teamed.
Like if Red Teaming is opt-in by default, I’d worry a lot about this not taking off the ground, while if Red Teaming is opt-out by default I’d just find it very suss for anybody to refuse (speaking for myself, I can’t imagine ever refusing Red Teaming even if I would rather it not happen).
Easy steps could be to add a “red team” tag on the forum and point to this post to encourage people to do this.
I have at times given advice to early career EA’s mostly in AI Safety similar to this. When people have trouble coming up with something they might want to write about on the forum, I encourage them to look for the things they don’t think are true. Most people are passively reading the forum anyway but actively looking for something the reader doesn’t think is true or is unconvinced by can be a good starting point for a post. It may be that they end up convinced of the point but can still write a post making is clearer and adding the arguments they found.
Having said this, most peoples first reaction is a terrified look. Encouraging someone’s first post to be a criticism is understandably scary.
It may be hard to get both the benefit to the participants and to the orgs. Anyone not intimidated by this might already have enough experience and career capital. To give juniors the experience you might have to make it more comfortable school work where the paper is written but only read by one other person. This makes it harder to capture the career capital.
I’d expect this to be unlikely for someone to do individually and of their own accord. At the very least best to do this in small groups to create social accountability and commitment pressures. While also defusing the intimidation. Alternately part of an existing program like an EA Fellowship. Even better as it’s own program, with all the overhead that comes with that.
I would be very excited about someone experimenting with this and writing up the results. (And would be happy to provide EAIF funding for this if I thought the details of the experiment were good and the person a good fit for doing this.)
If I had had more time, I would have done this for the EA In-Depth Fellowship seminars I designed and piloted recently.
I would be particularly interested in doing this for cases where there is some amount of easily transmissible ‘ground truth’ people can use as feedback signal. E.g.
You first let people red-team deworming papers and then give them some more nuanced ‘Worm Wars’ stuff. (Where ideally you want people to figure out “okay, despite paper X making that claim we shouldn’t believe that deworming helps with short/mid-term education outcomes, but despite all the skepticism by epidemiologists here is why it’s still a great philanthropic bet overall”—or whatever we think the appropriate conclusion is.)
You first let people red-team particular claims about the effects on hen welfare from battery cages vs. cage-free environments and then you show them Ajeya’s report.
You first let people red-team particular claims about the impacts of the Justinian plague and then you show them this paper.
Hmm I feel more uneasy about the truthiness grounds of considering some of these examples as “ground truth” (except maybe the Clauset et al example, not sure). I’d rather either a) train people to Red Team existing EA orthodoxy stuff and let their own internal senses + mentor guidance decide whether the red teaming is credible or b) for basic scientific literacy stuff where you do want clear ground truths, let them challenge stuff that’s closer to obvious junk (Why We Sleep, some climate science stuff, maybe some covid papers, maybe pull up examples from Calling Bullshit, which I have not read).
That seems fair. To be clear, I think “ground truth” isn’t the exact framing I’d want to use, and overall I think the best version of such an exercise would encourage some degree of skepticism about the alleged ‘better’ answer as well.
Assuming it’s framed well, I think there are both upsides and downsides to using examples that are closer to EA vs. clearer-cut. I’m uncertain on what seemed better overall if I could only do one of them.
Another advantage of my suggestion in my view is that it relies less on mentors. I’m concerned that having mentors that are less epistemically savvy than the best participants can detract a lot from the optimal value that exercise might provide, and that it would be super hard to ensure adequate mentor quality for some audiences I’d want to use this exercise for. Even if you’re less concerned about this, relying on any kind of plausible mentor seems like less scaleable than a version that only relies on access to published material.
Upon (brief) reflection I agree that relying on the epistemic savviness of the mentors might be too much and the best version of the training program will train a sort of keen internal sense of scientific skepticism that’s not particularly reliant on social approval.
If we have enough time I would float a version of a course that slowly goes from very obvious crap (marketing tripe, bad graphs) into things that are subtler crap (Why We Sleep, Bem ESP stuff) into weasely/motivated stuff (Hickel? Pinker? Sunstein? popular nonfiction in general?) into things that are genuinely hard judgment calls (papers/blog posts/claims accepted by current elite EA consensus).
But maybe I’m just remaking the Calling Bullshit course but with a higher endpoint.
___
(I also think it’s plausible/likely that my original program of just giving somebody an EA-approved paper + say 2 weeks to try their best to Red Team it will produce interesting results, even without all these training wheels).
I think your cons are good things to have noted, but here are reasons why two of them might matter less than one might think:
I think the very fact that “It’s possible that doing deliberate “red-teaming” would make one predisposed to spot trivial issues rather than serious ones, or falsely identify issues where there aren’t any” could actually also make this useful for skill-building and testing fit; people will be forced to learn to avoid those failure modes, and “we” (the community, potential future hirers, etc.) can see how well they do so.
E.g., to do this red teaming well, they may have to learn to identify how central an error is to a paper/post’s argument, to think about whether a slightly different argument could reach the same conclusion without needing the questionable premise, etc.
I have personally found that the line between “noticing errors in existing work” and “generating novel research” is pretty blurry.
A decent amount of the research I’ve done (especially some that is unfortunately nonpublic so far) has basically followed the following steps:
“This paper/post/argument seems interesting and important”
“Oh wait, it actually requires a premise that they haven’t noted and that seems questionable” / “It ignores some other pathway by which a bad thing can happen” / “Its concepts/definitions are fuzzy or conflate things in way that may obscure something important”
[I write a post/doc that discusses that issue, provides some analysis in light of this additional premise being required or this other pathway being possible or whatever, and discussing what implications this has—e.g., whether some risk is actually more or less important than we thought, or what new intervention ideas this alternative risk pathway suggests might be useful]
I’d guess that the same could also sometimes happen with this red teaming, especially if that was explicitly encouraged, people were given guidance on how to lean into this more “novel research” element when they notice something potentially major during the red teaming, people were given examples of how that has happened in the past, etc.
I want to have a program to fund people to write book reviews and post them to the EA Forum or LessWrong. (This idea came out of a conversation with a bunch of people at a retreat; I can’t remember exactly whose idea it was.)
Basic structure:
Someone picks a book they want to review.
Optionally, they email me asking how on-topic I think the book is (to reduce the probability of not getting the prize later).
They write a review, and send it to me.
If it’s the kind of review I want, I give them $500 in return for them posting the review to EA Forum or LW with a “This post sponsored by the EAIF” banner at the top. (I’d also love to set up an impact purchase thing but that’s probably too complicated).
If I don’t want to give them the money, they can do whatever with the review.
[...]
Suggested elements of a book review:
One paragraph summary of the book
How compelling you found the book’s thesis, and why
The main takeaways that relate to vastly improving the world, with emphasis on the surprising ones
I think by default good ideas like this never really end up happening, which is sad. Do you or other people have thoughts on how to make your idea actually happen? Some quick thoughts from me:
Just highlight this idea on the Forum more often/prominently
People giving career advice or mentorship to people interested in EA-aligned research careers mention this as one way of testing fit, having an impact, etc.
Heap an appropriate amount of status and attention on good instances of this having been done
That requires it to be done at least once first, of course, but can then increase the rate
E.g., Aaron Gertler could feature it in the EA Forum Digest newsletter, people could make positive comments on the post, someone can share it in a Facebook group and/or other newsletter
I know I found this sort of thing a useful and motivating signal when I started posting stuff (though not precisely this kind of stuff)
Publicly offer to provide financial prizes for good instances of this having been done
One way to do this could mirror Buck’s idea for getting more good book reviews to happen (see my other comment): “If it’s the kind of review I want, I give them $500 in return for them posting the review to EA Forum or LW with a “This post sponsored by the EAIF” banner at the top. (I’d also love to set up an impact purchase thing but that’s probably too complicated).”
Find case studies where someone found such a post useful or having written it helped someone get a good job or something, and then publicise those
Another possible (but less realistic?) way to make this happen:
Organisations/researchers do something like encouraging red teaming of their own output, setting up a bounty/prize for high-quality instances of that, or similar
An example of something roughly like this is a post on the GiveWell blog that says at the start: “This is a guest post by David Barry, a GiveWell supporter. He emailed us at the end of December to point out some mistakes and issues in our cost-effectiveness calculations for deworming, and we asked him to write up his thoughts to share here. We made minor wording and organizational suggestions but have otherwise published as is; we have not vetted his sources or his modifications to our spreadsheet for comparing deworming and cash. Note that since receiving his initial email, we have discussed the possibility of paying him to do more work like this in the future.”
But I think GiveWell haven’t done that since then?
It seems like this might make sense and be mutually beneficial
Orgs/researchers presumably want more ways to increase the accuracy of their claims and conclusions
A good red teaming of their work might also highlight additional directions for further research and surface someone who’d be a good employee for that org or collaborator for that researcher
Red teaming of that work might provide a way for people to build skills and test fit for work on precisely the topics that the org/researcher presumably considers important and wants more people working on
But I’d guess that this is unlikely to happen in this form
I think this is mainly due to inertia plus people feeling averse to the idea
Another argument against is that, for actually directly improving the accuracy of some piece of work, it’s probably more effective to pay people who are already know to be good at relevant work to do reviewing / red-teaming prior to publication
Another argument against is that, for actually directly improving the accuracy of some piece of work, it’s probably more effective to pay people who are already know to be good at relevant work to do reviewing / red-teaming prior to publication
Yeah I think this is key. I’m much more optimistic about getting trainees to do this being a good training intervention than a “directly improve research quality” intervention. There are some related arguments why you want to pay people who are either a) already good at the relevant work or b) specialized reviewers/red-teamers
paying people to criticize your work would risk creating a weird power dynamic, and more experienced reviewers would be better at navigating this
For example, trainees may be afraid of criticizing you too harshly.
Also, if the critique is in fact bad, you may be placed in a somewhat awkward position when deciding whether to publish/publicize it.
As an aside, I decided to focus my shortform on critiques of public research rather than critiques of organizations/people, even though I think the latter is quite valuable too, since a) my intuition is that the former is less acrimonious, b) relatedly, critiques of organizations may be worse at training dispassionate analysis skills (vs eg tribalistic feelings or rhetoric), c) critiques of orgs or people might be easier for newbies to fuck up and d) I think empirically, critiques of organizations have a worse hit rate than critiques of research posts.
I think by default good ideas like this never really end up happening, which is sad. Do you or other people have thoughts on how to make your idea actually happen?
As you know, one of my interns is doing something adjacent to this idea (though framed in a different way), and I may convince another intern to do something similar (depending on their interests and specific project ideas in mind).
Yeah, good point—I guess a more directed version of “People giving career advice or mentorship to people interested in EA-aligned research careers mention this as one way of testing fit, having impact, etc.” is just people encouraging people they manage to do this, or maybe even hiring people with this partly in mind.
Though I think that that wouldn’t capture most of the potential value of this idea, since part of what’s good about is that, as you say, this idea:
requires relatively little training/guidance from external mentors, meaning
our movement devotes less scarce mentorship resources into this
people with worse social skills/network/geographical situation don’t feel (as much) at a disadvantage for getting the relevant training
(People who’ve already gone through a hiring process and have an at least somewhat more experienced researcher managing them will have an easier time than other people in testing fit, having impact, building skills, etc. in other ways as well.)
Yeah I agree that a major upside to this idea (and a key differentiator between it and other proposed interventions for fixing early stages of the research pipeline) is that it ought to be doable without as much guidance from external mentors. I guess my own willingness to suggest this as an intern project suggests that I believe it must comparatively be even more exciting for people without external guidance.
When I was 19, I moved to San Francisco to do a coding bootcamp. I got a bunch better at Ruby programming and also learned a bunch of web technologies (SQL, Rails, JavaScript, etc).
It was a great experience for me, for a bunch of reasons.
I got a bunch better at programming and web development.
It was a great learning environment for me. We spent basically all day pair programming, which makes it really easy to stay motivated and engaged. And we had homework and readings in the evenings and weekends. I was living in the office at the time, with a bunch of the other students, and it was super easy for me to spend most of my waking hours programming and learning about web development. I think that it was very healthy for me to practice working really long hours in a supportive environment.
The basic way the course worked is that every day you’d be given a project with step-by-step instructions, and you’d try to implement the instructions with your partner. I think it was really healthy for me to repeatedly practice the skill of reading the description of a project, then reading the step-by-step breakdown, and then figuring out how to code everything.
Because we pair programmed every day, tips and tricks quickly percolated through the cohort. We were programming in Ruby, which has lots of neat little language features that it’s hard to pick up all of on your own; these were transmitted very naturally. I also was pushed to learn my text editor better.
The specific content that I learned was sometimes kind of fiddly; it was helpful to have more experienced people around to give advice when things went wrong.
I think that this was probably a better learning experience than most tech or research internships I could have gotten. If I’d had access to the best tech/research internships, maybe that would have been better. I think that this was probably a much better learning experience than eg most Google internships seem to be.
I met rationalists and EAs in the Bay.
I spent a bunch of time with real adults who had had real jobs before. The median age of students was like 25. Most of the people had had jobs before and felt dissatisfied with them and wanted to make a career transition. I think that spending this time with them helped me grow up faster.
I somehow convinced my university that this coding bootcamp was a semester abroad (thanks to my friend Andrew Donnellan for suggesting this to me; that suggestion plausibly accelerated my career by six months), which meant that I graduated on schedule even though I then spent six months working for App Academy as a TA (which incidentally was also a good experience.)
Some ways in which my experience was unusual:
I was a much stronger programmer on the way in to the program than most of my peers.
I am deeply extroverted and am fine with pair programming every day.
It seems plausible to me that more undergrad EAs should do something like this, especially if they can get college credit for it (which I imagine might be hard for most students—I think I only got away with it because my university didn’t really know what was going on). The basic argument here is that it might be good for them the same way it was good for me.
More specifically, I think that there are a bunch of EAs who want to do technical AI alignment work and who are reasonably strong but not stellar coders. I think that if they did a coding bootcamp between, say, freshman and sophomore year, they might come back to school and be a bunch stronger. The bootcamp I did was focused on web app programming with Ruby and Rails and JavaScript. I think that these skills are pretty generically useful to software engineers. I often am glad to be better than my coworkers at quickly building web apps, and I learned those skills at App Academy (though being a professional web developer for a while also helped). Eg in our current research, even aside from the web app we use for getting our contractors to label data, we have to deal with a bunch of different computers that are sending data back and forth and storing it in databases or Redis queues or whatever. A reasonable fraction of undergrad EAs would seem like much more attractive candidates to me if they’d done a bootcamp. (They’d probably seem very marginally less attractive to normal employers than if they’d done something more prestigious-seeming with that summer, but most people don’t do very prestigious-seeming things in their first summer anyway. And the skills they had learned would probably be fairly attractive to some employers.)
This is just a speculative idea, rather than a promise, but I’d be interested in considering funding people to do bootcamps over the summer—they often cost maybe $15k. I am most interested in funding people to do bootcamps if they are already successful students at prestigious schools, or have other indicators of talent and conscientious, and have evidence that they’re EA aligned.
Another thing I like about this is that a coding bootcamp seems like a fairly healthy excuse to hang out in the Bay Area for a summer. I like that they involve working hard and being really focused on a concrete skill that relates to the outside world.
I am not sure whether I’d recommend someone do a web programming bootcamp or a data science bootcamp—though data science might seem more relevant, I think the practical programming stuff in the web programming bootcamp might actually be more helpful on the margin. (Especially for people who are already doing ML courses in school.)
I don’t think there are really any bootcamps focused on ML research and engineering. I think it’s plausible that we could make one happen. Eg I know someone competent and experienced who might run a bootcamp like this over a summer if we paid them a reasonable salary.
See my comment here, which applies to this Shortform as well; I think it would be a strong top-level post, and I’d be interested to see how other users felt about tech bootcamps they attended.
Here in the EA community, we’re trying to do lots of good. Recently I’ve been thinking about the similarities and differences between a community focused on doing lots of good and a community focused on getting really rich.
I think this is interesting for a few reasons:
I found it clarifying to articulate the main differences between how we should behave and how the wealth-seeking community should behave.
I think that EAs make mistakes that you can notice by thinking about how the wealth-seeking community would behave, and then thinking about whether there’s a good reason for us behaving differently.
——
Here are some things that I think the wealth-seeking community would do.
There are some types of people who should try to get rich by following some obvious career path that’s a good fit for them. For example, if you’re a not-particularly-entrepreneurial person who won math competitions in high school, it seems pretty plausible that you should work as a quant trader. If you think you’d succeed at being a really high-powered lawyer, maybe you should do that.
But a lot of people should probably try to become entrepreneurs. In college, they should start small businesses, develop appropriate skills (eg building web apps), start trying to make various plans about how they might develop some expertise that they could turn into a startup, and otherwise practice skills that would help them with this. These people should be thinking about what risks to take, what jobs to maybe take to develop skills that they’ll need later, and so on.
I often think about EA careers somewhat similarly:
Some people are natural good fits for particular cookie-cutter roles that give them an opportunity to have a lot of impact. For example, if you are an excellent programmer and ML researcher, I (and many other people) would love to hire you to work on applied alignment research; basically all you have to do to get these roles is to obtain those skills and then apply for a job.
But for most people, the way they will have impact is much more bespoke and relies much more on them trying to be strategic and spot good opportunities to do good things that other people wouldn’t have otherwise done.
I feel like many EAs don’t take this distinction as seriously as they should. I fear that EAs see that there exist roles of the first type—you basically just have to learn some stuff, show up, and do what you’re told, and you have a bunch of impact—and then they don’t realize that the strategy they should be following is going to involve being much more strategic and making many more hard decisions about what risks to take. Like, I want to say something like “Imagine you suddenly decided that your goal was to make ten million dollars in the next ten years. You’d be like, damn, that seems hard, I’m going to have to do something really smart in order to do that, I’d better start scheming. I want you to have more of that attitude to EA.”
Important differences:
Members of the EA community are much more aligned with each other than wealth-seeking people are. (Maybe we’re supposed to be imagining a community of people who wanted to maximize total wealth of the community for some reason.)
Opportunities for high impact are biased to be earlier in your career than opportunities for high income. (For example, running great student groups at top universities is pretty high up there in impact-per-year according to me; there isn’t really a similarly good moneymaking opportunity for which students are unusually well suited.)
The space of opportunities to do very large amounts of good seems much narrower than the space of opportunities to make money. So you end up with EAs wanting to work with each other much more than the wealth-maximizing people want to work with each other.
It seems harder to make lots of money in a weird, bespoke, non-entrepreneurial role than it is to have lots of impact. There are many EAs who have particular roles which are great fits for them and which allow them to produce a whole bunch of value. I know of relatively fewer cases where someone gets a job which seems weirdly tailored to them and is really high paid.
I think this is mostly because my sense is that in the for-profit world, it’s hard to get people to be productive in weird jobs, and you’re mostly only able to hire people for roles that everyone involved understands very well already. And so even if someone would be able to produce a huge amount of value in some particular role, it’s hard for them to get paid commensurately, because the employer will be skeptical that they’ll actually produce all that value, and other potential employers will also be skeptical and so won’t bid their price up.
I’m commenting on a few Shortforms I think should be top-level posts so that more people see them, they can be tagged, etc. This is one of the clearest cases I’ve seen; I think the comparison is really interesting, and a lot of people who are promising EA candidates will have “become really rich” as a viable option, such that they’d benefit especially from thinking about this comparisons themselves.
Anyway, would you consider making this a top-level post? I don’t think the text would need to be edited all — it could be as-is, plus a link to the Shortform comments.
Thanks for writing this up. At the risk of asking obvious question, I’m interested in why you think entrepreneurship is valuable in EA.
One explanation for why entrepreneurship has high financial returns is information asymmetry/adverse selection: it’s hard to tell if someone is a good CEO apart from “does their business do well”, so they are forced to have their compensation tied closely to business outcomes (instead of something like “does their manager think they are doing a good job”), which have high variance; as a result of this variance and people being risk-averse, expected returns need to be high in order to compensate these entrepreneurs.
It’s not obvious to me that this information asymmetry exists in EA. E.g. I expect “Buck thinks X is a good group leader” correlates better with “X is a good group leader” than “Buck thinks X will be a successful startup” correlates with “X is a successful startup”.
It seems like there might be a “market failure” in EA where people can reasonably be known to be doing good work, but are not compensated appropriately for their work, unless they do some weird bespoke thing.
You seem to be wise and thoughtful, but I don’t understand the premise of this question or this belief:
One explanation for why entrepreneurship has high financial returns is information asymmetry/adverse selection: it’s hard to tell if someone is a good CEO apart from “does their business do well”, so they are forced to have their compensation tied closely to business outcomes (instead of something like “does their manager think they are doing a good job”), which have high variance; as a result of this variance and people being risk-averse, expected returns need to be high in order to compensate these entrepreneurs.
It’s not obvious to me that this information asymmetry exists in EA. E.g. I expect “Buck thinks X is a good group leader” correlates better with “X is a good group leader” than “Buck thinks X will be a successful startup” correlates with “X is a successful startup”.
But the reasoning [that existing orgs are often poor at rewarding/supporting/fostering new (extraordinary) leadership] seems to apply:
For example, GiveWell was a scrappy, somewhat polemical startup, and the work done there ultimately succeeded and created Open Phil and to a large degree, the present EA movement.
I don’t think any of this would have happened if Holden Karnofsky and Elie Hassenfeld had to say, go into Charity Navigator (or a dozen other low-wattage meta-charities that we will never hear of) and try to turn it around from the inside. While being somewhat vague, my models of orgs and information from EA orgs do not suggest that they are any better at this (for mostly benign, natural reasons, e.g. “focus”).
It seems that the main value of entrepreneurship is the creation of new orgs to have impact, both from the founder and from the many other staff/participants in the org.
Typically (and maybe ideally) new orgs are in wholly new territory (underserved cause areas, untried interventions) and inherently there are fewer people who can evaluate them.
It seems like there might be a “market failure” in EA where people can reasonably be known to be doing good work, but are not compensated appropriately for their work, unless they do some weird bespoke thing.
It seems that the now canonized posts Really Hard and Denise Melchin’s experiences suggest this has exactly happened, extensively even. I think it is very likely that both of these people are not just useful, but are/could be highly impactful in EA and do not “deserve” the experiences their described.
[I think the main counterpoint would be that only the top X% of people are eligible for EA work or something like that and X% is quite small. I would be willing to understand this idea, but it doesn’t seem plausible/acceptable to me. Note that currently, there is a concerted effort to foster/sweep in very high potential longtermists and high potential EAs in early career stages, which seems invaluable and correct. In this effort, my guess is that the concurrent theme of focusing on very high quality candidates is related to experiences of the “production function” of work in AI/longtermism. However, I think this focus does not apply in the same way to other cause areas.]
Again, as mentioned at the top, I feel like I’ve missed the point and I’m just beating a caricature of what you said.
Thanks! “EA organizations are bad” is a reasonable answer.
(In contrast, “for-profit organizations are bad” doesn’t seem like reasonable answer for why for-profit entrepreneurship exists, as adverse selection isn’t something better organizations can reasonably get around. It seems important to distinguish these, because it tells us how much effort EA organizations should put into supporting entrepreneur-type positions.)
Maybe there’s some lesson to be learned. And I do think that EAs should often aspire to be more entrepreneurial.
But maybe the main lesson is for the people trying to get really rich, not the other way round. I imagine both communities have their biases. I imagine that lots of people try entrepreneurial schemes for similar reasons to why lots of people buy lottery tickets. And Id guess that this often has to do with scope neglect, excessive self confidence / sense of exceptionalism, and/or desperation.
Something I imagined while reading this was being part of a strangely massive (~1000 person) extended family whose goal was to increase the net wealth of the family. I think it would be natural to join one of the family businesses, it would be natural to make your own startup, and also it would be somewhat natural to provide services for the family that aren’t directly about making the money yourself. Helping make connections, find housing, etc.
Reminds me of The House of Saud (although I’m not saying they have this goal, or any shared goal): ”The family in total is estimated to comprise some 15,000 members; however, the majority of power, influence and wealth is possessed by a group of about 2,000 of them. Some estimates of the royal family’s wealth measure their net worth at $1.4 trillion” https://en.wikipedia.org/wiki/House_of_Saud
If too few EAs go into more bespoke roles, then one reason could be risk-aversion. Rightly or wrongly, they may view those paths as more insecure and risky (for them personally; though I expect personal and altruistic risk correlate to a fair degree). If so, then one possibility is that EA funders and institutions/orgs should try to make them less risky or otherwise more appealing (there may already be some such projects).
In recent years, EA has put less emphasis on self-sacrifice, arguing that we can’t expect people to live on very little. There may be a parallel to risk—that we can’t expect people to take on more risk than they’re comfortable with, but instead must make the risky paths more appealing.
I like this chain of reasoning. I’m trying to think of concrete examples, and it seems a bit hard to come up with clear ones, but I think this might just be a function of the bespoke-ness.
A thought on how we describe existential risks from misaligned AI:
Sometimes discussions focus on a fairly specific version of AI risk, which involves humanity being quickly wiped out. Increasingly, though, the emphasis seems to be on the more abstract idea of “humanity losing control of its future.” I think it might be worthwhile to unpack this latter idea a bit more.
There’s already a fairly strong sense in which humanity has never controlled its own future. For example, looking back ten thousand years, no one decided that the sedentary agriculture would increasingly supplant hunting and gathering, that increasingly complex states would arise, that slavery would become common, that disease would take off, that social hierarchies and gender divisions would become stricter, etc. The transition to the modern world, and everything that came with this transition, also doesn’t seem to have been meaningfully chosen (or even really understood by anyone). The most serious effort to describe a possible future in detail — Hanson’s Age of Em — also describes a future with loads of features that most present-day people would not endorse.
As long as there are still strong competitive pressures or substantial random drift, it seems to me, no generation ever really gets to choose the future.[1] It’s actually sort of ambiguous, then, what it means to worry about “losing control of our future.”
Here are a few alternative versions of the concern that feel a bit crisper to me:
If we ‘mess up on AI,’ then even the most powerful individual humans will have unusually little influence over their own lives or the world around them.[2]
If we ‘mess up on AI,’ then future people may be unusually dissatisfied about the world they live in. In other words, people’s preferences will be unfilled to an unusually large degree.
Humanity may have a rare opportunity to take control of its own future, by achieving strong coordination and then locking various things in. But if we ‘mess up on AI,’ then we’ll miss out on this opportunity.[3]
Something that’s a bit interesting about these alternative versions of the concern, though, is that they’re not inherently linked to AI alignment issues. Even if AI systems behave roughly as their users intend, I believe each of these outcomes is still conceivable. For example, if there’s a missed opportunity to achieve strong coordination around AI, the story might look like the failure of the Baruch Plan for international control of nuclear weapons: that failure had much more to do with politics than it had to do with the way engineers designed the technology in question.
In general, if we move beyond discussing very sharp alignment-related catastrophes (e.g. humanity being quickly wiped out), then I think concerns about misaligned AI start to bleed into broader AI governance concerns. It starts to become more ambiguous whether technical alignment issues are actually central or necessary to the disaster stories people tell.
Although, admittedly, notable individuals or groups (e.g. early Christians) do sometimes have a fairly lasting and important influence. ↩︎
As an analogy, in the world of The Matrix, people may not actually have much less control over the long-run future than hunter-gatherers did twenty thousand years ago. But they certainly have much less control over their own lives. ↩︎
Notably, this is only a bad thing if we expect the relevant generation of humans to choose a better future than would be arrived at by default. ↩︎
Would you consider making this into a top-level post? The discussion here is really interesting and could use more attention, and a top-level post helps to deliver that (this also means the post can be tagged for greater searchability).
I think the top-level post could be exactly the text here, plus a link to the Shortform version so people can see those comments. Though I’d also be interested to see the updated version of the original post which takes comments into account (if you felt like doing that).
[ETA: I now realize that I think the following is basically just restating what Pablo already suggested in another comment.]
I think the following is a plausible & stronger concern, which could be read as a stronger version of your crisp concern #3.
“Humanity has not had meaningful control over its future, but AI will now take control one way or the other. Shaping the transition to a future controlled by AI is therefore our first and last opportunity to take control. If we mess up on AI, not only have we failed to seize this opportunity, there also won’t be any other.”
Of course, AI being our first and only opportunity to take control of the future is a strictly stronger claim than AI being one such opportunity. And so it must be less likely. But my impression is that the stronger claim is sufficiently more important that it could be justified to basically ‘wager’ most AI risk work on it being true.
I agree with this general point. I’m not sure if you think this is an interesting point to notice that’s useful for building a world-model, and/or a reason to be skeptical of technical alignment work. I’d agree with the former but disagree with the latter.
I’m not sure if you think this is an interesting point to notice that’s useful for building a world-model, and/or a reason to be skeptical of technical alignment work. I’d agree with the former but disagree with the latter.
Mostly the former!
I think the point may have implications for how much we should prioritize alignment research, relative to other kinds of work, but this depends on what the previous version of someone’s world model was.
For example, if someone has assumed that solving the ‘alignment problem’ is close to sufficient to ensure that humanity has “control” of its future, then absorbing this point (if it’s correct) might cause them to update downward on the expected impact of technical alignment research. Research focused on coordination-related issues (e.g. cooperative AI stuff) might increase in value, at least in relative terms.
Do you have the intuition that absent further technological development, human values would drift arbitrarily far? It’s not clear to me that they would—in that sense, I do feel like we’re “losing control” in that even non-extinction AI is enabling a new set of possibilities that modern-day humans would endorse much less than the decisions of future humans otherwise. (It does also feel like we’re missing the opportunity to “take control” and enable a new set of possibilities that we would endorse much more.)
Relatedly, it doesn’t feel to me like the values of humans 150,000 years ago and humans now and even ems in Age of Em are all that different on some more absolute scale.
Do you have the intuition that absent further technological development, human values would drift arbitrarily far?
Certainly not arbitrarily far. I also think that technological development (esp. the emergence of agriculture and modern industry) has played a much larger role in changing the world over time than random value drift has.
[E]ven non-extinction AI is enabling a new set of possibilities that modern-day humans would endorse much less than the decisions of future humans otherwise.
I definitely think that’s true. But I also think that was true of agriculture, relative to the values of hunter-gatherer societies.
To be clear, I’m not downplaying the likelihood or potential importance of any of the three crisper concerns I listed. For example, I think that AI progress could conceivably lead to a future that is super alienating and bad.
I’m just (a) somewhat pedantically arguing that we shouldn’t frame the concerns as being about a “loss of control over the future” and (b) suggesting that you can rationally have all these same concerns even if you come to believe that technical alignment issues aren’t actually a big deal.
FWIW, I wouldn’t say I agree with the main thesis of that post.
However, while I expect machines that outcompete humans for jobs, I don’t see how that greatly increases the problem of value drift. Human cultural plasticity already ensures that humans are capable of expressing a very wide range of values. I see no obviously limits there. Genetic engineering will allow more changes to humans. Ems inherit human plasticity, and may add even more via direct brain modifications.
In principle, non-em-based artificial intelligence is capable of expressing the entire space of possible values. But in practice, in the shorter run, such AIs will take on social roles near humans, and roles that humans once occupied....
I don’t see why people concerned with value drift should be especially focused on AI. Yes, AI may accompany faster change, and faster change can make value drift worse for people with intermediate discount rates. (Though it seems to me that altruistic discount rates should scale with actual rates of change, not with arbitrary external clocks.)
I definitely think that human biology creates at least very strong biases toward certain values (if not hard constraints) and that AI system would not need to have these same biases. If you’re worried about future agents having super different and bad values, then AI is a natural focal point for your worry.
A couple other possible clarifications about my views here:
I think that the outcome of the AI Revolution could be much worse, relative to our current values, than the Neolithic Revolution was relative to the values of our hunter-gatherer ancestors. But I think the question “Will the outcome be worse?” is distinct from the question “Will we have less freedom to choose the outcome?”
I’m personally not so focused on value drift as a driver of long-run social change. For example, the changes associated with the Neolithic Revolution weren’t really driven by people becoming less egalitarian, more pro-slavery, more inclined to hold certain religious beliefs, more ideologically attached to sedentism/farming, more happy to accept risks from disease, etc. There were value changes, but, to some significant degree, they seem to have been downstream of technological/economic change.
Really appreciate the clarifications! I think I was interpreting “humanity loses control of the future” in a weirdly temporally narrow sense that makes it all about outcomes, i.e. where “humanity” refers to present-day humans, rather than humans at any given time period. I totally agree that future humans may have less freedom to choose the outcome in a way that’s not a consequence of alignment issues.
I also agree value drift hasn’t historically driven long-run social change, though I kind of do think it will going forward, as humanity has more power to shape its environment at will.
I also agree value drift hasn’t historically driven long-run social change
My impression is that the differences in historical vegetarianism rates between India and China, and especially India and southern China (where there is greater similarity of climate and crops used), is a moderate counterpoint. At the timescale of centuries, vegetarianism rates in India are much higher than rates in China. Since factory farming is plausibly one of the larger sources of human-caused suffering today, the differences aren’t exactly a rounding error.
I do agree that quasi-random variation in culture can be really important. And I agree that this variation is sometimes pretty sticky (e.g. Europe being predominantly Christian and the Middle East being predominantly Muslim for more than a thousand years). I wouldn’t say that this kind of variation is a “rounding error.”
Over sufficiently long timespans, though, I think that technological/economic change has been more significant.
As an attempt to operationalize this claim: The average human society in 1000AD was obviously very different than the average human society in 10,000BC. I think that the difference would have been less than half as large (at least in intuitive terms) if there hadn’t been technological/economic change.
I think that the pool of available technology creates biases in the sorts of societies that emerge and stick around. For large enough amounts of technological change, and long enough timespans (long enough for selection pressures to really matter), I think that shifts in these technological biases will explain a large portion of the shifts we see in the traits of the average society.[1]
If selection pressures become a lot weaker in the future, though, then random drift might become more important in relative terms. ↩︎
Another interpretation of the concern, though related to your (3), is that misaligned AI may cause humanity to lose the potential to control its future. This is consistent with humanity not having (and never having had) actual control of its future; it only requires that this potential exists, and that misaligned AI poses a threat to it.
Recently I was asked for tips on how to be less captured by motivated reasoning and related biases, a goal/quest I’ve slowly made progress on for the last 6+ years. I don’t think I’m very good at this, but I do think I’m likely above average, and it’s also something I aspire to be better at. So here is a non-exhaustive and somewhat overlapping list of things that I think are helpful:
Treat this as a serious problem that needs active effort.
In general, try to be more of a “scout” and less of a soldier/zealot.
As much as possible, try to treat ideas/words/stats as tools to aid your perception, not weapons to advance your cause or “win” arguments.
I’ve heard good things about this book by Julia Galef, but have not read it
Relatedly, have a mental check and try to notice when you are being emotional and in danger of succumbing to motivated reasoning. Always try to think as coherently as you can if possible, and acknowledge it (preferably publicly) when you notice you’re not.
sometimes, you don’t notice this in real-time, but only long afterwards.
If this happens, try to postmortem your mistakes and learn from them.
In my case, I noticed myself having a bunch of passion that’s neither truth- nor utility- tracking with the whole Scott Alexander/NYT debacle.
I hope to not repeat this mistake.
Be precise with your speech, and try not to hide behind imprecise words that can mean anything.
especially bad to have words/phrases that connotes a lot of stuff while having empty denotation
Relatedly, make explicit public predictions or monetary bets whenever possible.
Just the fact that you’re betting either your $s or your reputation on a prediction often reminds you to be concrete and aware of your own BS.
But it’s also helpful to learn from and form your own opinions on how to have relatively accurate+well-calibrated forecasts.
Surround yourself with people who you respect as people who care a lot about the truth and try to seek it
For humans (well, at least me), clear thinking is often at least a bit of a social practice.
Just being around them helps make you try to emulate their behavior
Even more helpful if they’re at least a little bit confrontational and willing to call you out on your BS.
If you are in a situation where surrounding yourself with truthseekers is not a practical solution for your offline interactions (eg live in a small town, most government bureaucracy work), make “people who are extremely invested in truth-seeking” a top priority in the content you consume (in blog posts, social media, books, videocalls, and so forth).
Conversely, minimize the number of conversations (online and otherwise) where you feel like your interlocutor can’t change your mind
Maybe this is just a rephrase of a soldier/scout mindset distinction?
But I find it a helpful heuristic anyway.
In general, I think conversations where I’m trying to “persuade” the other person while not being open to being persuaded myself are a) somewhat impolite/disrespectful and b) has a corrosive effect (not just costly in time) to my intellectual progress
This doesn’t apply to everything important (eg intro to EA talks)
But nonetheless a good heuristic to have. If your job/hobbies pushes you to be in an “expert talking to non-expert” roles too much, you need to actively fight the compulsion to getting high on your own epistemic supply.
My personal view is that this sense of righteous certainty is more corrosive with one-to-one conversations than one-to-many conversations, and worse online than offline, but it’s just a personal intuition and I haven’t studied this rigorously.
Be excited to learn facts about the world, and about yourself.
I mostly covered psychological details here, but learning true facts about the world a) is directly useful in limiting your ability to lie about facts and b) somewhat constrains your thinking to reasoning that can explain actual facts about the world.
Critical thinking is not just armchair philosophy, but involves empirical details and hard-fought experience.
I’ve previously shared this post on CEA’s social media and (I think) in an edition of the Forum Digest. I think it’s really good, and I’d love to see it be a top-level post so that more people end up seeing it, it can be tagged, etc.
Would you be interested in creating a full post for it? (I don’t think you’d have to make any changes — this still deserves to be read widely as-is.)
I was listening/chatting to the British philosopher, Jonathan Wolff on how to value life. We went through some expected value and cost-benefit theory as applied to philosophy and healthcare spending.
We then came across the challenge of potentially incorporating “society preferences”. For instance, in the UK it seems—from surveys and outreach as well as policy practice—that much more money is spent on pre-term babies and also rare disease healthcare spend then woul dbe suggested by the more “normal” quality adjusted life year (QALY) calculation for eg diabetes treatment.
We didn’t discuss this that much further, but it seems challenging to me that people/society does seem to value some times of health/life differently and to what extent we need to think about that when allocating healthcare or other scare but beneficial sources.
It does seem to me something EA or society has to account for? Or not? And then also how? As revealed preference is tricky here?
Podcast with philosopher here: https://www.thendobetter.com/arts/2021/9/26/jonathan-wolff-valuing-life-philosophy-disability-models-society-of-equals-musical-performance-podcast
I’ve been thinking about AI safety again, and this is what I’m thinking:
The main argument of Stuart Russell’s book focuses on reward modeling as a way to align AI systems with human preferences. But reward modeling seems more like an AI capabilities technology than an AI safety one. If it’s really difficult to write a reward function for a given task Y, then it seems unlikely that AI developers would deploy a system that does it in an unaligned way according to a misspecified reward function. Instead, reward modeling makes it feasible to design an AI system to do the task at all.
Even with reward modeling, though, AI systems are still going to have similar drives due to instrumental convergence: self-preservation, goal preservation, resource acquisition, etc., even if they have goals that were well specified by their developers. Although maybe corrigibility and not doing bad things can be built into the systems’ goals using reward modeling.
The ways I could see reward modeling technology failing to prevent AI catastrophes (other than misuse) are:
An AI system is created using reward modeling, but the learned reward function still fails in a catastrophic, unexpected way. This is similar to how humans often take actions that unintentionally cause harm, such as habitat destruction, because they’re not thinking about the harms that occur.
Possible solution: create a model garden for open source reward models that developers can use when training new systems with reward modeling. This way, developers start from a stronger baseline with better safety guarantees than they would have if they were developing reward modeling systems from scratch/with only their proprietary training data.
A developer cuts corners while creating an AI system (perhaps due to economic pressure) and doesn’t give the system a robust enough learned reward function, and the system fails catastrophically.
Lots of ink has been spilled about arms race dynamics 😛
Possible solution: Make sure reward models can be run efficiently. For example, if reward modeling is done using a neural network that outputs a reward value, make sure it can be done well even with slimmer neural networks (fewer parameters, lower bit depth, etc.).
Hmm, I remember him talking more about IRL and CIRL and less about reward modeling. But it’s been a little while since I read it, could be wrong.
Maybe there’s an analogy where someone would say “If it’s really difficult to prevent accidental release of pathogens from your lab, then it seems unlikely that bio researchers would do research on pathogens whose accidental release would be catastrophic”. Unfortunately there’s a horrifying many-decades-long track record of accidental release of pathogens from even BSL-4 labs, and it’s not like this kind of research has stopped. Instead it’s like, the bad thing doesn’t happen every time, and/or things seem to be working for a while before the bad thing happens, and that’s good enough for the bio researchers to keep trying.
So as I talk about here, I think there are going to be a lot of proposals to modify an AI to be safe that do not in fact work, but do seem ahead-of-time like they might work, and which do in fact work for a while as training progresses. I mean, when x-risk-naysayers like Yann LeCun or Jeff Hawkins are asked how to avoid out-of-control AGIs, they can spout off a list of like 5-10 ideas that would not in fact work, but sound like they would. These are smart people and a lot of other smart people believe them too. Also, even something as dumb as “maximize the amount of money in my bank account” would plausibly work for a while and do superhumanly-helpful things for the programmers, before it starts doing superhumanly-bad things for the programmers.
Yup, if you don’t get corrigibility then you failed.
Regardless of overarching opinions you may or may not have about the unilateralist’s curse, I think Petrov Day is a uniquely bad time to lambast the foibles of being a well-intentioned unilateralist.
I worry that people are updating in exactly the wrong way from the events, possibly to fit preconceived ideas of what’s correct.
Independent impressions
Your independent impression about X is essentially what you’d believe about X if you weren’t updating your beliefs in light of peer disagreement—i.e., if you weren’t taking into account your knowledge about what other people believe and how trustworthy their judgement seems on this topic relative to yours. Your independent impression can take into account the reasons those people have for their beliefs (inasmuch as you know those reasons), but not the mere fact that they believe what they believe.
Armed with this concept, I try to stick to the following epistemic/discussion norms, and think it’s good for other people to do so as well:
Trying to keep track of my own independent impressions separately from my all-things-considered beliefs (which also takes into account peer disagreement)
Trying to be clear about whether I’m reporting my independent impression or my all-things-considered belief
Feeling comfortable reporting my own independent impression, even when I know it differs from the impressions of people with more expertise in a topic
One rationale for that bundle of norms is to avoid information cascades.
In contrast, when I actually make decisions, I try to make them based on my all-things-considered beliefs.
For example, my independent impression is that it’s plausible that a stable, global authoritarian regime, or some other unrecoverable dystopia, is more likely than extinction, and that we should prioritise those risks more than we currently do. But I think that this opinion is probably uncommon among people who’ve thought a lot about existential risks. And that makes me somewhat less confident in this opinion and somewhat less likely to actually act on it. But I think it’s still useful for me to keep track of my independent impression and report it sometimes, or else the community might end up with overly certain and overly homogenous beliefs.
This term and concept and these suggested norms aren’t at all original to me—see in particular Naming beliefs and several of the posts tagged Epistemic humility (especially this one). But I wanted a clear, concise description of this specific set of terms and norms so that I could link to it whenever I say I’m reporting my independent impression, ask someone for theirs, or ask someone whether an opinion they’ve given is their independent impression or their all-things-considered belief.
Thanks, I appreciate having something to link to! My independent impression is that it would be even easier to link to and easier to find as a top-level post.
Thanks for the suggestion—I’ve now gone ahead and made that top-level post :)
I just re-read this comment by Claire Zabel, which is also good and is probably where I originally encountered the “impressions” vs “beliefs” distinction.
(Though I still think that this shortform serves a somewhat distinct purpose, in that it jumps right to discussing that distinction, uses terms I think are a bit clearer—albeit clunkier—than just “impressions” vs “beliefs”, and explicitly proposes some discussion norms that Claire doesn’t quite explicitly propose.)
Content warning: missing persons, violence against women, racism.
Amid the media coverage of the Gabby Petito case in the United States, there’s been some discussion of how missing persons cases for women and girls of color are more neglected than those for missing White women. Some statistics:
Local vs. global optimization in career choice
Like many young people in the EA community, I often find myself paralyzed by career planning and am quick to second-guess my current path, developing an unhealthy obsession for keeping doors open in case I realize that I really should have done this other thing.
Many posts have been written recently about the pitfalls of planning your career as if you were some generic template to be molded by 80,000 Hours [reference Holden’s aptitudes post, etc.]. I’m still trying to process these ideas and think that the distinction between local and global optimization may help me (and hopefully others) with career planning.
Global optimization involves finding the best among all possible solutions. By its nature, EA is focused on global optimization, identifying the world’s most pressing problems and what we can do to solve them. This technique works well at the community level: we can simultaneously explore and exploit, transfer money between cause areas and strategies, and plan across long timescales. But global optimization is not as appropriate in career planning. Instead, perhaps it is better to think about career choice in terms of local optimization, finding the best solution in a limited set. Local optimization is more action-oriented, better at developing aptitudes, and less time-intensive.
The differences between global and local optimization are perhaps similar to the differences between sequence-based and cluster-based thinking [reference Holden’s post]. Like sequence-based thinking, which asks and answers questions with linear, expected-value style reasoning, global optimization is too vulnerable to subtle changes in parameters. Perhaps I’ve enrolled in a public health program but find AI safety and animal suffering equally compelling cause areas. Suppose I’m too focused on global optimization. In that case, a single new report by the Open Philantrophy Project suggesting shorter timelines for transformative AI might lead me to drop out of my program and begin anew as a software engineer. But perhaps the next day, I find out that clean meat is not as inevitable as I once thought, so I leave my job and begin studying bioengineering.
Global optimization makes us more likely to vacillate between potential paths excessively. The problem, though, is that we need some stability in our goals to make progress and develop the aptitudes necessary for impact in any field. Adding onto this the psychological stress of constant skepticism about one’s trajectory, it seems that global optimization can be a bad strategy for career planning. The alternative, local optimization, would have us look around our most immediate surroundings and do our best within that environment. Local optimization seems like a better strategy if we think that “good correlates with good,” and aptitudes are likely to transfer if we later become convinced that no, really, I should have done this other thing.
I think the difficult thing for us is to find the right balance between these two optimization techniques. We don’t want to fall into value traps or otherwise miss the forest for the trees, focusing too much on our most immediate options without considering more drastic changes. But too much global optimization can be similarly dangerous.
Redwood Research is looking for people to help us find flaws in our injury-detecting model. We’ll pay $20/hour for this, for up to 2 hours; after that, if you’ve found interesting stuff, we’ll pay you for more of this work. We might stop doing this (or stop recruiting additional people) in a few days.
If you’re interested, please email adam@rdwrs.com so he can add you to a Slack channel with other people who are working on this. This might be a fun task for people who like being creative, being tricky, and figuring out how language models understand language.
You can try out the interface here. The task is to find things that the model classifies as non-injurious that are actually injurious according to our definition. Full instructions here.
This is in service of this research project.
Correct me if I’m wrong, but I think in Christianity, there’s a lot of respect and positive affect for the “ordinary believer”. Christians who identify as “ordinary Christians” feel good about themselves for that fact. You don’t have to be among the brightest stars of the community in order to feel like you belong.
I think in EA, we’re extremely kind, but we somehow have less of this. Like, unless you have 2 PhD’s by the age of 25 and you’re able to hold your own in a conversation about AI-alignment theory with the top researchers in the world… you sadly have to “settle” for menial labour with impact hardly worth talking about. I’m overstating it, of course, but am I wrong?
I’m not saying ambition is bad. I think shooting for the stars is a great way to learn your limits. But I also notice a lot of people suffering under intellectual pressure, and I think we could collectively be more effective (and just feel better) if we had more… room for “ordinary folk dignity”?
My experience as a non-PhD who dropped out of EA things for two years before returning is that I felt welcome and accepted when I started showing up in EA spaces again. And now that I’ve been at CEA for three years, I still spend a lot of my time talking to and helping out people who are just getting started and don’t have any great credentials or accomplishments; I hope that I’m not putting pressure on them when I do this.
That said, every person’s experience is unique, and some people have certainly felt this kind of pressure, whether self-imposed as a result of perceived community norms or thrust upon them by people who were rude or dismissive at some point. And that’s clearly awful — people shouldn’t be made to feel this way in general, and it’s especially galling to hear about it sometimes happening within EA.
My impression is that few of these rude or dismissive people are themselves highly invested in the community, but my impression may be skewed by the relationships I’ve built with various highly invested people in the job I now have.
Lots of people with pretty normal backgrounds have clearly had enormous impact (too many examples to list!). And within the EA spaces I frequent, there’s a lot of interest and excitement about people sharing their stories of joining the movement, even if those people don’t have any special credentials. The most prominent example of this might be Giving What We Can.
I don’t understand the “menial labor” point; the most common jobs for people in the broader EA community are very white-collar (programmers, lawyers, teachers...) What did you mean by that?
Personally, the way I view “ordinary folk dignity” in EA is through something I call “the airplane test”. If I sat next to someone on an airplane and saw them reading Doing Good Better, and they seemed excited about EA when I talked to them, I’d be very happy to have met them, even if they didn’t have any special ambitions beyond finding a good charity and making occasional donations. There aren’t many people in the world who share our unusual collection of values; every new person is precious.
And to respond to your question about what I meant by “menial labour”. I was being poetic. I just mean that I feel like EA places a lot of focus on the very most high-status jobs, and I’ve heard friends despairing for having to “settle” for anything less. I sense that this type of writing might not be the norm for EA shortform, but I wasn’t sure.
Nono, I’m not trying to point to a problem of EAs trying to make others feel unwelcome or dumb. I think EA is extremely kind, and almost universally tries hard to make people feel welcome. I’m just pointing to the existence of an unusually strong intellectual pressure, perhaps combined with lots of focus on world-saving heroes and talk about “what should talented people do?”
I think ambition is good, but I think we can find ways of encouraging ambition while also mitigating at least some of the debilitating intelligence-dysphoria many in our community suffer from.
I’m writing this in reaction to talking to three of my friends who suffer under the intellectual pressure they feel. (Note that the following are all about the intellectual pressure they get from EA, and not just in general due to academic life.)
Friend1: “EA makes me feel real dumb XD i think i feel out of place by being less intelligent”
_
Friend2: “I’m not worried that I’m not smart, but I am worried that I am not smart enough to meet a certain threshold that is required for me to do the things I want to do. … I think I have very low odds of achieving things I deeply want to achieve. I think that is at least partially responsible for me being as extremely uncomfortable about my intelligence as I am, and not being able to snap out of it.”
_
Me: “Do you ever refrain from trying to contribute intellectually because you worry about taking up more attention than it’s worth?”
Friend3: “hmm, not really for that reason. because I’m afraid my contribution will be wrong or make me look stupid. wrong in a way that reflects negatively on me—stupid errors, revealing intellectual or character weakness.
_
Some of this is a natural and unavoidable result of the large focus EA places on intellectual labour, but I think it’s worse than it needs to be. I think some effort to instil some “ordinary EA dignity” into our culture wouldn’t hurt. I might have a skewed sample, however.
I think an EA career fair would be a good idea. It could have EA orgs as well as non-EA orgs that are relevant to EAs (for gaining career capital or earning to give)
EA Global normally has an EA career fair, or something similar
Naïve question: What’s the deal with the cheapest CO2 offset prices?
It seems, though, that the current price of credible offsets is much lower than the social cost of carbon, and possibly so low that just buying offsets starts to look competitive with GiveWell top charities.
I’m not an expert on this. (I run an offsetting program for a small organization, but that takes about 4h/year. Otherwise I don’t think about this much.) I’m also not anywhere near advocating that we should sink tons of money into offsets. But this observation strikes me as unintuitive enough that I suspect I’m missing something.
Cost of offsetting:
I’ve generally seen the UN FCCC’s carbon offset market presented as credible, if not the biggest or most scalable. They make it pretty easy direct money to specific projects, and most of them pass the smell test as being both verifiable and additive. One project that popped up last year involved converting the operations of a platinum mining company in Bihar from burning coal to another burning another fossil fuel in a slightly-lower-emissions way. That’s easy to verify, and there was a clear argument for why it wouldn’t make economic sense for them to transition without the offset money.
Current prices on that market are around $1.75/tonne, and at other times of year recently, I’ve seen numbers dip as low as ~$0.33/tonne.
Value of offsetting:
My impression is that the social cost of carbon is somewhere in the $30–300/tonne range, suggesting that cheap credible offsets are plausibly pretty high-leverage, but I don’t have good framework for thinking about this kind of impact.
I found this easier to visualize in light of this line from a recent Future Perfect piece:
Combining that with the numbers above yields a price-per-life-saved in the $1k-10k range, which is within the same order of magnitude as charities like AMF, IIRC.
So?
What am I missing? Is there something fishy with that use of of dollars per life saved? Are these cheap UN-monitored offsets actually bullshit? Or is it actually just very impactful to buy this kind of offset while they’re still cheap (...even if it’s still not the most effective way to spend money, or even the most effective way to spend money on climate impacts)?
I am also confused about the general question, but I found this intervention interesting to think about. It seems like the legitimacy of this comes down to the elasticity of demand for coal in India (basically, if someone buys 1 ton less of coal, will someone else buy that same ton for a lower price, or will coal producers make one ton less?). I couldn’t find any data on elasticity of demand for coal in India, but this paper estimates it for China as 0.3 to 0.7, which is maybe an OK proxy? And I don’t know if it’s reasonable to model the elasticity of demand of coal and of the other fossil fuel as the same (eg, it would be terrible if not buying 1 ton of coal reduces the total coal by 0.2 tons produced, but buying 1 ton of oil increases total oil produced by 0.8 tons).
Overall it feels non-obvious to me whether it’s legit, though I lean towards “probably, but about half as effective as a naive calculation suggests”
Collection of collections of resources relevant to (research) management, mentorship, training, etc.
(See the linked doc for the most up-to-date version of this.)
The scope of this doc is fairly broad and nebulous. This is not The Definitive Collection of collections of resources on these topics—it’s just the relevant things that I (Michael Aird) happen to have made or know of.
Management & mentoring—EA Forum
Management-related books [shared] - me
Meeting templates for mentors/managers [shared] - me
Goal-setting templates or processes [shared] - me
Readings and notes on how to do high-impact research—me
Readings and notes on how to do high-quality, efficient research—me
Readings and notes on how to write/communicate well—me
Resources—Effective Thesis
Tips On Doing Impactful Research—Effective Thesis
SERI Research Proposal Generation Tips 2021 - SERI
Research Training Programs—EA Forum Wiki
Review of SRF 2019 - Max Daniel and/or Rose Hadshar of FHI
See also Review of FHI’s Summer Research Fellowship 2020
Research project planning templates/resources [shared]
Resource library of the Management Center
This is a non-EA nonprofit that provides management training and resources to other nonprofits
I and several other Rethink Priorities staff did their management training. I thought it was fairly useful (though with some flaws).
Some quick reflections on mentoring from Max Daniel
Here are some things that are “internal” or that I don’t have permission to share, but that I might be able to make shareable versions of, share after asking permission, or talk to people about, if someone is interested::
Summary and commentary on the book Managing to Change the World—someone at RP
“Management / org structure things RP could consider”—me
“Aspects of CLR’s SRF which RP’s internship could draw on”—me
My Anki cards on Getting Things Done—me
“RSP Personal Review: Meta doc”—someone who was at FHI
Performance review templates collected at the bottom of the FHI doc “Performance feedback—Meta doc”—someone who was at FHI
Quick notes on management—someone at RP
Finally, here are my notes on relevant books (though each of these links isn’t really a “collection”):
Notes on Mochary’s “The Great CEO Within” (2019)
[more to be added later]
Collection of AI governance reading lists, syllabi, etc.
This is a doc I made, and I suggest reading the doc rather than shortform version (assuming you want to read this at all). But here it is copied out anyway:
What is this doc, and why did I make it?
AI governance is a large, complex, important area that intersects with a vast array of other fields. Unfortunately, it’s only fairly recently that this area started receiving substantial attention, especially from specialists with a focus on existential risks and/or the long-term future. And as far as I’m aware there aren’t yet any canonical, high-quality textbooks or online courses on the topic.[1] It seems to me that this means this is an area where well-curated and well-structured reading lists, syllabi, or similar can be especially useful, helping to fill the role that textbooks otherwise could.[2]
Fortunately, when I started looking for relevant reading lists and syllabi, I was surprised by how many there were. So I decided to try to collect them all in one place. I also tried to put them in very roughly descending order of how useful I’d guess they’d be to a randomly chosen EA-aligned person interested in learning about AI governance.
I think this might help myself, my colleagues, and others who are trying to “get up to speed”, for the reasons given in the following footnote.[3]
I might later turn this doc into a proper post on the EA Forum.
See also EA syllabi and teaching materials and Courses on longtermism.
How can you help
Please comment if you know of anything potentially relevant which I haven’t included!
Please comment if you have opinions on anything listed!
The actual collection
September AGI safety fundamentals curriculum—Richard Ngo
Alignment Newsletter Database—Rohin Shah
This is more relevant to technical AI safety than to AI governance, but some categories are pretty relevant to AI governance, especially “AI strategy and policy”, “Forecasting”, and “Field building”
AI Governance Reading List—SERI 2021 Summer—Mauricio Baker
Mauricio had also previously made a syllabus on the same topics: AI Governance Syllabus ’21.docx
Governance of AI Reading List – Oxford Spring 2020 - Markus Anderljung
Reading Guide for the Global Politics of Artificial Intelligence—Allan Dafoe
I’m guessing other lists made by people associated with GovAI already draw on and superseded this, but I don’t know
“Resources” section from Guide to working in artificial intelligence policy and strategy − 80,000 Hours
Note: I think the only book from there that’s available on Audible UK is The Second Machine Age.
But the description of the book sounds to me kind-of basic and not especially longtermism-relevant.
AI policy introductory reading list—Niel Bowerman (I think)
Governance of AI—Some suggested readings [v0.5, shared] - Ashwin Acharya
Drawn on for SERI’s reading list
Artificial Intelligence and International Security Syllabus [public] - Remco Zwetsloot, 2018 (I think)
Books and lecture series relevant to AI governance—me and commenters
Section on “Unaligned artificial intelligence” from Syllabus — The Precipice
Tangential critique: I personally think that it’s problematic and misleading that both The Precipice and this syllabus use the heading “unaligned artificial intelligence” while seeming to imply that this covers all key aspects of AI risk, since I think this obscures some risk pathways.
AI Policy Readings Draft.docx—EA Oxford
Drawn on for SERI’s reading list
My post Crucial questions for longtermists includes a structured list of questions related to the “Value of, and best approaches to, work related to AI”, and this associated doc contains readings related to each of those questions
I haven’t updated this much since 2020
Questions listed there include:
Is it possible to build an artificial general intelligence (AGI) and/or transformative AI (TAI) system? Is humanity likely to do so?
What form(s) is TAI likely to take? What are the implications of that? (E.g., AGI agents vs comprehensive AI services)
What will the timeline of AI developments be?
How much should longtermists’ prioritise AI?
What forms might an AI catastrophe take? How likely is each?
What are the best approaches to reducing AI risk or increasing AI benefits?
Good resources for getting a high-level understanding of AI risk—Michael Aird
AI governance intro readings—Felipe Calero
Luke Muehlhauser’s 2013 and 2014 lists of books he’d listened to recently
I think many/most of these books were chosen for “seem[ing] likely to have passages relevant to the question of how well policy-makers will deal with AGI”
Many/most of these aren’t available as audiobooks; Luke turned them into audiobooks himself
A Contra AI FOOM Reading List – Magnus Vinding
Described in SERI’s reading list as a “List of arguments (of varied quality) against ‘fast takeoff’”
List of resources on AI and agency—Ben Pace
You could also use research agendas related to AI governance as reading lists, by following the sources they cite on various topics. Relevant agendas include:
(Note that I haven’t checked how well each of these agendas would work for this purpose. This list is taken from my central directory for open research questions.)
The Centre for the Governance of AI’s research agenda − 2018
Some AI Governance Research Ideas—the Centre for the Governance of AI, 2021
Promising research projects—AI Impacts, 2018
They also made a list in 2015; I haven’t checked how much they overlap
Cooperation, Conflict, and Transformative Artificial Intelligence (the Center on Long-Term Risk’s research agenda) - Jesse Clifton, 2019
Open Problems in Cooperative AI—Dafoe et al., 2020
Problems in AI Alignment that philosophers could potentially contribute to—Wei Dai, 2019
Problems in AI risk that economists could potentially contribute to—Michael Aird, 2021
Technical AGI safety research outside AI—Richard Ngo, 2019
Artificial Intelligence and Global Security Initiative Research Agenda—Centre for a New American Security, no date
A survey of research questions for robust and beneficial AI—Future of Life Institute, no date
“studies which could illuminate our strategic situation with regard to superintelligence”—Luke Muehlhauser, 2014 (he also made a list in 2012)
A shift in arguments for AI risk—Tom Sittler, 2019
Longtermist AI policy projects for economists—Risto Uuk (this doc was originally just made for Risto’s own use, so the ideas shouldn’t be taken as high-confidence recommendations to anyone else)
Annotated Bibliography of Recommended Materials—CHAI
I think this is much more focused on technical AI safety than AI governance
Some Rethink Priorities staff may soon make a long, tiered reading list tailored to the AI governance project ideas we may work on. If it seems to me that this would be useful to other people, I might add a link to a version of it here.
There may be additional relevant reading lists / syllabi / sections in the links given here: EA syllabi and teaching materials—EA Forum
And here Courses on longtermism | Pablo’s miscellany
I think there was also a short reading list associated with the EA In-Depth Fellowship
A related category to reading lists is newsletters that provide summaries and commentary of a bunch of research outputs. E.g.:
Rohin’s
Jack Clarke’s
CSET’s
…
Mauricio Baker suggested that I or people reading this doc might also be interested in “syllabi aimed at aspiring AI technical safety researchers, such as this one: Technical AI Safety Reading List. I have a vague sense that engaging with some of this content has been helpful for my having a better broad sense of what’s going on with AI safety, which seems helpful for governance.”
Some parts of Krakovna’s AI safety resources and Maini’s AI Reading List may be quite useful for AI governance people, though I think they’re more relevant for technical AI safety people
My thanks to everyone who made these lists, as well as to Mauricio Baker for pointing me to some of the lists.
Footnotes
[1] Though there are various presumably high-quality textbooks or courses with some relevance, some high-quality non-textbook books on the topic, some in-person courses that might be high-quality (I haven’t participated in them), and some things that fill somewhat similar roles (like EA seminar series, reading groups, or fellowships).
[2] See also Research Debt and Suggestion: EAs should post more summaries and collections.
[3]
This collection should make it easier to find additional reading lists, syllabi, etc., and thus easier to find additional readings that have been evaluated as especially worth reading in general, especially worth reading on a given topic, and/or especially good as introductory resources.
This collection should make it easier to find and focus on reading lists, syllabi, etc. that are better and/or more relevant to one’s specific needs.
To help with this, please comment on this doc if you have opinions about anything listed.
Even before or without engaging with the actual items included in a given reading list, syllabus, or similar, engaging with the structure and commentary in that document itself could help one understand what the important components, divisions, concepts, etc. within AI governance are. And this collection should help people find more, better, and/or more relevant such documents.
I think it is a cool idea for people to take a giving pledge on the same day. For example, you and your friend both decide to pledge 10% to charity on the same day. It would be even more fun if you did it with strangers. Call it “giving twins” or “giving siblings”.
Imagine that you met a couple strangers and they pledged with you. Imagine that after pledging you all just decided to be friends, or at least a type of support group for one another. Like “Hey, you and I took the further pledge together on New Year’s Day last year. When I’m in your city let’s go have a pint or maybe we can email each other about our career plans to discuss ways we can help people more!”
Which leads me to my final bit here: would anyone be interested in being my giving sibling? Haha I am interested in taking the further pledge in 2022, and it would be fun to have the same ‘giving birthday’ as other people so I could befriend them, meet people in the community, and get a couple lifelong friends.
Giving what we can could even take this idea and run with it. They could assign you a giving sibling if you entered into the sibling program, this could help increase the feeling that we are a community.
does giving sibling mean giving somethin g to each other?
That could be the case, but I think the emphasis is more on the idea that you have the same “birthdate” to be considered a giving sibling.
Like on February 15 you and a friend took the Giving Pledge together and then that date was the same day you became siblings. Then you celebrate that day every year or form a bond around this shared experience.
Hi, I am working on a non profit to help animals in another country by creating a sanctuary for them. I already know how to setup a corporation, convert it to a non profit, and operate it with a board of directors. For this project, I will be opening a US non-profit and using funds to help animals in other countries.
I am looking for additional guides on how to establish a non-profit organization in another country and wanted to know if there is anything different about running a US non profit that does work in another country and if the process is any different when operating only in the US.
I wrote a very rough draft of an idea I had. It was just a stream of consciousness and I didn’t really edit it. I’m not sure what the standards are like on the EA Forum: I would like to invest time in developing this further, as well as other posts I could possibly write, but as I mentioned at the bottom of this post, I’m at a critical juncture in my career and need to invest my time and energy elsewhere.
If I can produce forum posts with potentially some interesting ideas, but that are of a relatively low standard, is is better for me to post these without worrying about developing them, or waiting until I have more time to invest in developing better forum posts/writing less and editing more?
Epistemic status: medium-low (I have some conviction in this idea, but I’m posting this without much rework in an attempt to overcome perfectionism, partially inspired by Neel Nanda. This likely means that some of the things in this post will be wrong, but I hope it could be useful nonetheless).
Summary: we should systematically proceduralise activities more. Learning how to proceduralise tasks requires a lot of up-front investment. This could be reduced by centralised, shared ‘checklists’ for different tasks which are designed to increase effectiveness in particular tasks.
I’ve always had a preference for proceduralising (approaching tasks in a systematised manner), but my exposure to EA and the rationalist community has inundated me with high quality information about how to do effective research, learn effectively, prioritise effectively and make better decisions etc.
The bottleneck holding back my progress is now the implementation of ideas that I have high conviction in. To effectively implement these ideas and integrate them into my system 2 would require a lot of deliberate practice and highly engaged, focussed efforts.
I want to somehow outsource or expedite this process, and this seems like an important issue that is likely to be affecting other EAs as well. One potential solution is the more deliberate development of effective checklists, ideally some kind of centralised database.
Checklist have been advocated by Charlie Munger (Warren Buffet’s business partner) as an effective way of making better decisions and overcoming biases. They help ensure that we prime our brain to consider different important aspects of a problem that we might not naturally consider (discussed more thoroughly in https://www.amazon.de/-/en/Atul-Gawande/dp/0312430000).
My mental model of highly skilled workers in a complex field is essentially people who have internalised complex procedures internally, but that with some effort these mental processes could be proceduralised to a greater degree. Checklists do not have to involve mindlessly going through a recipe book: they could involve open ended questions designed to elicit deeper thinking.
Implementation
I’m still in the early phases of developing this idea, but I’ve started to implement it as part of my workflow. I use OneNote and have sections entitled ‘Working Best Practices’, ‘Research Best Practices’ etc. As an example, here’s something taken the page ‘when reading a research paper’ (I stole the questions from EA-related sources, mostly from: https://effectivethesis.org/resources/)
Short:
Main question
Methods used
Data required
Results
Long:
Can the approach used in a study be applied to other contexts, countries or time periods?
What assumptions are implicit?
Are all the assumptions sensible?
To what extent might the results be sensitive to the assumptions?
How can they be relaxed?
Are there any unnecessary assumptions?
Is the approach used the most appropriate one?
Have new techniques been developed since the paper was first written?
Have all relevant statistical tests been carried out?
Are the results consistent with expectations, or earlier work?
Are the surrogate or constructed variables the most appropriate for the task and can anything be said about likely biases?
Are there any implications of the study which have not been fully drawn out by the author?
Can these be exploited in your work?
With a checklist, rather than having to learn all of the questions to ask yourself when reading a paper, you just have to remember where to find these questions, which hopefully frees up time to focus on other things.
I imagine a centralised EA checklist, with different sections differentiated by clear cues. A section would have an inbox, where people add in ideas with low epistemic standards, to increase creativity. There would then be two additional aspects to the section: a short version and a long version. The short version would highlight the best 4 parts of a checklist to follow, the long version would highlight the best 20, with voting to help ensure that the best ones rise to the top. Comments and feedback would also be useful as a way of developing our understanding of the pitfalls of certain procedures.
This approach might work well for activities where the order of actions is not important. However in many cases it is. Perhaps then a better solution would be for users to submit either individual items or whole procedures. Users would vote on individual procedures, which would be collated into a second inbox. From here, users could bundle procedures together to create ‘packages’, which could then be voted on.
These are early stage ideas and could almost certainly be improved upon. I would be happy to help develop this but I have limited programming experience so would struggle to implement as-hoc features. If anyone was interested in helping me implement this that would be great! I do think this has the potential to multiply EAs impact.
Caveats and downside risk: it’s possible that these checklists would act as a mental crutch and undermine the long-term development of people using them: learning the procedures of a task and internalising this kind of knowledge may facilitate greater creativity and better work in the long-run, and developing these checklists could end up being a distracting diversion for EAs and not a good use of time.
I think it’s worthwhile proceeding despite these risks: if they end up being true, I hope that this would be noticed by EAs, and users could be encouraged to learn the checklists rather than blindly apply them.
Another risk is that this could result in EAs approaching problems in a similar way, which could stymie productivity in the long run. This could be mitigated by having another section with more creative ways of approaching problems. But I think the gains to optimisation are so large that they’re much more important than the trade off between creativity and proceduralisation. The ‘efficient frontier’ comes to mind. If we reach a point where EAs are optimising consistently to a high degree, then we can start having more conversations about potential trade-offs.
Implementation
This is something I would be interested in taking on at some point, but I’m at a critical juncture in my career right now and need to devote my time and energy to figuring out my next steps (likely applying for Economics PhDs).
I haven’t yet looked into how this could be effectively implemented: it seems likely that there are online platforms that could incorporate this, but I don’t have much expertise in the field.
If anyone is interested in collaborating on making this happen, please comment ‘interested’.
In answer to your question, I think that it’s generally better to create a top-level post than a Shortform post, as long as you’re comfortable doing so. Shortform serves a useful purpose, but top-level posts have a better chance of getting useful engagement.
I think this would make sense as a top-level post, and would encourage you to try sharing it in that context!
My thoughts on the checklist compilation idea: It’s hard to make a given intellectual resource very popular, but if you can pull off this one, I think it could be really useful. Many EA orgs have their own private procedural checklists; just taking those and bringing them out into the public could be valuable on its own. (For example, I have a checklist for creating email newsletters in Mailchimp that I think could be useful for others who do similar things.)
Red teaming papers as an EA training exercise?
I think a plausibly good training exercise for EAs wanting to be better at empirical/conceptual research is to deep dive into seminal papers/blog posts and attempt to identify all the empirical and conceptual errors in past work, especially writings by either a) other respected EAs or b) other stuff that we otherwise think of as especially important.
I’m not sure how knowledgeable you have to be to do this well, but I suspect it’s approachable for smart people who finish high school, and certainly by the time they finish undergrad^ with a decent science or social science degree.
I think this is good career building for various reasons:
you can develop a healthy skepticism of the existing EA orthodoxy
I mean skepticism that’s grounded in specific beliefs about why things ought to be different, rather than just vague “weirdness heuristics” or feeling like the goals of EA conflict with other tribal goals.
(I personally have not found high-level critiques of EA, and I have read many, to be particularly interesting or insightful, but this is just a personal take).
you actually deeply understand at least one topic well enough to point out errors
For many people and personalities, critiquing a specific paper/blog post may be a less hairy “entry point” into doing EA-research adjacent work than plausible alternatives like trying to form your own deep inside views on questions that are really open-ended and ambiguous like “come up with a novel solution in AI alignment” or “identify a new cause X”
creates legible career capital (at least within EA)
requires relatively little training/guidance from external mentors, meaning
our movement devotes less scarce mentorship resources into this
people with worse social skills/network/geographical situation don’t feel (as much) at a disadvantage for getting the relevant training
you can start forming your own opinions/intuitions of both object-level and meta-level heuristics for what things are likely to be correct vs wrong.
In some cases, the errors are actually quite big, and worth correcting (relevant parts of ) the entire EA movement on.
Main “cons” I can think of:
I’m not aware of anybody successfully doing a really good critique for the sake of doing a really good critique. The most exciting things I’m aware of (publicly, zdgroff’s critique of Ng’s original paper on wild animal suffering, alexrjl’s critique of Giving Green. I also have private examples) mostly comes from people trying to deeply understand a thing for themselves, and then along the way spotting errors with existing work.
It’s possible that doing deliberate “red-teaming” would make one predisposed to spot trivial issues rather than serious ones, or falsely identify issues where there aren’t any.
Maybe critiques are a less important skill to develop than forming your own vision/research direction and executing on it, and telling people to train for this skill might actively hinder their ability to be bold & imaginative?
^ Of course, this depends on field. I think even relatively technical papers within EA are readable to a recent undergrad who cares enough, but this will not be true for eg (most) papers in physics or math.
This is another example of a Shortform that could be an excellent top-level post (especially as it’s on-theme with the motivated reasoning post that was just published). I’d love to see see this spend a week on the front page and perhaps convince some readers to try doing some red-teaming for themselves. Would you consider creating a post?
This idea sounds really cool. Brainstorming: a variant could be several people red teaming the same paper and not conferring until the end.
One additional risk: if done poorly, harsh criticism of someone else’s blog post from several years ago could be pretty unpleasant and make the EA community seem less friendly.
I’m actually super excited about this idea though—let’s set some courtesy norms around contacting the author privately before red-teaming their paper and then get going!
I think I agree this is a concern. But just so we’re on the same page here, what’s your threat model? Are you more worried about
The EA community feeling less pleasant and friendly to existing established EAs, so we’ll have more retention issues with people disengaging?
The EA community feeling less pleasant and friendly to newcomers, so we have more issues with recruitment and people getting excited to join novel projects?
Criticism makes being open about your work less pleasant, and open Red Teaming about EA projects makes EA move even further in the direction of being less open than we used to be. See also Responsible Transparency Consumption.
Something else?
It’s actually a bit of numbers 1-3; I’m imagining decreased engagement generally, especially sharing ideas transparently.
Thanks for the excitement! I agree that contacting someone ahead of time might be good (so at least someone doesn’t learn about their project being red teamed until social media blows up), but I feel like it might not mitigate most of the potential unpleasantness/harshness. Like I don’t see a good cultural way to both incentivize Red Teaming and allow a face-saving way to refuse to let your papers be Red Teamed.
Like if Red Teaming is opt-in by default, I’d worry a lot about this not taking off the ground, while if Red Teaming is opt-out by default I’d just find it very suss for anybody to refuse (speaking for myself, I can’t imagine ever refusing Red Teaming even if I would rather it not happen).
Easy steps could be to add a “red team” tag on the forum and point to this post to encourage people to do this.
I have at times given advice to early career EA’s mostly in AI Safety similar to this. When people have trouble coming up with something they might want to write about on the forum, I encourage them to look for the things they don’t think are true. Most people are passively reading the forum anyway but actively looking for something the reader doesn’t think is true or is unconvinced by can be a good starting point for a post. It may be that they end up convinced of the point but can still write a post making is clearer and adding the arguments they found.
Having said this, most peoples first reaction is a terrified look. Encouraging someone’s first post to be a criticism is understandably scary.
It may be hard to get both the benefit to the participants and to the orgs. Anyone not intimidated by this might already have enough experience and career capital. To give juniors the experience you might have to make it more comfortable school work where the paper is written but only read by one other person. This makes it harder to capture the career capital.
I’d expect this to be unlikely for someone to do individually and of their own accord. At the very least best to do this in small groups to create social accountability and commitment pressures. While also defusing the intimidation. Alternately part of an existing program like an EA Fellowship. Even better as it’s own program, with all the overhead that comes with that.
I would be very excited about someone experimenting with this and writing up the results. (And would be happy to provide EAIF funding for this if I thought the details of the experiment were good and the person a good fit for doing this.)
If I had had more time, I would have done this for the EA In-Depth Fellowship seminars I designed and piloted recently.
I would be particularly interested in doing this for cases where there is some amount of easily transmissible ‘ground truth’ people can use as feedback signal. E.g.
You first let people red-team deworming papers and then give them some more nuanced ‘Worm Wars’ stuff. (Where ideally you want people to figure out “okay, despite paper X making that claim we shouldn’t believe that deworming helps with short/mid-term education outcomes, but despite all the skepticism by epidemiologists here is why it’s still a great philanthropic bet overall”—or whatever we think the appropriate conclusion is.)
You first let people red-team particular claims about the effects on hen welfare from battery cages vs. cage-free environments and then you show them Ajeya’s report.
You first let people red-team particular claims about the impacts of the Justinian plague and then you show them this paper.
You first let people red-team particular claims about “X is power-law distributed” and then you show them Clauset et al., Power-law distributions in empirical data.
(Collecting a list of such examples would be another thing I’d be potentially interested to fund.)
Hmm I feel more uneasy about the truthiness grounds of considering some of these examples as “ground truth” (except maybe the Clauset et al example, not sure). I’d rather either a) train people to Red Team existing EA orthodoxy stuff and let their own internal senses + mentor guidance decide whether the red teaming is credible or b) for basic scientific literacy stuff where you do want clear ground truths, let them challenge stuff that’s closer to obvious junk (Why We Sleep, some climate science stuff, maybe some covid papers, maybe pull up examples from Calling Bullshit, which I have not read).
That seems fair. To be clear, I think “ground truth” isn’t the exact framing I’d want to use, and overall I think the best version of such an exercise would encourage some degree of skepticism about the alleged ‘better’ answer as well.
Assuming it’s framed well, I think there are both upsides and downsides to using examples that are closer to EA vs. clearer-cut. I’m uncertain on what seemed better overall if I could only do one of them.
Another advantage of my suggestion in my view is that it relies less on mentors. I’m concerned that having mentors that are less epistemically savvy than the best participants can detract a lot from the optimal value that exercise might provide, and that it would be super hard to ensure adequate mentor quality for some audiences I’d want to use this exercise for. Even if you’re less concerned about this, relying on any kind of plausible mentor seems like less scaleable than a version that only relies on access to published material.
Upon (brief) reflection I agree that relying on the epistemic savviness of the mentors might be too much and the best version of the training program will train a sort of keen internal sense of scientific skepticism that’s not particularly reliant on social approval.
If we have enough time I would float a version of a course that slowly goes from very obvious crap (marketing tripe, bad graphs) into things that are subtler crap (Why We Sleep, Bem ESP stuff) into weasely/motivated stuff (Hickel? Pinker? Sunstein? popular nonfiction in general?) into things that are genuinely hard judgment calls (papers/blog posts/claims accepted by current elite EA consensus).
But maybe I’m just remaking the Calling Bullshit course but with a higher endpoint.
___
(I also think it’s plausible/likely that my original program of just giving somebody an EA-approved paper + say 2 weeks to try their best to Red Team it will produce interesting results, even without all these training wheels).
I think your cons are good things to have noted, but here are reasons why two of them might matter less than one might think:
I think the very fact that “It’s possible that doing deliberate “red-teaming” would make one predisposed to spot trivial issues rather than serious ones, or falsely identify issues where there aren’t any” could actually also make this useful for skill-building and testing fit; people will be forced to learn to avoid those failure modes, and “we” (the community, potential future hirers, etc.) can see how well they do so.
E.g., to do this red teaming well, they may have to learn to identify how central an error is to a paper/post’s argument, to think about whether a slightly different argument could reach the same conclusion without needing the questionable premise, etc.
I have personally found that the line between “noticing errors in existing work” and “generating novel research” is pretty blurry.
A decent amount of the research I’ve done (especially some that is unfortunately nonpublic so far) has basically followed the following steps:
“This paper/post/argument seems interesting and important”
“Oh wait, it actually requires a premise that they haven’t noted and that seems questionable” / “It ignores some other pathway by which a bad thing can happen” / “Its concepts/definitions are fuzzy or conflate things in way that may obscure something important”
[I write a post/doc that discusses that issue, provides some analysis in light of this additional premise being required or this other pathway being possible or whatever, and discussing what implications this has—e.g., whether some risk is actually more or less important than we thought, or what new intervention ideas this alternative risk pathway suggests might be useful]
Off the top of my head, some useful pieces of public work by other people that I feel could be roughly described as “red teaming that turned into novel research” include A Proposed Adjustment to the Astronomical Waste Argument and The long-term significance of reducing global catastrophic risks
I’d guess that the same could also sometimes happen with this red teaming, especially if that was explicitly encouraged, people were given guidance on how to lean into this more “novel research” element when they notice something potentially major during the red teaming, people were given examples of how that has happened in the past, etc.
This also reminds me of a recent shortform by Buck:
(I think the full shortform and the comments below it are also worth reading.)
Strong upvote for a idea that seems directly actionable and useful for addressing important problem.
I’m gonna quote your shortform in full (with a link and attribution, obviously) in a comment on my post about Intervention options for improving the EA-aligned research pipeline.
I think by default good ideas like this never really end up happening, which is sad. Do you or other people have thoughts on how to make your idea actually happen? Some quick thoughts from me:
Just highlight this idea on the Forum more often/prominently
People giving career advice or mentorship to people interested in EA-aligned research careers mention this as one way of testing fit, having an impact, etc.
I add the idea to Notes on EA-related research, writing, testing fit, learning, and the Forum [done!]
Heap an appropriate amount of status and attention on good instances of this having been done
That requires it to be done at least once first, of course, but can then increase the rate
E.g., Aaron Gertler could feature it in the EA Forum Digest newsletter, people could make positive comments on the post, someone can share it in a Facebook group and/or other newsletter
I know I found this sort of thing a useful and motivating signal when I started posting stuff (though not precisely this kind of stuff)
Publicly offer to provide financial prizes for good instances of this having been done
One way to do this could mirror Buck’s idea for getting more good book reviews to happen (see my other comment): “If it’s the kind of review I want, I give them $500 in return for them posting the review to EA Forum or LW with a “This post sponsored by the EAIF” banner at the top. (I’d also love to set up an impact purchase thing but that’s probably too complicated).”
Find case studies where someone found such a post useful or having written it helped someone get a good job or something, and then publicise those
See also my thoughts on discovering, writing, and/or promoting case studies of people successfully using interventions for improving the EA-aligned research pipeline
Another possible (but less realistic?) way to make this happen:
Organisations/researchers do something like encouraging red teaming of their own output, setting up a bounty/prize for high-quality instances of that, or similar
An example of something roughly like this is a post on the GiveWell blog that says at the start: “This is a guest post by David Barry, a GiveWell supporter. He emailed us at the end of December to point out some mistakes and issues in our cost-effectiveness calculations for deworming, and we asked him to write up his thoughts to share here. We made minor wording and organizational suggestions but have otherwise published as is; we have not vetted his sources or his modifications to our spreadsheet for comparing deworming and cash. Note that since receiving his initial email, we have discussed the possibility of paying him to do more work like this in the future.”
But I think GiveWell haven’t done that since then?
It seems like this might make sense and be mutually beneficial
Orgs/researchers presumably want more ways to increase the accuracy of their claims and conclusions
A good red teaming of their work might also highlight additional directions for further research and surface someone who’d be a good employee for that org or collaborator for that researcher
Red teaming of that work might provide a way for people to build skills and test fit for work on precisely the topics that the org/researcher presumably considers important and wants more people working on
But I’d guess that this is unlikely to happen in this form
I think this is mainly due to inertia plus people feeling averse to the idea
But there may also be good arguments against
This post is probably relevant: https://forum.effectivealtruism.org/posts/gTaDDJFDzqe7jnTWG/some-thoughts-on-public-discourse
Another argument against is that, for actually directly improving the accuracy of some piece of work, it’s probably more effective to pay people who are already know to be good at relevant work to do reviewing / red-teaming prior to publication
Yeah I think this is key. I’m much more optimistic about getting trainees to do this being a good training intervention than a “directly improve research quality” intervention. There are some related arguments why you want to pay people who are either a) already good at the relevant work or b) specialized reviewers/red-teamers
paying people to criticize your work would risk creating a weird power dynamic, and more experienced reviewers would be better at navigating this
For example, trainees may be afraid of criticizing you too harshly.
Also, if the critique is in fact bad, you may be placed in a somewhat awkward position when deciding whether to publish/publicize it.
Thanks for linking my idea in your sequence! (onlookers note: MichaelA and I are coworkers)
This arguably happened to alexrjl’s critique of Giving Green, though it was a conjunction of a critique of an organization and a critique of research done.
As an aside, I decided to focus my shortform on critiques of public research rather than critiques of organizations/people, even though I think the latter is quite valuable too, since a) my intuition is that the former is less acrimonious, b) relatedly, critiques of organizations may be worse at training dispassionate analysis skills (vs eg tribalistic feelings or rhetoric), c) critiques of orgs or people might be easier for newbies to fuck up and d) I think empirically, critiques of organizations have a worse hit rate than critiques of research posts.
As you know, one of my interns is doing something adjacent to this idea (though framed in a different way), and I may convince another intern to do something similar (depending on their interests and specific project ideas in mind).
Yeah, good point—I guess a more directed version of “People giving career advice or mentorship to people interested in EA-aligned research careers mention this as one way of testing fit, having impact, etc.” is just people encouraging people they manage to do this, or maybe even hiring people with this partly in mind.
Though I think that that wouldn’t capture most of the potential value of this idea, since part of what’s good about is that, as you say, this idea:
(People who’ve already gone through a hiring process and have an at least somewhat more experienced researcher managing them will have an easier time than other people in testing fit, having impact, building skills, etc. in other ways as well.)
Yeah I agree that a major upside to this idea (and a key differentiator between it and other proposed interventions for fixing early stages of the research pipeline) is that it ought to be doable without as much guidance from external mentors. I guess my own willingness to suggest this as an intern project suggests that I believe it must comparatively be even more exciting for people without external guidance.
When I was 19, I moved to San Francisco to do a coding bootcamp. I got a bunch better at Ruby programming and also learned a bunch of web technologies (SQL, Rails, JavaScript, etc).
It was a great experience for me, for a bunch of reasons.
I got a bunch better at programming and web development.
It was a great learning environment for me. We spent basically all day pair programming, which makes it really easy to stay motivated and engaged. And we had homework and readings in the evenings and weekends. I was living in the office at the time, with a bunch of the other students, and it was super easy for me to spend most of my waking hours programming and learning about web development. I think that it was very healthy for me to practice working really long hours in a supportive environment.
The basic way the course worked is that every day you’d be given a project with step-by-step instructions, and you’d try to implement the instructions with your partner. I think it was really healthy for me to repeatedly practice the skill of reading the description of a project, then reading the step-by-step breakdown, and then figuring out how to code everything.
Because we pair programmed every day, tips and tricks quickly percolated through the cohort. We were programming in Ruby, which has lots of neat little language features that it’s hard to pick up all of on your own; these were transmitted very naturally. I also was pushed to learn my text editor better.
The specific content that I learned was sometimes kind of fiddly; it was helpful to have more experienced people around to give advice when things went wrong.
I think that this was probably a better learning experience than most tech or research internships I could have gotten. If I’d had access to the best tech/research internships, maybe that would have been better. I think that this was probably a much better learning experience than eg most Google internships seem to be.
I met rationalists and EAs in the Bay.
I spent a bunch of time with real adults who had had real jobs before. The median age of students was like 25. Most of the people had had jobs before and felt dissatisfied with them and wanted to make a career transition. I think that spending this time with them helped me grow up faster.
I somehow convinced my university that this coding bootcamp was a semester abroad (thanks to my friend Andrew Donnellan for suggesting this to me; that suggestion plausibly accelerated my career by six months), which meant that I graduated on schedule even though I then spent six months working for App Academy as a TA (which incidentally was also a good experience.)
Some ways in which my experience was unusual:
I was a much stronger programmer on the way in to the program than most of my peers.
I am deeply extroverted and am fine with pair programming every day.
It seems plausible to me that more undergrad EAs should do something like this, especially if they can get college credit for it (which I imagine might be hard for most students—I think I only got away with it because my university didn’t really know what was going on). The basic argument here is that it might be good for them the same way it was good for me.
More specifically, I think that there are a bunch of EAs who want to do technical AI alignment work and who are reasonably strong but not stellar coders. I think that if they did a coding bootcamp between, say, freshman and sophomore year, they might come back to school and be a bunch stronger. The bootcamp I did was focused on web app programming with Ruby and Rails and JavaScript. I think that these skills are pretty generically useful to software engineers. I often am glad to be better than my coworkers at quickly building web apps, and I learned those skills at App Academy (though being a professional web developer for a while also helped). Eg in our current research, even aside from the web app we use for getting our contractors to label data, we have to deal with a bunch of different computers that are sending data back and forth and storing it in databases or Redis queues or whatever. A reasonable fraction of undergrad EAs would seem like much more attractive candidates to me if they’d done a bootcamp. (They’d probably seem very marginally less attractive to normal employers than if they’d done something more prestigious-seeming with that summer, but most people don’t do very prestigious-seeming things in their first summer anyway. And the skills they had learned would probably be fairly attractive to some employers.)
This is just a speculative idea, rather than a promise, but I’d be interested in considering funding people to do bootcamps over the summer—they often cost maybe $15k. I am most interested in funding people to do bootcamps if they are already successful students at prestigious schools, or have other indicators of talent and conscientious, and have evidence that they’re EA aligned.
Another thing I like about this is that a coding bootcamp seems like a fairly healthy excuse to hang out in the Bay Area for a summer. I like that they involve working hard and being really focused on a concrete skill that relates to the outside world.
I am not sure whether I’d recommend someone do a web programming bootcamp or a data science bootcamp—though data science might seem more relevant, I think the practical programming stuff in the web programming bootcamp might actually be more helpful on the margin. (Especially for people who are already doing ML courses in school.)
I don’t think there are really any bootcamps focused on ML research and engineering. I think it’s plausible that we could make one happen. Eg I know someone competent and experienced who might run a bootcamp like this over a summer if we paid them a reasonable salary.
See my comment here, which applies to this Shortform as well; I think it would be a strong top-level post, and I’d be interested to see how other users felt about tech bootcamps they attended.
This seems like really good advice, thanks for writing this!
Also, I’m compiling a list of CS/ML bootcamps here (anyone should feel free to add items).
Doing lots of good vs getting really rich
Here in the EA community, we’re trying to do lots of good. Recently I’ve been thinking about the similarities and differences between a community focused on doing lots of good and a community focused on getting really rich.
I think this is interesting for a few reasons:
I found it clarifying to articulate the main differences between how we should behave and how the wealth-seeking community should behave.
I think that EAs make mistakes that you can notice by thinking about how the wealth-seeking community would behave, and then thinking about whether there’s a good reason for us behaving differently.
—— Here are some things that I think the wealth-seeking community would do.
There are some types of people who should try to get rich by following some obvious career path that’s a good fit for them. For example, if you’re a not-particularly-entrepreneurial person who won math competitions in high school, it seems pretty plausible that you should work as a quant trader. If you think you’d succeed at being a really high-powered lawyer, maybe you should do that.
But a lot of people should probably try to become entrepreneurs. In college, they should start small businesses, develop appropriate skills (eg building web apps), start trying to make various plans about how they might develop some expertise that they could turn into a startup, and otherwise practice skills that would help them with this. These people should be thinking about what risks to take, what jobs to maybe take to develop skills that they’ll need later, and so on.
I often think about EA careers somewhat similarly:
Some people are natural good fits for particular cookie-cutter roles that give them an opportunity to have a lot of impact. For example, if you are an excellent programmer and ML researcher, I (and many other people) would love to hire you to work on applied alignment research; basically all you have to do to get these roles is to obtain those skills and then apply for a job.
But for most people, the way they will have impact is much more bespoke and relies much more on them trying to be strategic and spot good opportunities to do good things that other people wouldn’t have otherwise done.
I feel like many EAs don’t take this distinction as seriously as they should. I fear that EAs see that there exist roles of the first type—you basically just have to learn some stuff, show up, and do what you’re told, and you have a bunch of impact—and then they don’t realize that the strategy they should be following is going to involve being much more strategic and making many more hard decisions about what risks to take. Like, I want to say something like “Imagine you suddenly decided that your goal was to make ten million dollars in the next ten years. You’d be like, damn, that seems hard, I’m going to have to do something really smart in order to do that, I’d better start scheming. I want you to have more of that attitude to EA.”
Important differences:
Members of the EA community are much more aligned with each other than wealth-seeking people are. (Maybe we’re supposed to be imagining a community of people who wanted to maximize total wealth of the community for some reason.)
Opportunities for high impact are biased to be earlier in your career than opportunities for high income. (For example, running great student groups at top universities is pretty high up there in impact-per-year according to me; there isn’t really a similarly good moneymaking opportunity for which students are unusually well suited.)
The space of opportunities to do very large amounts of good seems much narrower than the space of opportunities to make money. So you end up with EAs wanting to work with each other much more than the wealth-maximizing people want to work with each other.
It seems harder to make lots of money in a weird, bespoke, non-entrepreneurial role than it is to have lots of impact. There are many EAs who have particular roles which are great fits for them and which allow them to produce a whole bunch of value. I know of relatively fewer cases where someone gets a job which seems weirdly tailored to them and is really high paid.
I think this is mostly because my sense is that in the for-profit world, it’s hard to get people to be productive in weird jobs, and you’re mostly only able to hire people for roles that everyone involved understands very well already. And so even if someone would be able to produce a huge amount of value in some particular role, it’s hard for them to get paid commensurately, because the employer will be skeptical that they’ll actually produce all that value, and other potential employers will also be skeptical and so won’t bid their price up.
I’m commenting on a few Shortforms I think should be top-level posts so that more people see them, they can be tagged, etc. This is one of the clearest cases I’ve seen; I think the comparison is really interesting, and a lot of people who are promising EA candidates will have “become really rich” as a viable option, such that they’d benefit especially from thinking about this comparisons themselves.
Anyway, would you consider making this a top-level post? I don’t think the text would need to be edited all — it could be as-is, plus a link to the Shortform comments.
Thanks for writing this up. At the risk of asking obvious question, I’m interested in why you think entrepreneurship is valuable in EA.
One explanation for why entrepreneurship has high financial returns is information asymmetry/adverse selection: it’s hard to tell if someone is a good CEO apart from “does their business do well”, so they are forced to have their compensation tied closely to business outcomes (instead of something like “does their manager think they are doing a good job”), which have high variance; as a result of this variance and people being risk-averse, expected returns need to be high in order to compensate these entrepreneurs.
It’s not obvious to me that this information asymmetry exists in EA. E.g. I expect “Buck thinks X is a good group leader” correlates better with “X is a good group leader” than “Buck thinks X will be a successful startup” correlates with “X is a successful startup”.
It seems like there might be a “market failure” in EA where people can reasonably be known to be doing good work, but are not compensated appropriately for their work, unless they do some weird bespoke thing.
You seem to be wise and thoughtful, but I don’t understand the premise of this question or this belief:
But the reasoning [that existing orgs are often poor at rewarding/supporting/fostering new (extraordinary) leadership] seems to apply:
For example, GiveWell was a scrappy, somewhat polemical startup, and the work done there ultimately succeeded and created Open Phil and to a large degree, the present EA movement.
I don’t think any of this would have happened if Holden Karnofsky and Elie Hassenfeld had to say, go into Charity Navigator (or a dozen other low-wattage meta-charities that we will never hear of) and try to turn it around from the inside. While being somewhat vague, my models of orgs and information from EA orgs do not suggest that they are any better at this (for mostly benign, natural reasons, e.g. “focus”).
It seems that the main value of entrepreneurship is the creation of new orgs to have impact, both from the founder and from the many other staff/participants in the org.
Typically (and maybe ideally) new orgs are in wholly new territory (underserved cause areas, untried interventions) and inherently there are fewer people who can evaluate them.
It seems that the now canonized posts Really Hard and Denise Melchin’s experiences suggest this has exactly happened, extensively even. I think it is very likely that both of these people are not just useful, but are/could be highly impactful in EA and do not “deserve” the experiences their described.
[I think the main counterpoint would be that only the top X% of people are eligible for EA work or something like that and X% is quite small. I would be willing to understand this idea, but it doesn’t seem plausible/acceptable to me. Note that currently, there is a concerted effort to foster/sweep in very high potential longtermists and high potential EAs in early career stages, which seems invaluable and correct. In this effort, my guess is that the concurrent theme of focusing on very high quality candidates is related to experiences of the “production function” of work in AI/longtermism. However, I think this focus does not apply in the same way to other cause areas.]
Again, as mentioned at the top, I feel like I’ve missed the point and I’m just beating a caricature of what you said.
Thanks! “EA organizations are bad” is a reasonable answer.
(In contrast, “for-profit organizations are bad” doesn’t seem like reasonable answer for why for-profit entrepreneurship exists, as adverse selection isn’t something better organizations can reasonably get around. It seems important to distinguish these, because it tells us how much effort EA organizations should put into supporting entrepreneur-type positions.)
Maybe there’s some lesson to be learned. And I do think that EAs should often aspire to be more entrepreneurial.
But maybe the main lesson is for the people trying to get really rich, not the other way round. I imagine both communities have their biases. I imagine that lots of people try entrepreneurial schemes for similar reasons to why lots of people buy lottery tickets. And Id guess that this often has to do with scope neglect, excessive self confidence / sense of exceptionalism, and/or desperation.
Something I imagined while reading this was being part of a strangely massive (~1000 person) extended family whose goal was to increase the net wealth of the family. I think it would be natural to join one of the family businesses, it would be natural to make your own startup, and also it would be somewhat natural to provide services for the family that aren’t directly about making the money yourself. Helping make connections, find housing, etc.
Reminds me of The House of Saud (although I’m not saying they have this goal, or any shared goal):
”The family in total is estimated to comprise some 15,000 members; however, the majority of power, influence and wealth is possessed by a group of about 2,000 of them. Some estimates of the royal family’s wealth measure their net worth at $1.4 trillion”
https://en.wikipedia.org/wiki/House_of_Saud
Thanks, this is an interesting analogy.
If too few EAs go into more bespoke roles, then one reason could be risk-aversion. Rightly or wrongly, they may view those paths as more insecure and risky (for them personally; though I expect personal and altruistic risk correlate to a fair degree). If so, then one possibility is that EA funders and institutions/orgs should try to make them less risky or otherwise more appealing (there may already be some such projects).
In recent years, EA has put less emphasis on self-sacrifice, arguing that we can’t expect people to live on very little. There may be a parallel to risk—that we can’t expect people to take on more risk than they’re comfortable with, but instead must make the risky paths more appealing.
I like this chain of reasoning. I’m trying to think of concrete examples, and it seems a bit hard to come up with clear ones, but I think this might just be a function of the bespoke-ness.
A thought on how we describe existential risks from misaligned AI:
Sometimes discussions focus on a fairly specific version of AI risk, which involves humanity being quickly wiped out. Increasingly, though, the emphasis seems to be on the more abstract idea of “humanity losing control of its future.” I think it might be worthwhile to unpack this latter idea a bit more.
There’s already a fairly strong sense in which humanity has never controlled its own future. For example, looking back ten thousand years, no one decided that the sedentary agriculture would increasingly supplant hunting and gathering, that increasingly complex states would arise, that slavery would become common, that disease would take off, that social hierarchies and gender divisions would become stricter, etc. The transition to the modern world, and everything that came with this transition, also doesn’t seem to have been meaningfully chosen (or even really understood by anyone). The most serious effort to describe a possible future in detail — Hanson’s Age of Em — also describes a future with loads of features that most present-day people would not endorse.
As long as there are still strong competitive pressures or substantial random drift, it seems to me, no generation ever really gets to choose the future.[1] It’s actually sort of ambiguous, then, what it means to worry about “losing control of our future.”
Here are a few alternative versions of the concern that feel a bit crisper to me:
If we ‘mess up on AI,’ then even the most powerful individual humans will have unusually little influence over their own lives or the world around them.[2]
If we ‘mess up on AI,’ then future people may be unusually dissatisfied about the world they live in. In other words, people’s preferences will be unfilled to an unusually large degree.
Humanity may have a rare opportunity to take control of its own future, by achieving strong coordination and then locking various things in. But if we ‘mess up on AI,’ then we’ll miss out on this opportunity.[3]
Something that’s a bit interesting about these alternative versions of the concern, though, is that they’re not inherently linked to AI alignment issues. Even if AI systems behave roughly as their users intend, I believe each of these outcomes is still conceivable. For example, if there’s a missed opportunity to achieve strong coordination around AI, the story might look like the failure of the Baruch Plan for international control of nuclear weapons: that failure had much more to do with politics than it had to do with the way engineers designed the technology in question.
In general, if we move beyond discussing very sharp alignment-related catastrophes (e.g. humanity being quickly wiped out), then I think concerns about misaligned AI start to bleed into broader AI governance concerns. It starts to become more ambiguous whether technical alignment issues are actually central or necessary to the disaster stories people tell.
Although, admittedly, notable individuals or groups (e.g. early Christians) do sometimes have a fairly lasting and important influence. ↩︎
As an analogy, in the world of The Matrix, people may not actually have much less control over the long-run future than hunter-gatherers did twenty thousand years ago. But they certainly have much less control over their own lives. ↩︎
Notably, this is only a bad thing if we expect the relevant generation of humans to choose a better future than would be arrived at by default. ↩︎
Would you consider making this into a top-level post? The discussion here is really interesting and could use more attention, and a top-level post helps to deliver that (this also means the post can be tagged for greater searchability).
I think the top-level post could be exactly the text here, plus a link to the Shortform version so people can see those comments. Though I’d also be interested to see the updated version of the original post which takes comments into account (if you felt like doing that).
I agree with most of what you say here.
[ETA: I now realize that I think the following is basically just restating what Pablo already suggested in another comment.]
I think the following is a plausible & stronger concern, which could be read as a stronger version of your crisp concern #3.
“Humanity has not had meaningful control over its future, but AI will now take control one way or the other. Shaping the transition to a future controlled by AI is therefore our first and last opportunity to take control. If we mess up on AI, not only have we failed to seize this opportunity, there also won’t be any other.”
Of course, AI being our first and only opportunity to take control of the future is a strictly stronger claim than AI being one such opportunity. And so it must be less likely. But my impression is that the stronger claim is sufficiently more important that it could be justified to basically ‘wager’ most AI risk work on it being true.
I agree with this general point. I’m not sure if you think this is an interesting point to notice that’s useful for building a world-model, and/or a reason to be skeptical of technical alignment work. I’d agree with the former but disagree with the latter.
Mostly the former!
I think the point may have implications for how much we should prioritize alignment research, relative to other kinds of work, but this depends on what the previous version of someone’s world model was.
For example, if someone has assumed that solving the ‘alignment problem’ is close to sufficient to ensure that humanity has “control” of its future, then absorbing this point (if it’s correct) might cause them to update downward on the expected impact of technical alignment research. Research focused on coordination-related issues (e.g. cooperative AI stuff) might increase in value, at least in relative terms.
Do you have the intuition that absent further technological development, human values would drift arbitrarily far? It’s not clear to me that they would—in that sense, I do feel like we’re “losing control” in that even non-extinction AI is enabling a new set of possibilities that modern-day humans would endorse much less than the decisions of future humans otherwise. (It does also feel like we’re missing the opportunity to “take control” and enable a new set of possibilities that we would endorse much more.)
Relatedly, it doesn’t feel to me like the values of humans 150,000 years ago and humans now and even ems in Age of Em are all that different on some more absolute scale.
Certainly not arbitrarily far. I also think that technological development (esp. the emergence of agriculture and modern industry) has played a much larger role in changing the world over time than random value drift has.
I definitely think that’s true. But I also think that was true of agriculture, relative to the values of hunter-gatherer societies.
To be clear, I’m not downplaying the likelihood or potential importance of any of the three crisper concerns I listed. For example, I think that AI progress could conceivably lead to a future that is super alienating and bad.
I’m just (a) somewhat pedantically arguing that we shouldn’t frame the concerns as being about a “loss of control over the future” and (b) suggesting that you can rationally have all these same concerns even if you come to believe that technical alignment issues aren’t actually a big deal.
Wow, I just learned that Robin Hanson has written about this, because obviously, and he agrees with you.
FWIW, I wouldn’t say I agree with the main thesis of that post.
I definitely think that human biology creates at least very strong biases toward certain values (if not hard constraints) and that AI system would not need to have these same biases. If you’re worried about future agents having super different and bad values, then AI is a natural focal point for your worry.
A couple other possible clarifications about my views here:
I think that the outcome of the AI Revolution could be much worse, relative to our current values, than the Neolithic Revolution was relative to the values of our hunter-gatherer ancestors. But I think the question “Will the outcome be worse?” is distinct from the question “Will we have less freedom to choose the outcome?”
I’m personally not so focused on value drift as a driver of long-run social change. For example, the changes associated with the Neolithic Revolution weren’t really driven by people becoming less egalitarian, more pro-slavery, more inclined to hold certain religious beliefs, more ideologically attached to sedentism/farming, more happy to accept risks from disease, etc. There were value changes, but, to some significant degree, they seem to have been downstream of technological/economic change.
Really appreciate the clarifications! I think I was interpreting “humanity loses control of the future” in a weirdly temporally narrow sense that makes it all about outcomes, i.e. where “humanity” refers to present-day humans, rather than humans at any given time period. I totally agree that future humans may have less freedom to choose the outcome in a way that’s not a consequence of alignment issues.
I also agree value drift hasn’t historically driven long-run social change, though I kind of do think it will going forward, as humanity has more power to shape its environment at will.
My impression is that the differences in historical vegetarianism rates between India and China, and especially India and southern China (where there is greater similarity of climate and crops used), is a moderate counterpoint. At the timescale of centuries, vegetarianism rates in India are much higher than rates in China. Since factory farming is plausibly one of the larger sources of human-caused suffering today, the differences aren’t exactly a rounding error.
That’s a good example.
I do agree that quasi-random variation in culture can be really important. And I agree that this variation is sometimes pretty sticky (e.g. Europe being predominantly Christian and the Middle East being predominantly Muslim for more than a thousand years). I wouldn’t say that this kind of variation is a “rounding error.”
Over sufficiently long timespans, though, I think that technological/economic change has been more significant.
As an attempt to operationalize this claim: The average human society in 1000AD was obviously very different than the average human society in 10,000BC. I think that the difference would have been less than half as large (at least in intuitive terms) if there hadn’t been technological/economic change.
I think that the pool of available technology creates biases in the sorts of societies that emerge and stick around. For large enough amounts of technological change, and long enough timespans (long enough for selection pressures to really matter), I think that shifts in these technological biases will explain a large portion of the shifts we see in the traits of the average society.[1]
If selection pressures become a lot weaker in the future, though, then random drift might become more important in relative terms. ↩︎
And Paul Christiano agrees with me. Truly, time makes fools of us all.
Another interpretation of the concern, though related to your (3), is that misaligned AI may cause humanity to lose the potential to control its future. This is consistent with humanity not having (and never having had) actual control of its future; it only requires that this potential exists, and that misaligned AI poses a threat to it.
Recently I was asked for tips on how to be less captured by motivated reasoning and related biases, a goal/quest I’ve slowly made progress on for the last 6+ years. I don’t think I’m very good at this, but I do think I’m likely above average, and it’s also something I aspire to be better at. So here is a non-exhaustive and somewhat overlapping list of things that I think are helpful:
Treat this as a serious problem that needs active effort.
Personally, I repeat the mantra “To see the world as it is, not as I wish it to be” on a near-daily basis. You may have other mantras or methods to invoke this mindset that works better for you.
In general, try to be more of a “scout” and less of a soldier/zealot.
As much as possible, try to treat ideas/words/stats as tools to aid your perception, not weapons to advance your cause or “win” arguments.
I’ve heard good things about this book by Julia Galef, but have not read it
Relatedly, have a mental check and try to notice when you are being emotional and in danger of succumbing to motivated reasoning. Always try to think as coherently as you can if possible, and acknowledge it (preferably publicly) when you notice you’re not.
sometimes, you don’t notice this in real-time, but only long afterwards.
If this happens, try to postmortem your mistakes and learn from them.
In my case, I noticed myself having a bunch of passion that’s neither truth- nor utility- tracking with the whole Scott Alexander/NYT debacle.
I hope to not repeat this mistake.
Be precise with your speech, and try not to hide behind imprecise words that can mean anything.
especially bad to have words/phrases that connotes a lot of stuff while having empty denotation
Relatedly, make explicit public predictions or monetary bets whenever possible.
Just the fact that you’re betting either your $s or your reputation on a prediction often reminds you to be concrete and aware of your own BS.
But it’s also helpful to learn from and form your own opinions on how to have relatively accurate+well-calibrated forecasts.
Surround yourself with people who you respect as people who care a lot about the truth and try to seek it
For humans (well, at least me), clear thinking is often at least a bit of a social practice.
Just being around them helps make you try to emulate their behavior
Even more helpful if they’re at least a little bit confrontational and willing to call you out on your BS.
If you are in a situation where surrounding yourself with truthseekers is not a practical solution for your offline interactions (eg live in a small town, most government bureaucracy work), make “people who are extremely invested in truth-seeking” a top priority in the content you consume (in blog posts, social media, books, videocalls, and so forth).
Conversely, minimize the number of conversations (online and otherwise) where you feel like your interlocutor can’t change your mind
Maybe this is just a rephrase of a soldier/scout mindset distinction?
But I find it a helpful heuristic anyway.
In general, I think conversations where I’m trying to “persuade” the other person while not being open to being persuaded myself are a) somewhat impolite/disrespectful and b) has a corrosive effect (not just costly in time) to my intellectual progress
This doesn’t apply to everything important (eg intro to EA talks)
But nonetheless a good heuristic to have. If your job/hobbies pushes you to be in an “expert talking to non-expert” roles too much, you need to actively fight the compulsion to getting high on your own epistemic supply.
My personal view is that this sense of righteous certainty is more corrosive with one-to-one conversations than one-to-many conversations, and worse online than offline, but it’s just a personal intuition and I haven’t studied this rigorously.
Be excited to learn facts about the world, and about yourself.
I mostly covered psychological details here, but learning true facts about the world a) is directly useful in limiting your ability to lie about facts and b) somewhat constrains your thinking to reasoning that can explain actual facts about the world.
Critical thinking is not just armchair philosophy, but involves empirical details and hard-fought experience.
See Zeynep on Twitter
A robust understanding of facts can at least somewhat make up for epistemic shortcomings, and not necessarily vice versa.
I’ve previously shared this post on CEA’s social media and (I think) in an edition of the Forum Digest. I think it’s really good, and I’d love to see it be a top-level post so that more people end up seeing it, it can be tagged, etc.
Would you be interested in creating a full post for it? (I don’t think you’d have to make any changes — this still deserves to be read widely as-is.)