Studying behaviour and interactions of boundedly rational agents, AI alignment and complex systems.
Research fellow at Future of Humanity Institute, Oxford. Other projects: European Summer Program on Rationality. Human-aligned AI Summer School. Epistea Lab.
In my view this is a bad decision.
As I wrote on LW
Sorry but my rough impression from the post is you seem to be at least as confused about where the difficulties are as average of alignment researchers you think are not on the ball—and the style of somewhat strawmanning everyone & strong words is a bit irritating.
In particular I don’t appreciate the epistemic of these moves together
1. Appeal to seeing thinks from close proximity. Then I got to see things more up close. And here’s the thing: nobody’s actually on the friggin’ ball on this one!
2. Straw-manning and weakmaning what almost everyone else thinks and is doing
3. Use of an emotionally compelling words like ‘real science’ for vaguely defined subjects where the content may be the opposite of what people imagine. Is the empirical alchemy-style ML type of research what’s advocated for as the real science?
4. What overall sounds more like the aim is to persuade, rather than explain
I think curating this signals this type of bad epistemics is fine, as long as you are strawmanning and misrepresenting others in a legible way and your writing is persuasive. Also there is no need to actually engage with existing arguments, you can just claim seeing things more up close.
Also to what extent are moderator decisions influenced by status and centrality in the community...
… if someone new and non-central to the community came up with this brilliant set of ideas how to solve AI safety:
1. everyone working on it is not on the ball. why? they are all working on wrong things!
2. promising is to do something very close to how empirical ML capabilities research works
3. this is a type of problem where you can just throw money at it and attract better ML talent
… I doubt this would have a high chance of becoming curated.
Copy-pasting here from LW.
Sorry but my rough impression from the post is you seem to be at least as confused about where the difficulties are as average of alignment researchers you think are not on the ball—and the style of somewhat strawmanning everyone & strong words is a bit irritating.
Maybe I’m getting it wrong, but it seems the model you have for why everyone is not on the ball is something like “people are approaching it too much from a theory perspective, and promising approach is very close to how empirical ML capabilities research works” & “this is a type of problem where you can just throw money at it and attract better ML talent”.
I don’t think these two insights are promising.
Also, again, maybe I’m getting it wrong, but I’m confused how similar you are imagining the current systems to be to the dangerous systems. It seems either the superhuman-level problems (eg not lying in a way no human can recognize) are somewhat continuous with current problems (eg not lying), and in that case it is possible to study them empirically. Or they are not. But different parts of the post seem to point in different directions. (Personally I think the problem is somewhat continuous, but many of the human-in-the-loop solutions are not, and just break down.)
Also, with what you find promising I’m confused what do you think the ‘real science’ to aim for is - on one hand it seems you think the closer the thing is to how ML is done in practice the more real science it is. On the other hand, in your view all deep learning progress has been empirical, often via dumb hacks and intuitions (this isn’t true imo).
(crossposted from Alignment Forum)
While the claim—the task ‘predict next token on the internet’ absolutely does not imply learning it caps at human-level intelligence—is true, some parts of the post and reasoning leading to the claims at the end of the post are confused or wrong.
Let’s start from the end and try to figure out what goes wrong.
GPT-4 is still not as smart as a human in many ways, but it’s naked mathematical truth that the task GPTs are being trained on is harder than being an actual human.
And since the task that GPTs are being trained on is different from and harder than the task of being a human, it would be surprising—even leaving aside all the ways that gradient descent differs from natural selection—if GPTs ended up thinking the way humans do, in order to solve that problem.
From a high-level perspective, it is clear that this is just wrong. Part of what human brains are doing is to minimise prediction error with regard to sensory inputs. Unbounded version of the task is basically of same generality and difficulty as what GPT is doing, and is roughly equivalent to understand everything what is understandable in the observable universe. For example: a friend of mine worked at analysing the data from LHC, leading to the Higgs detection paper. Doing this type of work basically requires a human brain to have a predictive model of aggregates of outputs of a very large number of collisions of high-energy particles, processed by a complex configuration of computers and detectors.
Where GPT and humans differ is not some general mathematical fact about the task, but differences in what sensory data is a human and GPT trying to predict, and differences in cognitive architecture and ways how the systems are bounded. The different landscape of both boundedness and architecture can lead to both convergent cognition (thinking as the human would do) and the opposite, predicting what the human would output in highly non-human way.
The boundedness is overall a central concept here. Neither humans nor GPTs are attempting to solve ‘how to predict stuff with unlimited resources’, but a problem of cognitive economy—how to allocate limited computational resources to minimise prediction error.
Or maybe simplest:
Imagine somebody telling you to make up random words, and you say, “Morvelkainen bloombla ringa mongo.”Imagine a mind of a level—where, to be clear, I’m not saying GPTs are at this level yet -
Imagine a Mind of a level where it can hear you say ‘morvelkainen blaambla ringa’, and maybe also read your entire social media history, and then manage to assign 20% probability that your next utterance is ‘mongo’.
The fact that this Mind could double as a really good actor playing your character, does not mean They are only exactly as smart as you.
When you’re trying to be human-equivalent at writing text, you can just make up whatever output, and it’s now a human output because you’re human and you chose to output that.
GPT-4 is being asked to predict all that stuff you’re making up. It doesn’t get to make up whatever. It is being asked to model what you were thinking—the thoughts in your mind whose shadow is your text output—so as to assign as much probability as possible to your true next word.
If I try to imagine a mind which is able to predict my next word when asked to make up random words, and be successful at assigning 20% probability to my true output, I’m firmly in the realm of weird and incomprehensible Gods. If the Mind is imaginably bounded and smart, it seems likely it would not devote much cognitive capacity to trying to model in detail strings prefaced by a context like ‘this is a list of random numbers’, in particular if inverting the process generating the numbers would seem really costly. Being this good at this task would require so much data and cheap computation that this is way beyond superintelligence, in the realm of philosophical experiments.
Overall I think it is really unfortunate way how to think about the problem, where a system which is moderately hard to comprehend (like GPT) is replaced by something much more incomprehensible. Also it seems a bit of a reverse intuition pump—I’m pretty confident most people’s intuitive thinking about this ’simplest’ thing will be utterly confused.
How did we got here?
A human can write a rap battle in an hour. A GPT loss function would like the GPT to be intelligent enough to predict it on the fly.
Apart from the fact that humans are also able to rap battle or impro on the fly, notice that “what would the loss function like the system to do” in principle tells you very little about what the system will do. For example, the human loss function makes some people attempt to predict winning lottery numbers. This is an impossible task for humans and you can’t say much about the human based on this. Or you can speculate about minds which would be able to succeed in this task, but you soon get into the realm of Gods and outside of physics.
Consider that sometimes human beings, in the course of talking, make errors.
GPTs are not being trained to imitate human error. They’re being trained to *predict* human error.
Consider the asymmetry between you, who makes an error, and an outside mind that knows you well enough and in enough detail to predict *which* errors you’ll make.
Again, from the cognitive economy perspective, predicting my errors would often be wasteful. With some simplification, you can imagine I make two types of errors—systematic, and random. Often the simplest way how to predict the systematic error would be to emulate the process which led to the error. Random errors are … random, and a mind which knows me in enough detail to predict which random errors I’ll make seems a bit like the mind predicting the lottery numbers.
Consider that somewhere on the internet is probably a list of thruples: <product of 2 prime numbers, first prime, second prime>.
GPT obviously isn’t going to predict that successfully for significantly-sized primes, but it illustrates the basic point:
There is no law saying that a predictor only needs to be as intelligent as the generator, in order to predict the generator’s next token.
The general claim that some predictions are really hard and you need superhuman powers to be good at them is true, but notice that this does not inform us about what GPT-x will learn.
Imagine yourself in a box, trying to predict the next word—assign as much probability mass to the next token as possible—for all the text on the Internet.
Koan: Is this a task whose difficulty caps out as human intelligence, or at the intelligence level of the smartest human who wrote any Internet text? What factors make that task easier, or harder?
Yes this is clearly true: in the limit the task is of unlimited difficulty.
You are correct with some of the criticism, but as a side-note, completeness is actually crazy.
All real agents are bounded, and pay non-zero costs for bits, and as a consequence, don’t have complete preferences. Complete agents in real world do not exist. If they existed, correct intuitive model of them wouldn’t be ‘rational players’ but ‘utterly scary god, much bigger than the universe they live in’.
a.
Sequoia led FTX round B in Jul 2021 and had notably more time to notice any irregularities than grant recipients.
b.
I would expect the funds to have much better expertise in something like “evaluating the financial health of a company”.
Also it seem you are somewhat shifting the goalposts: Zoe’s paragraph with “On Halloween this past year, I was hanging out with a few EAs.” It is reasonable to assume the reader will interpret it as hanging out with basically random/typical EAs, and the argument should hold for these people. Your argument would work better if she was hanging out with “EAs working at FTX” or “EAs advising SBF” who could have probably done better than funds on evaluating stuff like how the specific people work.
The EA project is clearly not promised on the idea that it should, for example, “figure out stuff like stock price better than legacy institutions”. Quite the contrary—the claim is while humanity actually invests decent amount of competent effort in stock, in comparison, it neglects problems like poverty or xrisk.
In my view this is an example of a mistake in bounded/local consequentialism
From deontic perspective, there is a coordination problem, where “at least consistent handle” posts can be somewhat costly for the poster, but an atmosphere of an earnest discussion of real people has large social benefits. Vice versa, discussion with a large fraction of anonymous accounts—in particular if they are sniping at real people and each other—decreases trust, and is vulnerable to manipulation by sock puppets and nefarious players.
Also, I think there are some virtue ethics costs associated with anonymous posts, roughly in the direction of transparency and integrity.
For example, if I imagine myself anonymously posting something critical received unfavourably by someone, and later, meeting that someone in person, or collaborating on something relevant, I would find it integrity-decreasing to continue hiding the authorship. And if I’d be happy to reveal my identity to the people upset … why not reveal it directly?
While I don’t think these considerations add up to “never post anonymously”, I think they are pretty large, and usually much larger than e.g. “small probability of adverse career effects in the EA ecosystem”.
Seems worth trying
At the same time, I don’t think the community post / frontpage attention mechanism is the core of what’s going on. Which is, in my guess, often best understood as a fight between memeplexes about hearts and minds
The quality of reasoning in the text seems somewhat troublesome. Using two paragraphs as example
On Halloween this past year, I was hanging out with a few EAs. Half in jest, someone declared that the best EA Halloween costume would clearly be a crypto-crash — and everyone laughed wholeheartedly. Most of them didn’t know what they were dealing with or what was coming. I often call this epistemic risk: the risk that stems from ignorance and obliviousness, the catastrophe that could have been avoided, the damage that could have been abated, by simply knowing more. Epistemic risks contribute ubiquitously to our lives: We risk missing the bus if we don’t know the time, we risk infecting granny if we don’t know we carry a virus. Epistemic risk is why we fight coordinated disinformation campaigns and is the reason countries spy on each other.
Still, it is a bit ironic for EAs to have chosen ignorance over due diligence. Here are people who (smugly at times) advocated for precaution and preparedness, who made it their obsession to think about tail risks, and who doggedly try to predict the future with mathematical precision. And yet, here they were, sharing a bed with a gambler against whom it was apparently easy to find allegations of shady conduct. The affiliation was a gamble that ended up putting their beloved brand and philosophy at risk of extinction.
It appears that a chunk of Zoe’s epistemic risk bears a striking resemblance to financial risk. For instance, if one simply knew more about tomorrow’s stock prices, they could sidestep all stock market losses and potentially become stupendously rich.
This highlights the fact that gaining knowledge in certain domains can be difficult task, with big hedge funds splashing billions and hiring some of the brightest minds just to gain a slight edge in simply knowing a bit more about asset prices. It extends to having more info about which companies may go belly up or engage in fraud.
Acquiring more knowledge comes at a cost. Processing knowledge comes at cost. Choosing ignorance is mostly not a result of recklessness or EA institutional design but a practical choice given the resources required to process information. It’s actually rational for everyone to ignore most information most of the time (this is standard econ, check rational inattention and extensive literature on the topic).
One real question in this space is if EAs have allocated their attention wisely. The answer seems to be “mostly yes.” In case of FTX, heavyweights like Temasek, Sequoia Capital, and SoftBank with billions on the line did their due diligence but still missed what was happening. Expecting EAs to be better evaluators of FTX’s health than established hedge funds is somewhat odd. EAs, like everyone else, face the challenge of allocating attention and their expertise lies in “using money for good” rather than “evaluating the health of big financial institutions”. For the typical FTX grant recipient to assume they need to be smarter than Sequoia or SoftBank about FTX would likely not be a sound decision.
Also: Confido works for intuitively eliciting probabilities
I think this is a weird response to what Buck wrote. Buck also isn’t paid either to reform EA movement, or to respond to criticism on EA forum, and decided to spend his limited time to express how things realistically look from his perspective.
I think it is good if people write responses like that, and such responses should be upvoted, even if you disagree with the claims. Downvotes should not express ‘I disagree’, but ‘I don’t want to read this’.
Even if you believe EA orgs are horrible and should be completely reformed, in my view, you should be glad that Buck wrote his comment as you have better idea what people like him may think.
It’s important to understand the alternative to this comment is not Buck will write 30 page detailed response. The alternative is, in my guess, just silence.
Thanks for all the care and effort which went into writing this!
At the same time, while reading, my reactions were most of the time “this seems a bit confused”, “this likely won’t help” or “this seems to miss the fact that there is someone somewhere close to the core EA orgs who understands the topic pretty well, and has a different opinion”.
Unfortunately, to illustrate this in detail for the whole post would be a project for …multiple weeks.
At the same time I thought it could be useful to discuss at least one small part in detail, to illustrate how the actual in-the-detail disagreement could look like.
I’ve decided to write a detailed response for a few paragraphs about rationality and Bayesianism. This is from my perspective not cherry-picked part of the of the original text which is particularly wrong, but a part which seems representatively wrong/confused. I picked it for convenience, because I can argue and reference this particularly easily.
Individual Bayesian Thinking (IBT) is a technique inherited by EA from the Rationalist subculture, where one attempts to use Bayes’ theorem on an everyday basis. You assign each of your beliefs a numerical probability of being true and attempt to mentally apply Bayes’ theorem, increasing or decreasing the probability in question in response to new evidence. This is sometimes called “Bayesian epistemology” in EA, but to avoid confusing it with the broader approach to formal epistemology with the same name we will stick with IBT.
This seems pretty strange characterization. Even though I participated at multiple CFAR events, do teach various ‘rationality techniques’ and know decent amount of stuff about Bayesian inference, I think this is misleading/confused.
What’s called in rationalist circles “Bayesian epistemology” is basically what’s the common understanding of the term:
- you don’t hold beliefs to be true or false, but have credences in them
- normatively, you should update the credences based on evidence. normatively, the proper rule for that is the Bayes rule; this is intractable in practice, so you do various types of approximations
- you should strive for coherent beliefs if you don’t want to be Dutch-booked
It’s important to understand that in this frame, it is a normative theory. Bayes theorem in this perspective is not some sort of “minor aid for doing some sort of likelihood calculation” but a formal foundation for large part of epistemology.
The view that you believe different things to different degrees, and these credences basically are Bayesian probabilities and are normatively governed by the same theory isn’t an ‘EA’ or ‘rationalist’ thing but a standard Bayesian take. (cf Probability Theory: The Logic of Science, E. T. Jaynes)
Part of what Eliezer’s approach to ‘applied rationality’ aimed for is taking Bayesian epistemology seriously and applying this frame to improve everyday reasoning.
But this is almost never done by converting your implicit probability distributions to numerical credences , doing the explicit numerical math, and blindly trusting the result!
What’s done instead is
-noticing that your brain already internally used credences and probabilities all the time. you can easily access your “internal” (/S1/...) probabilities in an intuitive way by asking yourself questions like “how surprised you would be to see [a pink car today| Donald Trump reelection | SpaceX rocket landing in your backyard]” (the idea that brains do this is pretty mainstream eg Confidence as Bayesian Probability: From Neural Origins to Behavior, Meyniel et.al.)
- noticing you brain often clearly does something a bit similar to what Bayesians suggests as the normative idea - e.g. if two SpaceX rockets already landed in your backyard today, you would be way less surprised by the third one
- noticing there is often a disconnect between this intuitive / internal / informal calculation, and the explicit, verbal reasoning (cf alief/belief)
- …and using all of that to improve both the “implicit” and “explicit” reasoning!
The actual ‘techniques’ derived from this are often implicit. For example, one actual technique is: Imagine you are an alien who landed in one world of two. They differ in the respect that in one a proposition is true, in the other, the opposite is true. You ask yourself how the world would look like in the different worlds, and then look at the actual world.
For example, consider this proposition: “democratic vote is the optimal way how to make decision making in organizations”: How would the world where this is true look like? There are parts of the world with intense competition between organizations, e.g. companies in highly competitive industries, optimizing hard and measurable things. In the world where the proposition is true, I’d expect a lot of voting in these companies. We don’t see that, which decreases my credence in the proposition.
It is relatively easy to see how this both connected to Bayes and not asking people to do any explicit odd multiplications.
There is nothing wrong with quantitative thinking, and much of the power of EA grows from its dedication to the numerical. However, this is often taken to the extreme, where people try to think almost exclusively along numerical lines, causing them to neglect important qualitative factors or else attempt to replace them with doubtful or even meaningless numbers because “something is better than nothing”. These numbers are often subjective “best guesses” with little empirical basis.[27]
While some people make the error of trying to replace complex implicit calculations by the over-simplified spreadsheets with explicit numbers, this paragraph seems to conflate multiple things together as “numerical” or “quantitative”.
Assuming fairly standard standard cognitive science and neuroscience, at some level all thinking is “numerical”, including thinking which feels intuitive or qualitative. People usually express such thinking in words like “I strongly feel” or “I’m pretty confident”
What’s a classical rationalist move in such cases is to try to make the implicit explicit. E.g., if you are fairly confident, at what odds would you be willing to bet on it?
When done correctly, the ability and willingness to do that mostly exposes what’s already there. People already act based on the implicit credences and likelihoods, even if they don’t try to express them as probability distributions, and you don’t have access to them.
E.g., when some famous experts recommended ‘herd immunity’ strategy to deal with covid, using strong and confident words, such recommendation actually were subjective “best guess” with little empirical basis. Same is actually true for many expert opinions on policy topics!
Rationalist habit of reporting credences and predictions using numbers … basically exposes many things to possibility of being proved wrong, and exposes many personal best guesses for what they are.
Yes, for someone who isn’t used to this at all this may create fake aura of ‘certainty’, because use of numbers often signals ‘this is more clear’ and use of words signals ‘this is more slippery’ in common communication. But this is just a communication protocol.
Yes, as I wrote before, some people may make the mistake of trying to convert some basic things to numbers and replace their brains with spreadsheets with Bayes formulas in the next step, but it does not seem common at least in my social neighborhood.
For instance, Bayesian estimates are heavily influenced by one’s initial figure (one’s “prior”), which, especially when dealing with complex, poorly-defined, and highly uncertain and speculative phenomena, can become subjective (based on unspecified values, worldviews, and assumptions) to the point of arbitrary.[28] This is particularly true in existential risk studies where one may not have good evidence to update on.
I would be curious how the authors imagine the non-Bayesian thinking not depending on any priors internally works.
We assume that, with enough updating in response to evidence, our estimates will eventually converge on an accurate figure. However, this is dependent on several conditions, notably well-formulated questions, representative sampling of (accurate) evidence, and a rigorous and consistent method of translating real-world observations into conditional likelihoods.[29] This process is very difficult even when performed as part of careful and rigorous scientific study; attempting to do it all in your head, using rough-guess or even purely intuitional priors and likelihoods, is likely to lead to more confidence than accuracy.
This seems confused (the “common response” mentioned below applies here exactly). How do you imagine, for example, a group of people looking at a tree, manages to agree on seeing a tree? The process of converting raw sensory data to the tree-hypothesis is way more complicated than a typical careful and rigorous scientific study, and also way more reliable than a typical published scientific study.
Again: correctly understood, the applied rationalist idea is not to replace our mind’s natural ways of recognizing a tree by a process where you would assign numbers to statements like “green in upper left part of visual field” and do explicit Bayesian calculation in S2 way, but just to be …less wrong.
This is further complicated by the fact that probabilities are typically distributions rather than point values – often very messy distributions that we don’t have nice neat formulae for. Thus, “updating” properly would involve manipulating big and/or ugly matrices in your head. Perhaps this is possible for some people.
A common response to these arguments is that Bayesianism is “how the mind really works”, and that the brain already assigns probabilities to hypotheses and updates them similarly or identically to Bayes’ rule. There are good reasons to believe that this may be true. However, the fact that we may intuitively and subconsciously work along Bayesian lines does not mean that our attempts to consciously “do the maths” will work.
I think the “common response” is partially misunderstood here? The common response does not imply you can consciously explicitly multiply the large matrices or do the exact Bayesian inferences, any more than someone a catching a ball would be consciously and explicitly solving the equations of motion.
The correct ideas here are:
- you can often make some parts or results of the implicit subconscious calculations explicit and numeric (cf forecasting, betting, …)
- the implicit reasoning is often biased and influenced by wishes and wants
- explicitly stating things or betting on things sometimes exposes the problems
- explicit reasoning can be good for that
- explicit reasoning is also good for understanding what the normatively good move is in simple or idealized cases
- on the other hand, explicit reasoning alone is computationally underpowered for almost anything beyond very simple models. (compare how many FLOPs is your brain using, vs. how fast you can explicitly multiply numbers)
- what you usually need to do is use both, and watch for flaws
In addition, there seems to have been little empirical study of whether Individual Bayesian Updating actually outperforms other modes of thought, never mind how this varies by domain. It seems risky to put so much confidence in a relatively unproven technique.
Personally I don’t know anyone who would propose people should do the “Individual Bayesian Thinking” mode of thoughts in the way you describe, and I don’t see much reason to make a study on this. Also while a lot people in EA orgs subscribe to basically Bayesian epistemology, I don’t know anyone who would try to live by the “IBT”, so you should probably be less worried about the risks from the use of it.
The process of Individual Bayesian Updating can thus be critiqued on scientific grounds,
So, to me, this is characteristic—and, frankly, annoying—about the whole text. I don’t think you have properly engaged with Bayesian epistemology, state of art applied rationality practice, or relevant cognitive science. “critiqued on scientific grounds” sounds serious and authoritative … but where is the science?
but there is also another issue with it and hyper-quantitative thinking more generally: motivated reasoning. With no hard qualitative boundaries and little constraining empirical data, the combination of expected value calculations and Individual Bayesian Thinking in EA allows one to justify and/or rationalise essentially anything by generating suitable numbers.
This is both sad and funny. One of the good things about rationalist habits and techniques is, stating explicit numbers often allows one to spot and correct motivated reasoning. In relation to existential risk and similar domains, often the hope is that by practicing this in domains with good feedback and bets which are possible to empirically evaluate, you get better at thinking clearly… and this will at least partially generalize to epistemically more challenging domains.
Yes, you can overdo it, or do stupid or straw versions of this. Yes, it is not perfect.
But what’s the alternative? Honestly, in my view, in many areas of expertise the alternative is to state views, claims and predictions in sufficiently slippery and non-quantitative way that it is very difficult to clearly disprove them.
Take for example your text and claims about diversity. I think given the way you are using it, it seems anyone trying to refute the advise on empirical grounds would have really hard time, and you would be always able to write some story why some dimension of diversity is not important, or why some other piece of research states something else. (It seems a common occurrence in humanities that some confused ideas basically never die, unless they lose support on the level of ‘sociology of science’.)
Bottom line:
- these 8 paragraphs did not convince me about any mistake people at e.g. FHI may be making
- suggestion “Bayes’ theorem should be applied where it works” is pretty funny; I guess Bayesians wholeheartedly agree with this!
- suggestions like “studies of circumstances under which individual subjective Bayesian reasoning actually outperforms other modes of thought” seems irrelevant given the lack of understanding of actual ‘rationality techniques’
- we have real world evidence that getting better at some of the traditional “rationalist” skills makes you better at least at some measurable things, e.g. forecasting
I suspect … that even what I see as a wrong model of ‘erros of EA’ maybe points to some interesting evidence. For example maybe some EA community builders are actually teaching “individual bayesian thinking” as a technique you should do in the way described?
In the spirit of communication style you advocate for… my immediate emotional reaction to this is “Eternal September has arrived”.
I dislike my comment being summarized as “brings up the “declining epistemics” argument to defend EA orgs from criticism”. In the blunt style you want, this is something between distortion and manipulation.
On my side, I wanted to express my view on the Wytham debate. And I wrote a comment expressing my views on the debate.
I also dislike the way my comment is straw-manned by selective quotation.
In the next bullet point to “The discussion often almost completely misses the direct, object-level, even if just at back-of-the-envelope estimate way.” I do explicitly acknowledge the possible large effects of higher order factors.
In contrast, large fraction attention in the discussion seems spent on topics which are both two steps removed from the actual thing , and very open to opinions. Where by one step removed I mean e.g. “how was this announced” or “how was this decided”, and two steps removed is e.g. “what will be the impact of how was this announced on the sentiment of the twitter discussion”. While I do agree such considerations can have large effect, driving decisions by this type of reasoning in my view moves people and orgs into the sphere of pure PR, spin and appearance.
What I object to is a combination of
1. ignore the object level, or discuss it in a very lazy way
2. focus on the 2nd order … but not in a systematic way, but mostly based on saliency and emotional pull (e.g., how will this look on twitter)
Yes, it is a simple matter to judge where this leads in the limit. We have a bunch of examples how the discourse looks like when completely taken over by these considerations—e.g., political campaigns. Words have little meaning connected to physical reality, but are mostly tools in the fight for the emotional states and minds of other people.
Also: while those with high quality epistemics usually agree on similar things is a distortion making the argument personal, about people, in reality yes, good reasoning often converges to similar conclusions.
Also: It’s a given that the path of catering to a smaller group of people with higher quality epistemics will have more impact than spreading the core EA messaging to a larger group of people with lower quality epistemics
No, it’s not given. Just, so far, effective altruism was about using evidence and reason to figure out how to benefit others as much as possible, and acting based on that. Based on the thinking so far, It was decidedly not trying to be a mass movement, making our core insights more appealing to the public at large .
In my view, no one figured out yet how the appealing to the masses, don’t need to think much version of effective altruism should look like to be actually good.
(edit: Also, I quite dislike the frame-manipulation move of shifting from “epistemic decline of the community” to “less intelligent or thoughtful people joining”. You can imagine a randomized experiment where you take two groups of equally intelligent and thoughtful people, and you make them join community with different styles of epistemic culture (eg physics, and multi-level marketing). You will get very different results. While you seem to interpret a lot of things as about people (are they smart? had they studied philosophy?) I think it’s often much more about norms.)
I will try to paraphrase, please correct me if I’m wrong about this: the argument is, this particular bikeshed is important because it provides important evidence about how EA works, how trustworthy the people are, or what are the levels of transparency. I think this is a fair argument.
At the same time I don’t think it works in this case, because while I think EA has important issues, this purchase does not really illuminate them.
Specifically, object level facts about this bikeshed
do not provide that that much evidence, beyond basic facts like “people involved in this have access to money”
the things they tell you are mostly boring
they provide some weak positive evidence about the people involved being sane and reasonable
it is unclear how much evidence provided by this generalizes to nuclear reactors
Object level, you don’t need precise numbers and long spreadsheets to roughly evaluate it. As I gestured to, in late 2021, the “x-risk-reduction” area had billions of dollars committed to it, less than a thousand people working on it, and good experience with progress made on in person events. Given the ~ low millions pound effective cost of the purchase and the marginal costs of time and money, it seems like a sensible decision. In my view this conclusion does not strongly depend on priors about EA, but you can reach it by doing a quick calculation and a few google searches.
Things about the process seem mostly boring. How it went seems:
1. some people thought an events venue near Oxford is a sensible, even if uncertain, bet
2. they searched for venues
3. selected a candidate
4. got funding
5. EVF decided to fiscally sponsor this
6. the venue was bought
7. this was not announced with a fanfare
8. boring things like reconstructing some things started?
(Disclosure about step 2: I had seen the list of candidate venues, and actually visited one other place on the list. The process was in my view competent and sensible, for example in the aspect it involved talking with potential users of the venue)
What this tells us about the people involved seems …not much, but mostly weakly positive?
1. it seems the decision process involved some willingness to explore and do uncertain things; this is better than EA strawman of comparing every option to bednets
2. it seems based on understanding of real-world events organization
3. the decision to not announce it with fanfare seems sensible
4. my impression is the counterfactual PR impacts, if this was announced with a fanfare, pre-FTX, would have been worse
In contrast, some of the things critiques of the decision ask for seem pretty unreasonable to me. For example
1. discussing property purchases before they are made
2. creating a splash of publicity immediately after it was purchased
3. getting EA forum users somehow involved in the process
4. semi-formal numerical estimates of impact
I do think that what it does illuminate is a tension between
global poverty reduction EA memes, which includes stuff like comparing purchases to lives saved, and moral duty to do something about it
x-risk-reduction EA memes, which includes stuff like willingness to spend a lot of money to influence something important
rationality memes, which emphasize than spending $1000 to save 1h of time in the morning , and spending 1h to save $30 in the afternoon, is perhaps not an optimal decision pattern
And I do think it is something between really PR tricky and PR nightmare to have all of this under one brand. If this is the main point, than yes, Wytham is a piece of evidence, but this seemed clear much sooner.
With nuclear reactors, I don’t see a strong case how this evidence generalizes, in either direction.
For me, unfortunately, the discourse surrounding Wytham Abbey, seems like a sign of epistemic decline of the community, or at least on the EA forum.
The amount of attentions spent on this seems to be a textbook example of bikeshedding.
Quoting Parkinson :”The time spent on any item of the agenda will be in inverse proportion to the sum [of money] involved.” A reactor is so vastly expensive and complicated that an average person cannot understand it (see ambiguity aversion), so one assumes that those who work on it understand it. However, everyone can visualize a cheap, simple bicycle shed, so planning one can result in endless discussions because everyone involved wants to implement their own proposal and demonstrate personal contribution.
In case of EAs, there are complicated, high-stakes things, for example what R&D efforts to support around AI. This has scale of billions of dollars now, much higher stakes in the future, and there is a lot to understand.
In contrast, absolutely anyone can easily form opinions about appropriateness of a manor house purchase, based on reading a few tweets.
Repeatedly, the tone of the discussion is a bit like “I’ve read a tweet by Émile Torres, I got upset, and I’m writing on EA forum”. Well, tweets by Émile Torres are known to be an unreliable source of information, and in contrast are often optimized to create negative emotional response. (To not single out Torres, this is also true for many other tweets.)
The discussion often almost completely misses the direct, object-level, even if just at back-of-the-envelope estimate way. On this level venue purchases were sensible:
the amount of EA x-risk-reduction money per person in early 2022 was pretty high
organising events and setting up venues is labour intensive, often more difficult to delegate than people assume, and often constrained on time of people who have high opportunity costs
on the margin, you can often trade money, time, work … this trade seems to make sense
apparently, multiple people reached similar conclusion; apart from Wytham, there was for example a different venue purchased in the Bay area, and other near Prague (note: I’m leading the org fiscally sponsoring the later project)
In contrast, large fraction attention in the discussion seems spent on topics which are both two steps removed from the actual thing , and very open to opinions. Where by one step removed I mean e.g. “how was this announced” or “how was this decided”, and two steps removed is e.g. “what will be the impact of how was this announced on the sentiment of the twitter discussion”. While I do agree such considerations can have large effect, driving decisions by this type of reasoning in my view moves people and orgs into the sphere of pure PR, spin and appearance.
Nice post, but my rough take is there is
it’s relatively common markets are inefficient, but unexploitable; trading on “everyone dies” seems a clear case of hard-to-exploit inefficiency
markets are not magic; impacts of one-off events with complex consequences are difficult to price in, and what all the magical market aggregation boils down to are a bunch of human brains doing the trades; e.g. I was able to beat the market and get n-times return at point where markets were insane about covid; later, I talked about it with someone in one of the giant hedge funds, and the simple explanation is, while they were looking into it, at some point I knew more about covid than they were able to assemble
example of such hard-to-predict event are e.g. capabilities and impacts of a specific model
the dichotomy 30% growth / everyone dies is unrealistic for trading purposes
near term, there are various outcomes like “industry X gets disrupted” or “someone loses job due to automation” or “war”
if you anticipate fears of this type to dominate in next 10 years, you should price many people increasing their savings and borrowing less
Personally, I think the 1:1 meme is deeply confused.
A helpful analogy (thanks to Ollie Base) is with nutrition. Imagine someone hearing that “chia seeds are the nutritionally most valuable food, top rated in surveys” … and subsequently deciding to eat just chia seeds, and nothing else!
In my view, sort of obviously, intellectual conference diet consisting just of 1:1s is poor and unhealthy for almost everyone.