Thanks for the thoughtful reply.
I do think I was overestimating how robust you’re treating your numbers and premises, it seems like you’re holding them all much more lightly than I think I’d been envisioning.
FWIW I am more interested in engaging with some of what you wrote in in your other comment than engaging on the specific probability you assign, for some of the reasons I wrote about here.
I think I have more I could say on the methodology, but alas, I’m pretty blocked up with other work atm. It’d be neat to spend more time reading the report and leave more comments here sometime.
Great answer, thanks.
I tried to look for writing like this. I think that people do multiple hypothesis testing, like Harry in chapter 86 of HPMOR. There Harry is trying to weigh some different hypotheses against each other to explain his observations. There isn’t really a single train of conditional steps that constitutes the whole hypothesis.
My shoulder-Scott-Alexander is telling me (somewhat similar to my shoulder-Richard-Feynman) that there’s a lot of ways to trick myself with numbers, and that I should only do very simple things with them. I looked through some of his posts just now (1, 2, 3, 4, 5).
Here’s an example of a conclusion / belief from Scott’s post Teachers: Much More Than You Wanted to Know:
In summary: teacher quality probably explains 10% of the variation in same-year test scores. A +1 SD better teacher might cause a +0.1 SD year-on-year improvement in test scores. This decays quickly with time and is probably disappears entirely after four or five years, though there may also be small lingering effects. It’s hard to rule out the possibility that other factors, like endogenous sorting of students, or students’ genetic potential, contributes to this as an artifact, and most people agree that these sorts of scores combine some signal with a lot of noise. For some reason, even though teachers’ effects on test scores decay very quickly, studies have shown that they have significant impact on earning as much as 20 or 25 years later, so much so that kindergarten teacher quality can predict thousands of dollars of difference in adult income. This seemingly unbelievable finding has been replicated in quasi-experiments and even in real experiments and is difficult to banish. Since it does not happen through standardized test scores, the most likely explanation is that it involves non-cognitive factors like behavior. I really don’t know whether to believe this and right now I say 50-50 odds that this is a real effect or not – mostly based on low priors rather than on any weakness of the studies themselves. I don’t understand this field very well and place low confidence in anything I have to say about it.
I don’t know any post where Scott says “there’s a particular 6-step argument, and I assign 6 different probabilities to each step, and I trust that outcome number seems basically right”. His conclusions read more like 1 key number with some uncertainty, which never came from a single complex model, but from aggregating loads of little studies and pieces of evidence into a judgment.
I think I can’t think of a post like this by Scott or Robin or Eliezer or Nick or anyone. But would be interested in an example that is like this (from other fields or wherever), or feels similar.
One thing that I think would really help me read this document would be (from Joe) a sense of “here’s the parts where my mind changed the most in the course of this investigation”.
Something like (note that this is totally made up) “there’s a particular exploration of alignment where I had conceptualized it as kinda like about making the AI think right but now I conceptualize it as about not thinking wrong which I explore in section a.b.c”.
Also maybe something like a sense of which of the premises Joe changed his mind on the most – where the probabilities shifted a lot.
I think I share Robby’s sense that the methodology seems like it will obscure truth.
That said, I have neither your (Joe) extensive philosophical background nor have spent substantial time like you on a report like this, and I am interested in evidence to the contrary.
To me, it seems like you’ve tried to lay out a series of 6 steps of an argument, that you think each very accurately carve the key parts of reality that are relevant, and pondered each step for quite a while.
When I ask myself whether I’ve seen something like this produce great insight, it’s hard. It’s not something I’ve done much myself explicitly. However, I can think of a nearby example where I think this has produced great insight, which is Nick Bostrom’s work. I think (?) Nick spends a lot of his time considering a simple, single key argument, looking at it from lots of perspectives, scrutinizing wording, asking what people from different scientific fields would think of it, poking and prodding and rotating and just exploring it. Through that work, I think he’s been able to find considerations that were very surprising and invalidated the arguments, and proposed very different arguments instead.
When I think of examples here, I’m imagining that this sort of intellectual work produced the initial arguments about astronomical waste, and arguments since then about unilateralism and the vulnerable world hypothesis. Oh, and also simulation hypothesis (which became a tripartite structure).
I think of Bostrom as trying to consider a single worldview, and find out whether it’s a consistent object. One feeling I have about turning it into a multi-step probabilistic argument is that it does the opposite, it does not try to examine one worldview to find falsehoods, but instead integrates over all the parts of the worldview that Bostrom would scrutinize, to make a single clump of lots of parts of different worldviews. I think Bostrom may have literally never published a six-step argument of the form that you have, where it was meant to hold anything of weight in the paper or book, and also never done so assigning each step a probability.
To be clear, probabilistic discussions are great. Talking about precisely how strong a piece of evidence is – is it 2:1, 10:1, 100:1? Helps a lot in noticing which hypotheses to even pay attention to. The suspicion I have is that they are fairly different from the kind of cognition Bostrom does when doing this sort of philosophical argumentation that produces simple arguments of world-shattering importance. I suspect you’ve set yourself a harder task than Bostrom ever has (a 6-step argument), and thought you’ve made it easier for yourself by making it only probabilistic instead of deductive, whereas in fact this removes most of the tools that Bostrom was able to use to ensure he didn’t take mis-steps.
But I am pretty interested if there are examples of great work using your methodology that you were inspired by when writing this up, or great works with nearby methodologies that feel similar to you. I’d be excited to read/discuss some.
Pretty sure I picked those. I don’t know that the first two categories are as great a split as I once did. I was broadly trying to describe the difference between the sorts of basic theory work done by people like Alex Flint and Scott Garrabrant, and the sorts of ‘just solve the problem’ ideas by people like Paul Christiano and Alex Turner and Stuart Russell. But it’s not super clean, they dip into each other all the time e.g. Inner Alignment is a concept used throughout by people like Paul Christiano and Eliezer Yudkowsky in all sorts of research.
I worked with the belief that a very simple taxonomy even if wrong is far better than no taxonomy, so I still feel good about it. But am interested in an alternative.
Universal statements like this strike me as almost always wrong.
I appreciate the self-consistency of this sentence :)
What Failure Looks Like, hit LW post.
(Note: does not at all answer your question.)
I was surprised, this video was much less goodharted than I expected (after having been primed with the super-fast talking example). I was expecting more insane things.
Though overall it had the level of much broad public debate/discourse I’ve seen. I watched the first three speakers, and didn’t learn anything. In good debates I’ve seen I’ve felt that I’ve learned something from the debaters about their fields and their unique world views, these felt like two opposing sides in a broader political debate with kind of no grounding in reality. They were optimized for short-scale (e.g. <30 seconds) applause lights for the audience, when objected they’d make it a fight saying things like “Don‘t even try to win that example”, their examples seemed false yet rewarded (primarily attributing China’s rise out of poverty in the last 50 years to ‘redistribution’ and getting applause for it, which, correct me if I’m wrong, is not at all the primary reason, they had massive growth in industry in part by copying a lot of the west). I wouldn’t expect to learn anything, it just seemed like nobody understood economics and they were indexed off what was like 0-1 inferential steps from what the audience as a whole understood. I guess that was the worst part, how can you discuss interesting ideas if they have to be obvious to an audience that big and generic within 10-20 seconds?
I just want to say I, Ben Pace, feel attacked every time someone criticizes “BP” in this comment thread.
I’m open to a legal arrangement of shared nationalities, bank accounts, and professional roles.
“Hello, I’m an Effective Altruist.”
“Hello, I’m a world-unfucker.”
Honestly, I think the second one might be more action-oriented. And less likely to attract status-seekers. Alright, I’m convinced, let’s do it :)
I kinda think that “I’m an EA/he’s an EA/etc” is mega-cringey (a bad combo of arrogant + opaque acryonym + tribal)
It sounds like you think it’s bad that people have identified their lives with trying to help people as much as they can? Like, people like Julia Wise and Toby Ord shouldn’t have made it part of their life identity to do the most good they can do. They shouldn’t have said “I’m that sort of person” but they should have said “This is one of my interests”.
I do not know. Let me try generating names for a minute. Sorry. These will be bad.
“Marginal World Improvers”
”Black Swan Farmers”
Okay I will stop now.
Appreciate you drawing this, I like the idea.
we would be heading toward a more action-oriented and less communal group, which could reduce the attraction to manipulative people
I don’t expect a brand change to “Global Priorities” to bring in more action-oriented people. I expect fewer people would donate money themselves, for instance, they would see it as cute but obviously not having any “global” impact, and therefore below them.
(I think it was my inner Quirrell / inner cynic that wrote some of this comment, but I stand by it as honestly describing a real effect that I anticipate.)
The Defense Professor’s fingers idly spun the button, turning it over and over. “Then again, only a very few folk ever do anything interesting with their lives. What does it matter to you if they are mostly witches or mostly wizards, so long as you are not among them? And I suspect you will not be among them, Miss Davis; for although you are ambitious, you have no ambition.”“That’s not true!” said Tracey indignantly. “And what’s it mean?”Professor Quirrell straightened from where he had been leaning against the wall. “You were Sorted into Slytherin, Miss Davis, and I expect that you will grasp at any opportunity for advancement which falls into your hands. But there is no great ambition that you are driven to accomplish, and you will not make your opportunities. At best you will grasp your way upward into Minister of Magic, or some other high position of unimportance, never breaking the bounds of your existence.”
The Defense Professor’s fingers idly spun the button, turning it over and over. “Then again, only a very few folk ever do anything interesting with their lives. What does it matter to you if they are mostly witches or mostly wizards, so long as you are not among them? And I suspect you will not be among them, Miss Davis; for although you are ambitious, you have no ambition.”
“That’s not true!” said Tracey indignantly. “And what’s it mean?”
Professor Quirrell straightened from where he had been leaning against the wall. “You were Sorted into Slytherin, Miss Davis, and I expect that you will grasp at any opportunity for advancement which falls into your hands. But there is no great ambition that you are driven to accomplish, and you will not make your opportunities. At best you will grasp your way upward into Minister of Magic, or some other high position of unimportance, never breaking the bounds of your existence.”
—HPMOR, Chapter 70, Self-Actualization (part 5)
Added: The following is DEFINITELY NOT a strong argument, but just kind of an associative point. I think that Voldemort (both the real one from JK Rowling and also the one in HPMOR) would be much more likely to decide that he and his Death Eaters should have “Global Priorities” meetings than “Effective Altruist” meetings. (“We’re too focus on taking over the British Ministry for Magic, we need to also focus on our Global Priorities.“) In that way I think the former phrase has a more general connotation of ”taking power and changing the world” in a way the latter does not.
I was just reflecting on the term ‘global priorities’. I think to me it sounds like it’s asking “what should the world do”, in contrast to “what should I do”. The latter is far mode, the former is near. I think that staying near mode while thinking about improving the world is pretty tough. I think when people fail, they end making recommendations that could only work in-principle if everyone coordinates at the same time, and also as a result shape their speech to focus on signaling to achieve these ends, and often walk off a cliff of abstraction. I think when people stay in near mode, they focus on opportunities that do not require coordination, but opportunities they can personally achieve. I think that EAs caring very much about whether they actually helped someone with their donation has been one of the healthier epistemic things for the community. Though I do not mean to argue it should be held as a sacred value.
For example, I think the question “what should the global priority be on helping developing countries” is naturally answered by talking broadly about the West helping Africa build a thriving economy, talk about political revolution to remove corruption in governments, talk about what sorts of multi-billion dollar efforts could take place like what the Gates Foundation should do. This is a valuable conversation that has been going on for decades/centuries.
I think the question “what can I personally do to help people in Africa” is more naturally answered by providing cost-effectiveness estimates for marginal thousands of dollars to charities like AMF. This is a valuable conversation that I think has has orders of magnitude less effort put into it outside the EA community. It’s a standard idea in economics that you can reliably get incredibly high returns on small marginal investments, and I think it is these kind of investments that the EA community has been much more successful at finding, and has managed to exploit to great effect.
“global priorities (GP)” community is… more appropriate than “effective altruism (EA)” community… More appropriate (or descriptive) because it better focuses on large-scale change, rather than individual action
Anyway, I was surprised to read you say that, in direct contrast to what I was thinking, and I think how I have often thought of Effective Altruism.