Researcher at MIRI. http://shlegeris.com/
I think this is a great idea, and I’m excited that you’re doing it.
I’ve recently been thinking about medieval alchemy as a metaphor for longtermist EA.
I think there’s a sense in which it was an extremely reasonable choice to study alchemy. The basic hope of alchemy was that by fiddling around in various ways with substances you had, you’d be able to turn them into other things which had various helpful properties. It would be a really big deal if humans were able to do this.
And it seems a priori pretty reasonable to expect that humanity could get way better at manipulating substances, because there was an established history of people figuring out ways that you could do useful things by fiddling around with substances in weird ways, for example metallurgy or glassmaking, and we have lots of examples of materials having different and useful properties. If you had been particularly forward thinking, you might even have noted that it seems plausible that we’ll eventually be able to do the full range of manipulations of materials that life is able to do.
So I think that alchemists deserve a lot of points for spotting a really big and important consideration about the future. (I actually have no idea if any alchemists were thinking about it this way; that’s why I billed this as a metaphor rather than an analogy.) But they weren’t really very correct about how anything worked, and so most of their work before 1650 was pretty useless.
It’s interesting to think about whether EA is in a similar spot. I think EA has done a great job of identifying crucial and underrated considerations about how to do good and what the future will be like, eg x-risk and AI alignment. But I think our ideas for acting on these considerations seem much more tenuous. And it wouldn’t be super shocking to find out that later generations of longtermists think that our plans and ideas about the world are similarly inaccurate.
So what should you have done if you were an alchemist in the 1500s who agreed with this argument that you had some really underrated considerations but didn’t have great ideas for what to do about them?
I think that you should probably have done some of the following things:
Try to establish the limits of your knowledge and be clear about the fact that you’re in possession of good questions rather than good answers.
Do lots of measurements, write down your experiments clearly, and disseminate the results widely, so that other alchemists could make faster progress.
Push for better scientific norms. (Scientific norms were in fact invented in large part by Robert Boyle for the sake of making chemistry a better field.)
Work on building devices which would enable people to do experiments better.
Overall I feel like the alchemists did pretty well at making the world better, and if they’d been more altruistically motivated they would have been even better.
There are some reasons to think that pushing early chemistry forward is easier than working on improving the long term future, In particular, you might think that it’s only possible to work on x-risk stuff around the time of the hinge of history.
Yeah, I thought about this; it’s standard marketing terminology, and concise, which is why I ended up using it. Thanks though.
I thought this post was really bad, basically for the reasons described by Rohin in his comment. I think it’s pretty sad that that post has positive karma.
When I was 18 I watched a lot of videos of animal suffering, eg linked from Brian Tomasik’s list of distressing videos of suffering (extremely obvious content warning: extreme suffering). I am not sure whether I’d recommend this to others.
As a result, I felt a lot of hatred for people who were knowingly complicit in causing extreme animal suffering, which was basically everyone I knew. At the time I lived in a catered college university, where every day I’d see people around me eating animal products; I felt deeply alienated and angry and hateful.
This was good in some ways. I think it’s plausibly healthy to feel a lot of hatred for society. I think that this caused me to care even less about what people thought of me, which made it easier for me to do various weird things like dropping out of university (temporarily) and moving to America.
I told a lot of people to their faces that I thought they were contemptible. I don’t feel like I’m in the wrong for saying this, but this probably didn’t lead to me making many more friends than I otherwise would have. And on one occasion I was very cruel to someone who didn’t deserve it; I felt more bad about this than about basically anything else I’d done in my life.
I don’t know whether I’d recommend this to other people. Probably some people should feel more alienated and others should feel less alienated.
For what it’s worth, I think that EA related outreach to heirs seems much less promising than to founders or pro poker players.
Successful founders are often extremely smart in my experience; I expect pro poker players are also pretty smart on average.
It seems likely that pro athletes are more intelligent than average, but I’d be very surprised if they were as intelligent as pro poker players on average.
Edited to add: I think that I phrased this post misleadingly; I meant to complain mostly about low quality criticism of EA rather than eg criticism of comments. Sorry to be so unclear. I suspect most commenters misunderstood me.I think that EAs, especially on the EA Forum, are too welcoming to low quality criticism [EDIT: of EA]. I feel like an easy way to get lots of upvotes is to make lots of vague critical comments about how EA isn’t intellectually rigorous enough, or inclusive enough, or whatever. This makes me feel less enthusiastic about engaging with the EA Forum, because it makes me feel like everything I’m saying is being read by a jeering crowd who just want excuses to call me a moron.
I’m not sure how to have a forum where people will listen to criticism open mindedly which doesn’t lead to this bias towards low quality criticism.
I would be pretty surprised if most of the people from the EALF survey thought that forecasting is “very closely related” to good judgement.
Thanks for writing this post—it was useful to see the argument written out so I could see exactly where I agreed and disagreed. I think lots of people agree with this but I’ve never seen it written up clearly before.
I think I place substantial weight (30% or something) on you being roughly right about the relative contributions of EA safety and non-EA safety. But I think it’s more likely that the penalty on non-EA safety work is larger than you think.
I think the crux here is that I think AI alignment probably requires really focused attention, and research done by people who are trying to do something else will probably end up not being very helpful for some of the core problems.
It’s a little hard to evaluate the counterfactuals here, but I’d much rather have the contributions from EA safety than from non EA safety over the last ten years.
I think that it might be easier to assign a value to the discount factor by assessing the total contributions of EA safety and non-EA safety. I think that EA safety does something like 70% of the value-weighted work, which suggests a much bigger discount factor than 80%.
Assorted minor comments:
But this is only half of the ledger. One of the big advantages of academic work is the much better distribution of senior researchers: EA Safety seems bottlenecked on people able to guide and train juniors
Yes, but those senior researchers won’t necessarily have useful things to say about how to do safety research. (In fact, my impression is that most people doing safety research in academia have advisors who don’t have very smart thoughts on long term AI alignment.)
None of those parameters is obvious, but I make an attempt in the model (bottom-left corner).
I think the link is to the wrong model?
A cursory check of the model
In this section you count nine safety-relevant things done by academia over two decades, and then note that there were two things from within EA safety last year that seem more important. This doesn’t seem to mesh with your claim about their relative productivity.
MIRI is not optimistic about prosaic AGI alignment and doesn’t put much time into it.
I don’t think the evidence is very good; I haven’t found it more than slightly convincing. I don’t think that the harms of current systems are a very good line of argument for potential dangers of much more powerful systems.
I’m curious what your experience was like when you started talking to AI safety people after already coming to come of your own conclusions. Eg I’m curious if you think that you missed major points that the AI safety people had spotted which felt obvious in hindsight, or if you had topics on which you disagreed with the AI safety people and think you turned out right.
In an old post, Michael Dickens writes:
The closest thing we can make to a hedonium shockwave with current technology is a farm of many small animals that are made as happy as possible. Presumably the animals are cared for by people who know a lot about their psychology and welfare and can make sure they’re happy. One plausible species choice is rats, because rats are small (and therefore easy to take care of and don’t consume a lot of resources), definitively sentient, and we have a reasonable idea of how to make them happy.
Thus creating 1 rat QALY costs $120 per year, which is $240 per human QALY per year.
This is just a rough back-of-the-envelope calculation so it should not be taken literally, but I’m still surprised by how cost-inefficient this looks. I expected rat farms to be highly cost-effective based on the fact that most people don’t care about rats, and generally the less people care about some group, the easier it is to help that group. (It’s easier to help developing-world humans than developed-world humans, and easier still to help factory-farmed animals.) Again, I could be completely wrong about these calculations, but rat farms look less promising than I had expected.
I think this is a good example of something seeming like a plausible idea for making the world better, but which turned out to seem pretty ineffective.
What current controversy are you saying might make moderate pro-SJ EAs more wary of SSC?
I have two complaints: linking to a post which I think was made in bad faith in an attempt to harm EA, and seeming to endorse it by using it as an example of a perspective that some EAs have.
I think you shouldn’t update much on what EAs think based on that post, because I think it was probably written in an attempt to harm EA by starting flamewars.
EDIT: Also, I kind of think of that post as trying to start nasty rumors about someone; I think we should generally avoid signal boosting that type of thing.
I’d be interested to see a list of what kinds of systematic mistakes previous attempts at long-term forecasting made.
Also, I think that many longtermists (eg me) think it’s much more plausible to successfully influence the long run future now than in the 1920s, because of the hinge of history argument.
Many other people who are personally connected to the Chinese Cultural Revolution are the people making the comparisons, though. Eg the EA who I see posting the most about this (who I don’t think would want to be named here) is Chinese.