Error

Thanks, I found this post interesting.

(It also seems quite possible that people closer to being in the basin will tend to do more useful work than people further from it, although I’m not sure about that.)

Is part of why you’re not sure about that related to your observation in this post that someone becoming more “strategic” without becoming more “virtuous” could substantially increase the expected harm they do, since it might move them towards working in higher-leverage domains? In this case, the idea might be one of the following:

• People might move closer to the basin via becoming more truth-seeking and/​or self-aware, but without becoming more altruistic. If so, they might then start acting in higher-leverage domains, but act in overly self-interested, short-sighted, uncooperative, etc. ways, causing problems

• People might move closer to the basin via becoming more altruistic, but without becoming more truth-seeking or self-aware. Such people might have had a decent chance of making blunders regardless of what domains they worked in. But their increased altruism means those people are more likely to be acting in high-leverage domains or acting in a way perceived as “representative” of some important ideas/​movement, so the blunders are now more harmful.

Interesting post! I’m excited to see more thinking about memetics, for reasons sketched here and here. Some thoughts:

--In my words, what you’ve done is point out that approximate-consequentialism + large-scale preferences is an attractor. People with small-scale preferences (such as just caring about what happens to their village, or their family, or themselves, or a particular business) don’t have much to gain by spreading their memeplex to others. And people who aren’t anywhere close to being consequentialists might intellectually agree that spreading their memeplex to others would result in their preferences being satisfied to a greater extent, but this isn’t particularly likely to motivate them to do it. But people who are approximately consequentialist and who have large-scale preferences will be strongly motivated to spread their memeplex, because doing so is a convergent instrumental goal for people with large-scale preferences. Does this seem like a fair summary to you?

--I guess it leaves out the “truth-seeking” bit, maybe that should be bundled up with consequentialism. But I think that’s not super necessary. It’s not hard for people to come to believe that spreading their memeplex will be good by their lights; that is, you don’t have to be a rationalist to come to believe this. It’s pretty obvious.

--I think it’s not obvious this is the strongest attractor, in a world full of memetic attractors. Most major religions are memetic attractors, and they often rely on things other than convergent instrumental goals to motivate their members to spread the memeplex. And they’ve been extremely successful, far more so than “truth-seeking self-aware altruistic decision-making,” even though that memeplex has been around for millenia too.

--On the other hand, maybe truth-seeking self-aware altruistic decision-making has actually been even more successful than every major religion and ideology, we just don’t realize it because as a result of being truth-seeking, the memplex morphs constantly, and thus isn’t recognized as a single memplex. (By contrast with religions and ideologies which enforce conformity and dogma and thus maintain obvious continuity over many years and much territory.)

I read this post with a lot of interest; it has started to seem more likely to me lately that spreading productive, resilient norms about decision-making and altruism is a more effective means of improving decisions in the long run than any set of particular institutional structures. The knock-on effects of such a phenomenon would, on a long time scale, seem to dwarf the effects of many other ostensibly effective interventions.

So I get excited about this idea. It seems promising.

But some reflection about what is commonly considered precedent for something like this makes me a little bit more skeptical.

I think we see another kind of self-correction mechanism in the belief system of science. It provides tools for recognising truth and discarding falsehood, as well as cultural impetus to do so; this leads not just to the propagation of existing scientific beliefs, but to the systematic upgrading of those beliefs; this isn’t drift, but going deeper into the well of truth.

I have a sense that a large part of the success of scientific norms comes down to their utility being immediately visible. Children can conduct and repeat simple experiments (e.g. baking soda volcano); undergraduates can repeat famous projects with the same results (e.g. the double slit experiment), and even non-experimentalists can see the logic at the core of contemporary theory (e.g. in middle school geometry, or at the upper level in real analysis). What’s more, the norms seem to be cemented most effectively by precisely this kind of training, and not to spread freely without direct inculcation: scientific thinking is widespread among the trained, and (anecdotally) not so common among the untrained. For many Western non-scientists, science is just another source of formal authority, not a process that derives legitimacy from its robust efficacy.

I can see a way clear to a broadening of scientific norms to include what you’ve characterized as “truth-seeking self-aware altruistic decision-making.” But I’m having trouble imaging how it could be self-propagating. It would seem, at the very least, to require active cultivation in exactly the way that scientific norms do—in other words, that it would require a lot of infrastructure and investment so that proto-truth-seeking-altruists can see the value of the norms. Or perhaps I am having a semantic confusion: is science self-propagating in that scientists, once cultivated, go on to cultivate others?

Thanks for sharing your thoughts.

I don’t know if altruistic, truth seeking, and self aware are all necessary requirements though. It seems very much so to me that we’re never going to be able to convince the vast majority of people to have the excited attitude about EA that most of us do now. Maybe the right focus of the “altruism” meme like this should be on spreading the first two, altruistic and truth seeking.

Self awareness seems almost contrary to the idea of a meme like this, given that it relies on the spreading without too much questioning. Ideas with altruistic frameworks have done well in the past, i.e. ALS ice bucket challenge, but I don’t know how you would go about including a second idea into the existing matrix of a meme like that.

A scientific approach to memetics, I love our weird ideas!

Keep up the good posts Owen!