“In the day I would be reminded of those men and women,
Brave, setting up signals across vast distances,
Considering a nameless way of living, of almost unimagined values.”
Emrik
Karma: 1,759
[weirdness-filter: ur weird if you read m commnt n agree w me lol]
Doing private capabilities research seems not obviously net-bad, for some subcategories of capabilities research. It constrains your expectations about how AGI will unfold, meaning you have a narrower target for your alignment ideas (incl. strategies, politics, etc.) to hit. The basic case: If an alignment researcher doesn’t understand how gradient descent works, I think they’re going to be less effective at alignment. I expect this to generalise for most advances they could make in their theoretical understanding of how to build intelligences. And there’s no fundamental difference between learning the basics and doing novel research, as it all amounts to increased understanding in the end.
That said, it would in most cases be very silly to publish about that increased understanding, and people should be disincentivised from doing so.
(I’ll delete this comment if you’ve read it and you want it gone. I think the above can be very bad advice to give some poorly aligned selfish researchers, but I want reasonable people to hear it.)
EA: We should never trust ourselves to do act utilitarianism, we must strictly abide by a set of virtuous principles so we don’t go astray.
Also EA: It’s ok to eat animals as long as you do other world-saving work. The effort and sacrifice it would take to relearn my eating patterns just isn’t worth it on consequentialist grounds.
Sorry for the strawmanish meme format. I realise people have complex reasons for needing to navigate their lives the way they do, and I don’t advocate aggressively trying to make other people stop eating animals. The point is just that I feel like the seemingly universal disavowment of utilitarian reasoning has been insufficiently vetted for consistency. If we claim that utilitarian reasoning can be blamed for the FTX catastrophe, then we should ask ourselves what else we should apply that lesson to; or we should recognise that FTX isn’t a strong counterexample to utilitarianism, and we can still use it to make important decisions.
(I realised after I wrote this that the metaphor between brains and epistemic communities is less fruitfwl than it seems like I think, but it’s still a helpfwl frame in order to understand the differences anyway, so I’m posting it here. ^^)
TL;DR: I think people should consider searching for giving opportunities in their networks, because a community that efficiently capitalises on insider information may end up doing more efficient and more varied research. There are, as you would expect, both problems and advantages to this, but it definitely seems good to encourage on the margin.
Some reasons to prefer decentralised funding and insider trading
I think people are too worried about making their donations appear justifiable to others. And what people expect will appear justifiable to others, is based on the most visibly widespread evidence they can think of.[1] It just so happens that that is also the basket of information that everyone else bases their opinions on as well. The net effect is that a lot less information gets considered in total.
Even so, there are very good reasons to defer to consensus among people who know more, not act unilaterally, and be epistemically humble. I’m not arguing that we shouldn’t take these considerations into account. What I’m trying to say is that even after you’ve given them adequate consideration, there are separate social reasons that could make it tempting to defer, and we should keep this distinction is in mind so we don’t handicap ourselves just to fit in.
Consider the community from a bird’s eye perspective for a moment. Imagine zooming out, and seeing EA as a single organism. Information goes in, and causal consequences go out. Now, what happens when you make most of the little humanoid neurons mimic their neighbours in proportion to how many neighbours they have doing the same thing?
What you end up with is a Matthew effect not only for ideas, but also for the bits of information that get promoted to public consciousness. Imagine ripples of information flowing in only to be suppressed at the periphery, way before they’ve had a chance to be adequately processed. Bits of information accumulate trust in proportion to how much trust they already have, and there are no well-coordinated checks that can reliably abort a cascade past a point.
To be clear, this isn’t how the brain works. The brain is designed very meticulously to ensure that only the most surprising information gets promoted to universal recognition (“consciousness”). The signals that can already be predicted by established paradigms are suppressed, and novel information gets passed along with priority.[2] While it doesn’t work perfectly for all things, consider just the fact that our entire perceptual field gets replaced instantly every time we turn our heads.
And because neurons have been harshly optimised for their collective performance, they show a remarkable level of competitive coordination aimed at making sure there are no informational short-circuits or redundancies.
Returning to the societal perspective again, what would it look like if the EA community were arranged in a similar fashion?
I think it would be a community optimised for the early detection and transmission of market-moving information—which in a finance context refers to information that would cause any reasonable investor to immediately make a decision upon hearing it. In the case where, for example, someone invests in a company because they’re friends with the CEO and received private information, it’s called “insider trading” and is illegal in some countries.
But it’s not illegal for altruistic giving! Funding decisions based on highly valuable information only you have access to is precisely the thing we’d want to see happening.
If, say, you have a friend who’s trying to get time off from work in order to start a project, but no one’s willing to fund them because they’re a weird-but-brilliant dropout with no credentials, you may have insider information about their trustworthiness. That kind of information doesn’t transmit very readily, so if we insist on centralised funding mechanisms, we’re unknowingly losing out on all those insider trading opportunities.
Where the architecture of the brain efficiently promotes the most novel information to consciousness for processing, EA has the problem where unusual information doesn’t even pass the first layer.
(I should probably mention that there are obviously biases that come into play when evaluating people you’re close to, and that could easily interfere with good judgment. It’s a crucial consideration. I’m mainly presenting the case for decentralisation here, since centralisation is the default, so I urge you keep some skepticism in mind.)
There are no way around having to make trade-offs here. One reason to prefer a central team of highly experienced grant-makers to be doing most of the funding, is that they’re likely to be better at evaluating impact opportunities. But this needn’t matter much if they’re bottlenecked by bandwidth—both in terms of having less information reach them and in terms of having less time available to analyse what does come through.[3]
On the other hand, if you believe that most of the relevant market-moving information in EA is already being captured by relevant funding bodies, then their ability to separate the wheat from the chaff may be the dominating consideration.
While I think the above considerations make a strong case for encouraging people to look for giving opportunities in their own networks, I think they apply with greater force to adopting a model like impact markets.
They’re a sort of compromise between central and decentralised funding. The idea is that everyone has an incentive to fund individuals or projects where they believe they have insider information indicating that the project will show itself to be impactfwl later on. If the projects they opportunistically funded at an early stage do end up producing a lot of impact, a central funding body rewards the maverick funder by “purchasing the impact” second-hand.
Once a system like that is up and running, people can reliably expect the retroactive funders to make it worth their while to search for promising projects. And when people are incentivised to locate and fund projects at their earliest bottlenecks, the community could end up capitalising on a lot more (insider) information than would be possible if everything had to be evaluated centrally.
(There are of course, more complexities to this, and you can check out the previous discussions on the forum.)
- ^
This doesn’t necessarily mean that people defer to the most popular beliefs, but rather that even if they do their own thinking, they’re still reluctant to use information that other people don’t have access to, so it amounts to nearly the same thing.
- ^
This is sometimes called predictive processing. Sensory information comes in and gets passed along through increasingly conceptual layers. Higher-level layers are successively trying to anticipate the information coming in from below, and if they succeed, they just aren’t interested in passing it along.
(Imagine if it were the other way around, and neurons were increasingly shy to pass along information in proportion to how confused or surprised they were. What a brain that would be!)
- ^
As an extreme example of how bad this can get, an Australian study on medicinal research funding estimated the length of average grant proposals to be “between 80 and 120 pages long and panel members are expected to read and rank between 50 and 100 proposals. It is optimistic to expect accurate judgements in this sea of excessive information.” -- (Herbert et al., 2013)
Luckily it’s nowhere near as bad for EA research, but consider the Australian case as a clear example of how a funding process can be undeniably and extremely misaligned with the goal producing good research.
- ^
Hm, I think you may be reading the comment from a perspective of “what actions do the symbols refer to, and what would happen if readers did that?” as opposed to “what are the symbols going to cause readers to do?”[1]
The kinds of people who are able distinguish adequate vs inadequate good judgment shouldn’t be encouraged to defer to conventional signals of expertise. But those are also disproportionately the people who, instead of feeling like deferring to Eliezer’s comment, will respond “I agree, but...”
- ^
For lack of a better term, and because there should be a term for it: Dan Sperber calls this the “cognitive causal chain”, and contrasts it with the confabulated narratives we often have for what we do. I think it summons up the right image.
When you read something, aspire to always infer what people intend based on the causal chains that led them to write that. Well, no. Not quite. Instead, aspire to always entertain the possibility that the author’s consciously intended meaning may be inferred from what the symbols will cause readers to do. Well, I mean something along these lines. The point is that if you do this, you might discover a genuine optimiser in the wild. : )
- ^
Ideally, EigenTrust or something similar should be able to help with regranting once it takes off, no? : )
Really intrigued by the idea of debates! I was briefly reluctant about the concept at first, because what I associate with “debates” is usually from politics, religious disputes, debating contests, etc. where the debaters are usually lacking so much of essential internal epistemic infrastructure that the debating format often just makes it worse. Rambly, before I head off to bed:
Conditional on it being good for EA to have more of a culture for debating, how would we go about practically bring that about?
I wonder if EA Global features debates. I haven’t seen any. It’s mostly just people agreeing with each other and perhaps adding some nuance.
You don’t need to have people hostile towards each other in order for it to qualify as “debate”, I do think one of the key benefits of debates is that the disagreement is visible.
For one, it primes the debaters to hone in on disagreements, whereas perhaps EA in-group are overly primed to find agreements with each other in order to be nice.
Making disagreements more visible will hopefwly dispel the illusion that EA as a paradigm is “mostly settled”, and get people to question assumptions. This isn’t always the best course of action, but I think it’s still very needed on the margin, and could get into why if asked.
If the debate (and the mutually-agreed-upon mindset of trying to find each others’ weakest points) is handled well, it can onlookers feel like head-on disagreeing is more ok. I think we’re mostly a nice community, reluctant to step on toes, so if we don’t see any real disagreements, we might start to feel like the absence of disagreement is the polite thing to do.
A downside risk is that debating culture is often steeped in the “world of arguments”, or as Nate Soares put it: “The world is not made of arguments. Think not “which of whese arguments, for these two opposing sides, is more compelling? And how reliable is compellingness?” Think instead of the objects the arguments discuss, and let the arguments guide your thoughts about them.”
We shouldn’t be adopting mainstream debating norms, it won’t do anything for us. What I’m excited about is the idea making spaces for good-natured visible disagreements where people are encouraged to attack each others’ weakest points. I don’t think that mindset comes about naturally, so it could make sense to deliberately make room for it.
Also, if you want people to debate you, maybe you should make a shortlist of the top things you feel would be productive to debate you on. : )
Just wanted to say, I love the 500-word limit. A contest that doesn’t goodhart on effort moralization!
Oh, this is excellent! I do a version of this, but I haven’t paid enough attention to what I do to give it a name. “Blurting” is perfect.
I try to make sure to always notice my immediate reaction to something, so I can more reliably tell what my more sophisticated reasoning modules transforms that reaction into. Almost all the search-process imbalances (eg. filtered recollections, motivated stopping, etc.) come into play during the sophistication, so it’s inherently risky. But refusing to reason past the blurt is equally inadvisable.
This is interesting from a predictive-processing perspective.[1] The first thing I do when I hear someone I respect tell me their opinion, is to compare that statement to my prior mental model of the world. That’s the fast check. If it conflicts, I aspire to mentally blurt out that reaction to myself.
It takes longer to generate an alternative mental model (ie. sophistication) that is able to predict the world described by the other person’s statement, and there’s a lot more room for bias to enter via the mental equivalent of multiple comparisons. Thus, if I’m overly prone to conform, that bias will show itself after I’ve already blurted out “huh!” and made note of my prior. The blurt helps me avoid the failure mode of conforming and feeling like that’s what I believed all along.
Blurting is a faster and more usefwl variation on writing down your predictions in advance.
- ^
Speculation. I’m not very familiar with predictive processing, but the claim seems plausible to me on alternative models as well.
- ^
If Will thought SBF was a “bad egg”, then it could be more important to establish influence with him, because you don’t need to establish influence (as in ‘willingness to cooperate’) with someone who is entirely value-aligned with you.
Yes! That should work fine. That’s 21:00 CET for me. See you then!
My email is emrik.asheim@gmail.com btw.
I think it’d be easy to come up with highly impactfwl things to do with free reign over Twitter? Like, even before I’ve thought about it, there should be a high prior on usefwl patterns. Brainstorming:
Experiment with giving users control over recommender algorithms, and/or designing them to be in the long-term interests of the users themselves (because you’re ok with foregoing some profit in order to not aggressively hijacking people’s attention)
Optimising the algorithms for showing users what they reflectively prefer (eg. what do I want to want to see on my Twitter feed?)[1]
Optimising algorithms for making people kinder (eg. downweighting views that come from bandwagony effects and toxoplasma), but still allowing users to opt-out or opt-in, and clearly guiding them on how to do so.
Trust networks
Liquid democracy-like transitive trust systems (eg. here, here)
I can see several potential benefits to this, but most of the considerations are unknown to me, which just means that there could still be massive value that I haven’t seen yet.
This could be used to overcome Vingean deference limits and allow for hiring more competent people more reliably than academic credentials (I realise I’m not explaining this, I’m just pointing to the existence of ideas enabled with Twitter)
This could also be a way to “vote” for political candidates or decision-makers in general too, or be used as a trust metric to find out whether you want to vote for particular candidates in the first place.
Platform to arrange vote swapping and similar, allow for better compromises and reduce hostile zero-sum voting tendencies.
Platform for highly visible public assurance contracts (eg. here), could be potentially be great for cooperation between powerfwl actors or large groups of people.
This also enables more visibility for views that held back by pluralistic ignorance. This could be both good and bad, depending on the view (eg. both “it’s ok to be gay” and “it’s not ok to be gay” can be held back by pluralistic ignorance).
Could also be used to coordinate actions in a crisis
eg. the next pandemic is about to hit, and it’s a thousand times more dangerous than covid, and no one realises because it’s still early on the exponential curve. Now you utilise your power to influence people to take it seriously. You stop caring about whether this will be called “propaganda” because what matters isn’t how nice you’ll look to the newspapers, what matters is saving people’s lives.
Something-something nudging idk.
Mostly, even if I thought Sam was in the wrong for considering a deal with Elon, I find it strange to cast a negative light on Will for putting them in touch. That seems awfwly transitive. I think judgments for transitive associations are dangerous, especially given incomplete information. Sam/Will probably thought much longer on this than I have, so I don’t think I can justifiably fault their judgment even if I had no ideas on how to use twitter myself.
- ^
This idea was originally from a post by Paul Christiano some years ago where he urged FB to adopt an algorithm like this, but I can’t seem to find it rn.
Hmm, I suspect that anyone who had the potential to be bumped over the threshold for interest in EA, would be likely to view the EA Wikipedia article positively despite clicking through to it via SBF. Though I suspect there are a small number of people with the potential to be bumped over that threshold. I have around 10% probability on that the negative news has been positive for the movement, primarily because it gained exposure. Unlikely, but not beyond the realm of possibility. Oo
Mh, I had in mind both, and wanted to leave it up for interpretation. A public debate about something could be cool because I’ve never done that, but we’d need to know what we’re supposed to disagree about first. Though, I primarily just wish to learn from you, since you have a different perspective, so a private call would be my second offer.
I’m confused. You asked whether I had in mind a public or private call, and I said I’d be fine with either. Which question are you referring to?
Happy to either, but I’ll stay off-camera if it’s going to be recorded. Up to you, if you wish to prioritise it. : )
This is excellent. I hadn’t even thought to check. Though I think I disagree with the strength of your conclusion.
If you look back since 2015 (“all time”), it looks like this.[1] Keep in mind that Sam has 1.1M pageviews since before November[2] (before anyone could know anything about anything). Additionally, if you browse his Wikipedia page, EA is mentioned under “Careers”, and is not something you read about unless particularly interested. On the graph, roughly 5% click through to EA. (Plus, most of the news-readers are likely to be in the US, so the damage is localised?)
A point of uncertainty for me is to what extent news outlets will drag out the story. Might be that most of the negative-association-pageviews are yet to come, but I suspect not? Idk.
Not sure how to make the prediction bettable but I think I’m significantly less pessimistic about the brand value than you seem to be.
Feel free to grab these if you want to make a post of it. Seems very usefwly calibrating.
Hmm, not sure how that job would work, but if someone could be paid to fix all of EAs coordination problems, that’d probably be worth the money. It is the responsibility of anyone who wishes to assume that responsibility. And if they can, I really hope they do.
I’m a very slow reader, but do you wish to discuss (or debate it!) over a video call sometime? I’m eager to learn things from someone who’s likely to have a different background on the questions I’m interested in. : )
I also made a suggestion on Sasha’s post related to nudging people’s reading habits by separating out FTX posts by default. I don’t endorse the design, but it could look something like this.[1] Alternatively, could introduce ‘tag profiles’ or something, where you can select a profile, and define your filters within each profile.[2]
(P.S. Sorry for the ceaseless suggestions, haha! Brain goes all sparkly with an idea and doesn’t shut up until I make a comment about it. ^^’)
I have an unusual perspective on this. I skim nearly everything, and rarely see the benefit of completionism in reading. And I don’t benefit from the author being nuanced or give different views a balanced take. I don’t care about how epistemically virtuous the author is being! I just care about learning usefwl patterns from the stuff I’m reading, and for that, there’s no value to me knowing whether the author is obviously biased in favour of a view or not. There are more things to say here, but I say some of them here.
Btw, I appreciate your contributions to the forum, at least for what I’ve seen of it. : )
I predict with high uncertainty that this post will have been very usefwl to me. Thanks!
Here’s a potential missing mood: if you read/skim a post and you don’t go “ugh that was a waste of time” or “wow that was worth reading”[1], you are failing to optimise your information diet and you aren’t developing intuition for what/how to read.
This is importantly different from going “wow that was a good/impressive post”. If you’re just tracking how impressed you are by what you read (or how useful you predict it is for others), you could be wasting your time on stuff you already know and/or agree with. Succinctly, you need to track whether your mind has changed—track the temporal difference.