Edit: The post has excellent nuance, and I make no claim to support or defend Leverage specifically (idk them). My comment is intended more generally, and my disagreement concerns two points:
“The core problem was that Leverage 1.0 quickly became much too internally focused.”
“If they’re not publishing research I would take that as a strong negative signal for the project.”
You make several points, but I just want to respond to my impression that you’re trying to anchor wayward researchers or research groups to the “main paradigm” to decrease the chance that they’ll be wrong. I’m pretty strongly against this.
In a common-payoff game (like EA research), we all share the fruits of major discoveries regardless of who makes the discovery. So we should heavily prioritise sensitivity over specificity. It doesn’t matter how many research groups are wildly wrong, as long at least one research group figures out how to build the AI that satisfies our values with friendship and ponies. So when you’re trying to rein in researchers instead of letting them go off and explore highly variable crazy stuff, you’re putting all your eggs in one basket (the most respectable paradigm). Researchers are already heavily incentivised to research what other people are researching (the better to have a lively conversation!), so we do not need additional incentives against exploration.
The value distribution of research fruits is fat-tailed (citation needed). Strategies that are optimal for sampling normal distributions are unlikely to be optimal for fat tails. Sampling for outliers means that you should rely more on theoretical arguments, variability, and exploration, because you can’t get good data on the outliers—the only data that matters. If you insist on being legible and scientific, so you optimise your strategy based on the empirical data you can collect, you’re being fooled into mediocristan again.
Lemme cite a paper in network epistemology so I can fake looking like I know what I’m talking about,
“However, pure populations of mavericks, who try to avoid research approaches that have already been taken, vastly outperform the other strategies. Finally, we show that, in mixed populations, mavericks stimulate followers to greater levels of epistemic production, making polymorphic populations of mavericks and followers ideal in many research domains.”[1] -- Epistemic landscapes and the division of cognitive labor
That said, I also advocate against explorers being allowed to say
But I’m virtuously doing high-variance exploration, so I don’t need to worry about your rigorous schmigorous epistemology!
Explorers need to be way more epistemologically vigilant than staple researchers pursuing the safety of existing paradigms. If you leave your harbour to sail out into the open waters, that’s not a good time to forget your sextant, or pretend you’ll be a better navigator without studying the maps that do exist.
FWIW, I think conclusions from network-epistemological computer simulations are extremely weak evidence about what we as an irl research community should do, and I mainly benefit from it because they occasionally reveals patterns that help with analysing real-life phenomena. The field exists at all—despite their obviously irrelevant “experiments”—because it makes theoretical speculation seem more technical, impressive, professional.
It doesn’t matter how many research groups are wildly wrong, as long at least one research group figures out how to build the AI that satisfies our values with friendship and ponies.
Sort of? In your hypothetical there are two ways your research project could go once you believe you’ve succeeded:
You go and implement it, or
You figure out how to communicate your results to the rest of the industry.
If you go with (1) then it’s really important that you get things right, and if you’ve disconnected yourself from external evaluation I think there’s a large chance you haven’t. I’d much prefer to see (2), except now you do need to communicate your results in detail so the rest of the world can evaluate and so you didn’t gain that much by putting off the communication until the end.
I’ll also make a stronger claim, which is that communication improves your research and chances of success: figuring out how to communicate things to people who don’t have your shared context makes it a lot clearer which things you actually don’t understand yet.
trying to rein in researchers instead of letting them go off and explore highly variable crazy stuff
I’m not sure why you think I’m advocating avoiding high-variability lines of research? I’m saying research groups should make public updates on their progress to stay grounded, not that they should only take low-risk bets.
I edited my original comment to point out my specific disagreements. I’m now going to say a selection of plausibly false-but-interesting things, and there’s much more nuance here that I won’t explicitly cover because that’d take too long. It’s definitely going to seem very wrong at first glance without the nuance that communicates the intended domain.
I feel like I’m in a somewhat similar situation to Leverage, only in the sense that I feel like having to frequently publish would hinder my effectiveness. It would make it easier for others to see the value of my work, but in my own estimation that trades off against maximising actual value.
This isn’t generally the case for most research, and I might be delusional (ime 10%) to think it’s the case for my own, but I should be following the gradient of what I expect will be the most usefwl. It would be selfish of me to do the legible thing motivated just by my wish for people to respect me.
The thing I’m arguing for is not that people like me shouldn’t publish at all, it’s that we should be very reluctant to punish gambling sailors for a shortage of signals. They’ll get our attention once they can demonstrate their product.
The thing about having to frequently communicate your results is that it incentivises you to adopt research strategies that lets you publish frequently. This usually means forward-chaining to incremental progress without much strategic guidance. Plus, if you get into the habit of spending your intrinsic motivation on distilling your progress to the community, now your brain’s shifted to searching for ideas that fitinto the community, instead of aiming your search to solve the highest-priority confusion points in your own head.
To be an effective explorer, you have to get to the point where you can start to iterate on top of your own ideas. If you timidly “check in” with the community every time you think you have a novel thought, before you let yourself stand on it in order to explore further down the branch, then 1) you’re wasting their time, and 2) no one’s ever gonna stray far from home.
When you go from—
A) “huh, I wonder how this thing works, and how it fits into other things I have models of.” to B) “hmm, the community seems to behave as if Y is true, but I have a suspicion that ¬X, so I should research it and provide them with information they find valuable.”
—then a pattern for generating thoughts will mostly be rewarded based on your prediction about whether the community is likely to be persuaded by those thoughts. This makes it hard to have intrinsic motivation to explore anything that doesn’t immediately seem relevant to the community.
And while B is still reasonably aligned with producing value as long as the community is roughly as good at evaluating the claims as you are, it breaks down for researchers who are much better than their expected audience at what they specialise in. If the most competent researchers have brains that optimise for communal persuasiveness, they’re wasting their potential when they could be searching for ideas that optimise for persuading themselves—a much harder criteria to meet given that they’re more competent.
I think it’s unhealthy to–within your own brain–constantly try to “advance the communal frontier”. Sure, that could ultimately be the goal, but if you’re greedily and myopically only able to optimise for specifically that at every step, then that is like a chess player who’s compulsively only able to look for checkmate patterns–unable to see forks that merely win material or positional advantage.
How frequently do you have to make your progress legible to measurable or consensus criteria? How lenient is your legibility loop?
I’m not saying it’s easy to even start trying to feel intrinsic motivation for building models in your own mind based on your own criteria for success, but being stuck in a short legibility loop certainly doesn’t help.
If you’ve learned to play an instrument, or studied painting under a mentor, you may have heard the advice “you need to learn you trust in your own sense of aesthetics.” Think of the kid who, while learning the piano, expectantly looks to their parent after every key they press. They’re not learning to listen. Sort of like a GAN with a discriminator trusted so infrequently that it never learns anything. Training to both generate and discriminate within yourself, using your own observations, will be pretty embarrassing at first, but you’re running a much shorter feedback loop.
Maybe we’re talking about different timescales here? I definitely think researchers need to be able to make progress without checking in with the community at every step, and most people won’t do well to try and publish their progress to a broad group, say, weekly. For a typical researcher in an area with poor natural feedback loops I’d guess the right frequency is something like:
Quarterly: medium-context peers (distant internal colleagues / close external colleagues)
Yearly: low-context peers and the general world
(I think there are a lot of advantages to writing for these, including being able to go back later, though there are also big advantages to verbal interaction and discussion.)
I think Leverage was primarily short on (3); from the outside I don’t know how much of (2) they were doing and I have the impression they were investing heavily in (1).
Roughly agreed. Although I’d want to distinguish between feedback and legibility-requirement loops. One is optimised for making research progress, the other is optimised for being paid and respected.
When you’re talking to your weekly colleagues, you have enough shared context and trust that you can ramble about your incomplete intuitions and say “oops, hang on” multiple times in an exposition. And medium-context peers are essential for sanity-checking. This is more about actually usefwl feedback than about paying a tax on speed to keep yourself legible to low-context funders.
Ah, but part of my point is that they’re inextricably linked—at least for pre-paradigmatic research that requires creativity and don’t have cheap empirical-legible measures of progress. Shorter legibility loops puts a heavy tax on the speed of progress, at least for the top of the competence distribution. I can’t make very general claims here given how different research fields and groups are, but I don’t want us to be blind to important considerations.
There are deeper models behind this claim, but one point is that the “legibility loops” you have to obey to receive funding requires you to compromise between optimisation criteria, and there are steeper invisible costs there than people realise.
Edit: The post has excellent nuance, and I make no claim to support or defend Leverage specifically (idk them). My comment is intended more generally, and my disagreement concerns two points:
You make several points, but I just want to respond to my impression that you’re trying to anchor wayward researchers or research groups to the “main paradigm” to decrease the chance that they’ll be wrong. I’m pretty strongly against this.
In a common-payoff game (like EA research), we all share the fruits of major discoveries regardless of who makes the discovery. So we should heavily prioritise sensitivity over specificity. It doesn’t matter how many research groups are wildly wrong, as long at least one research group figures out how to build the AI that satisfies our values with friendship and ponies. So when you’re trying to rein in researchers instead of letting them go off and explore highly variable crazy stuff, you’re putting all your eggs in one basket (the most respectable paradigm). Researchers are already heavily incentivised to research what other people are researching (the better to have a lively conversation!), so we do not need additional incentives against exploration.
The value distribution of research fruits is fat-tailed (citation needed). Strategies that are optimal for sampling normal distributions are unlikely to be optimal for fat tails. Sampling for outliers means that you should rely more on theoretical arguments, variability, and exploration, because you can’t get good data on the outliers—the only data that matters. If you insist on being legible and scientific, so you optimise your strategy based on the empirical data you can collect, you’re being fooled into mediocristan again.
Lemme cite a paper in network epistemology so I can fake looking like I know what I’m talking about,
That said, I also advocate against explorers being allowed to say
Explorers need to be way more epistemologically vigilant than staple researchers pursuing the safety of existing paradigms. If you leave your harbour to sail out into the open waters, that’s not a good time to forget your sextant, or pretend you’ll be a better navigator without studying the maps that do exist.
FWIW, I think conclusions from network-epistemological computer simulations are extremely weak evidence about what we as an irl research community should do, and I mainly benefit from it because they occasionally reveals patterns that help with analysing real-life phenomena. The field exists at all—despite their obviously irrelevant “experiments”—because it makes theoretical speculation seem more technical, impressive, professional.
Sort of? In your hypothetical there are two ways your research project could go once you believe you’ve succeeded:
You go and implement it, or
You figure out how to communicate your results to the rest of the industry.
If you go with (1) then it’s really important that you get things right, and if you’ve disconnected yourself from external evaluation I think there’s a large chance you haven’t. I’d much prefer to see (2), except now you do need to communicate your results in detail so the rest of the world can evaluate and so you didn’t gain that much by putting off the communication until the end.
I’ll also make a stronger claim, which is that communication improves your research and chances of success: figuring out how to communicate things to people who don’t have your shared context makes it a lot clearer which things you actually don’t understand yet.
I’m not sure why you think I’m advocating avoiding high-variability lines of research? I’m saying research groups should make public updates on their progress to stay grounded, not that they should only take low-risk bets.
I edited my original comment to point out my specific disagreements. I’m now going to say a selection of plausibly false-but-interesting things, and there’s much more nuance here that I won’t explicitly cover because that’d take too long. It’s definitely going to seem very wrong at first glance without the nuance that communicates the intended domain.
I feel like I’m in a somewhat similar situation to Leverage, only in the sense that I feel like having to frequently publish would hinder my effectiveness. It would make it easier for others to see the value of my work, but in my own estimation that trades off against maximising actual value.
This isn’t generally the case for most research, and I might be delusional (ime 10%) to think it’s the case for my own, but I should be following the gradient of what I expect will be the most usefwl. It would be selfish of me to do the legible thing motivated just by my wish for people to respect me.
The thing I’m arguing for is not that people like me shouldn’t publish at all, it’s that we should be very reluctant to punish gambling sailors for a shortage of signals. They’ll get our attention once they can demonstrate their product.
The thing about having to frequently communicate your results is that it incentivises you to adopt research strategies that lets you publish frequently. This usually means forward-chaining to incremental progress without much strategic guidance. Plus, if you get into the habit of spending your intrinsic motivation on distilling your progress to the community, now your brain’s shifted to searching for ideas that fit into the community, instead of aiming your search to solve the highest-priority confusion points in your own head.
To be an effective explorer, you have to get to the point where you can start to iterate on top of your own ideas. If you timidly “check in” with the community every time you think you have a novel thought, before you let yourself stand on it in order to explore further down the branch, then 1) you’re wasting their time, and 2) no one’s ever gonna stray far from home.
When you go from—
A) “huh, I wonder how this thing works, and how it fits into other things I have models of.”
to
B) “hmm, the community seems to behave as if Y is true, but I have a suspicion that ¬X,
so I should research it and provide them with information they find valuable.”
—then a pattern for generating thoughts will mostly be rewarded based on your prediction about whether the community is likely to be persuaded by those thoughts. This makes it hard to have intrinsic motivation to explore anything that doesn’t immediately seem relevant to the community.
And while B is still reasonably aligned with producing value as long as the community is roughly as good at evaluating the claims as you are, it breaks down for researchers who are much better than their expected audience at what they specialise in. If the most competent researchers have brains that optimise for communal persuasiveness, they’re wasting their potential when they could be searching for ideas that optimise for persuading themselves—a much harder criteria to meet given that they’re more competent.
I think it’s unhealthy to–within your own brain–constantly try to “advance the communal frontier”. Sure, that could ultimately be the goal, but if you’re greedily and myopically only able to optimise for specifically that at every step, then that is like a chess player who’s compulsively only able to look for checkmate patterns–unable to see forks that merely win material or positional advantage.
I’m not saying it’s easy to even start trying to feel intrinsic motivation for building models in your own mind based on your own criteria for success, but being stuck in a short legibility loop certainly doesn’t help.
If you’ve learned to play an instrument, or studied painting under a mentor, you may have heard the advice “you need to learn you trust in your own sense of aesthetics.” Think of the kid who, while learning the piano, expectantly looks to their parent after every key they press. They’re not learning to listen. Sort of like a GAN with a discriminator trusted so infrequently that it never learns anything. Training to both generate and discriminate within yourself, using your own observations, will be pretty embarrassing at first, but you’re running a much shorter feedback loop.
Maybe we’re talking about different timescales here? I definitely think researchers need to be able to make progress without checking in with the community at every step, and most people won’t do well to try and publish their progress to a broad group, say, weekly. For a typical researcher in an area with poor natural feedback loops I’d guess the right frequency is something like:
Weekly: high-context peers (internal colleagues / advisor / manager)
Quarterly: medium-context peers (distant internal colleagues / close external colleagues)
Yearly: low-context peers and the general world
(I think there are a lot of advantages to writing for these, including being able to go back later, though there are also big advantages to verbal interaction and discussion.)
I think Leverage was primarily short on (3); from the outside I don’t know how much of (2) they were doing and I have the impression they were investing heavily in (1).
Roughly agreed. Although I’d want to distinguish between feedback and legibility-requirement loops. One is optimised for making research progress, the other is optimised for being paid and respected.
When you’re talking to your weekly colleagues, you have enough shared context and trust that you can ramble about your incomplete intuitions and say “oops, hang on” multiple times in an exposition. And medium-context peers are essential for sanity-checking. This is more about actually usefwl feedback than about paying a tax on speed to keep yourself legible to low-context funders.
Thank you for chatting with me! ^^
(I’m only trying to talk about feedback here as it relates to research progress, not funding etc.)
Ah, but part of my point is that they’re inextricably linked—at least for pre-paradigmatic research that requires creativity and don’t have cheap empirical-legible measures of progress. Shorter legibility loops puts a heavy tax on the speed of progress, at least for the top of the competence distribution. I can’t make very general claims here given how different research fields and groups are, but I don’t want us to be blind to important considerations.
There are deeper models behind this claim, but one point is that the “legibility loops” you have to obey to receive funding requires you to compromise between optimisation criteria, and there are steeper invisible costs there than people realise.