It doesnât matter how many research groups are wildly wrong, as long at least one research group figures out how to build the AI that satisfies our values with friendship and ponies.
Sort of? In your hypothetical there are two ways your research project could go once you believe youâve succeeded:
You go and implement it, or
You figure out how to communicate your results to the rest of the industry.
If you go with (1) then itâs really important that you get things right, and if youâve disconnected yourself from external evaluation I think thereâs a large chance you havenât. Iâd much prefer to see (2), except now you do need to communicate your results in detail so the rest of the world can evaluate and so you didnât gain that much by putting off the communication until the end.
Iâll also make a stronger claim, which is that communication improves your research and chances of success: figuring out how to communicate things to people who donât have your shared context makes it a lot clearer which things you actually donât understand yet.
trying to rein in researchers instead of letting them go off and explore highly variable crazy stuff
Iâm not sure why you think Iâm advocating avoiding high-variability lines of research? Iâm saying research groups should make public updates on their progress to stay grounded, not that they should only take low-risk bets.
I edited my original comment to point out my specific disagreements. Iâm now going to say a selection of plausibly false-but-interesting things, and thereâs much more nuance here that I wonât explicitly cover because thatâd take too long. Itâs definitely going to seem very wrong at first glance without the nuance that communicates the intended domain.
I feel like Iâm in a somewhat similar situation to Leverage, only in the sense that I feel like having to frequently publish would hinder my effectiveness. It would make it easier for others to see the value of my work, but in my own estimation that trades off against maximising actual value.
This isnât generally the case for most research, and I might be delusional (ime 10%) to think itâs the case for my own, but I should be following the gradient of what I expect will be the most usefwl. It would be selfish of me to do the legible thing motivated just by my wish for people to respect me.
The thing Iâm arguing for is not that people like me shouldnât publish at all, itâs that we should be very reluctant to punish gambling sailors for a shortage of signals. Theyâll get our attention once they can demonstrate their product.
The thing about having to frequently communicate your results is that it incentivises you to adopt research strategies that lets you publish frequently. This usually means forward-chaining to incremental progress without much strategic guidance. Plus, if you get into the habit of spending your intrinsic motivation on distilling your progress to the community, now your brainâs shifted to searching for ideas that fitinto the community, instead of aiming your search to solve the highest-priority confusion points in your own head.
To be an effective explorer, you have to get to the point where you can start to iterate on top of your own ideas. If you timidly âcheck inâ with the community every time you think you have a novel thought, before you let yourself stand on it in order to explore further down the branch, then 1) youâre wasting their time, and 2) no oneâs ever gonna stray far from home.
When you go fromâ
A) âhuh, I wonder how this thing works, and how it fits into other things I have models of.â to B) âhmm, the community seems to behave as if Y is true, but I have a suspicion that ÂŹX, so I should research it and provide them with information they find valuable.â
âthen a pattern for generating thoughts will mostly be rewarded based on your prediction about whether the community is likely to be persuaded by those thoughts. This makes it hard to have intrinsic motivation to explore anything that doesnât immediately seem relevant to the community.
And while B is still reasonably aligned with producing value as long as the community is roughly as good at evaluating the claims as you are, it breaks down for researchers who are much better than their expected audience at what they specialise in. If the most competent researchers have brains that optimise for communal persuasiveness, theyâre wasting their potential when they could be searching for ideas that optimise for persuading themselvesâa much harder criteria to meet given that theyâre more competent.
I think itâs unhealthy toâwithin your own brainâconstantly try to âadvance the communal frontierâ. Sure, that could ultimately be the goal, but if youâre greedily and myopically only able to optimise for specifically that at every step, then that is like a chess player whoâs compulsively only able to look for checkmate patternsâunable to see forks that merely win material or positional advantage.
How frequently do you have to make your progress legible to measurable or consensus criteria? How lenient is your legibility loop?
Iâm not saying itâs easy to even start trying to feel intrinsic motivation for building models in your own mind based on your own criteria for success, but being stuck in a short legibility loop certainly doesnât help.
If youâve learned to play an instrument, or studied painting under a mentor, you may have heard the advice âyou need to learn you trust in your own sense of aesthetics.â Think of the kid who, while learning the piano, expectantly looks to their parent after every key they press. Theyâre not learning to listen. Sort of like a GAN with a discriminator trusted so infrequently that it never learns anything. Training to both generate and discriminate within yourself, using your own observations, will be pretty embarrassing at first, but youâre running a much shorter feedback loop.
Maybe weâre talking about different timescales here? I definitely think researchers need to be able to make progress without checking in with the community at every step, and most people wonât do well to try and publish their progress to a broad group, say, weekly. For a typical researcher in an area with poor natural feedback loops Iâd guess the right frequency is something like:
Quarterly: medium-context peers (distant internal colleagues /â close external colleagues)
Yearly: low-context peers and the general world
(I think there are a lot of advantages to writing for these, including being able to go back later, though there are also big advantages to verbal interaction and discussion.)
I think Leverage was primarily short on (3); from the outside I donât know how much of (2) they were doing and I have the impression they were investing heavily in (1).
Roughly agreed. Although Iâd want to distinguish between feedback and legibility-requirement loops. One is optimised for making research progress, the other is optimised for being paid and respected.
When youâre talking to your weekly colleagues, you have enough shared context and trust that you can ramble about your incomplete intuitions and say âoops, hang onâ multiple times in an exposition. And medium-context peers are essential for sanity-checking. This is more about actually usefwl feedback than about paying a tax on speed to keep yourself legible to low-context funders.
Ah, but part of my point is that theyâre inextricably linkedâat least for pre-paradigmatic research that requires creativity and donât have cheap empirical-legible measures of progress. Shorter legibility loops puts a heavy tax on the speed of progress, at least for the top of the competence distribution. I canât make very general claims here given how different research fields and groups are, but I donât want us to be blind to important considerations.
There are deeper models behind this claim, but one point is that the âlegibility loopsâ you have to obey to receive funding requires you to compromise between optimisation criteria, and there are steeper invisible costs there than people realise.
Sort of? In your hypothetical there are two ways your research project could go once you believe youâve succeeded:
You go and implement it, or
You figure out how to communicate your results to the rest of the industry.
If you go with (1) then itâs really important that you get things right, and if youâve disconnected yourself from external evaluation I think thereâs a large chance you havenât. Iâd much prefer to see (2), except now you do need to communicate your results in detail so the rest of the world can evaluate and so you didnât gain that much by putting off the communication until the end.
Iâll also make a stronger claim, which is that communication improves your research and chances of success: figuring out how to communicate things to people who donât have your shared context makes it a lot clearer which things you actually donât understand yet.
Iâm not sure why you think Iâm advocating avoiding high-variability lines of research? Iâm saying research groups should make public updates on their progress to stay grounded, not that they should only take low-risk bets.
I edited my original comment to point out my specific disagreements. Iâm now going to say a selection of plausibly false-but-interesting things, and thereâs much more nuance here that I wonât explicitly cover because thatâd take too long. Itâs definitely going to seem very wrong at first glance without the nuance that communicates the intended domain.
I feel like Iâm in a somewhat similar situation to Leverage, only in the sense that I feel like having to frequently publish would hinder my effectiveness. It would make it easier for others to see the value of my work, but in my own estimation that trades off against maximising actual value.
This isnât generally the case for most research, and I might be delusional (ime 10%) to think itâs the case for my own, but I should be following the gradient of what I expect will be the most usefwl. It would be selfish of me to do the legible thing motivated just by my wish for people to respect me.
The thing Iâm arguing for is not that people like me shouldnât publish at all, itâs that we should be very reluctant to punish gambling sailors for a shortage of signals. Theyâll get our attention once they can demonstrate their product.
The thing about having to frequently communicate your results is that it incentivises you to adopt research strategies that lets you publish frequently. This usually means forward-chaining to incremental progress without much strategic guidance. Plus, if you get into the habit of spending your intrinsic motivation on distilling your progress to the community, now your brainâs shifted to searching for ideas that fit into the community, instead of aiming your search to solve the highest-priority confusion points in your own head.
To be an effective explorer, you have to get to the point where you can start to iterate on top of your own ideas. If you timidly âcheck inâ with the community every time you think you have a novel thought, before you let yourself stand on it in order to explore further down the branch, then 1) youâre wasting their time, and 2) no oneâs ever gonna stray far from home.
When you go fromâ
A) âhuh, I wonder how this thing works, and how it fits into other things I have models of.â
to
B) âhmm, the community seems to behave as if Y is true, but I have a suspicion that ÂŹX,
so I should research it and provide them with information they find valuable.â
âthen a pattern for generating thoughts will mostly be rewarded based on your prediction about whether the community is likely to be persuaded by those thoughts. This makes it hard to have intrinsic motivation to explore anything that doesnât immediately seem relevant to the community.
And while B is still reasonably aligned with producing value as long as the community is roughly as good at evaluating the claims as you are, it breaks down for researchers who are much better than their expected audience at what they specialise in. If the most competent researchers have brains that optimise for communal persuasiveness, theyâre wasting their potential when they could be searching for ideas that optimise for persuading themselvesâa much harder criteria to meet given that theyâre more competent.
I think itâs unhealthy toâwithin your own brainâconstantly try to âadvance the communal frontierâ. Sure, that could ultimately be the goal, but if youâre greedily and myopically only able to optimise for specifically that at every step, then that is like a chess player whoâs compulsively only able to look for checkmate patternsâunable to see forks that merely win material or positional advantage.
Iâm not saying itâs easy to even start trying to feel intrinsic motivation for building models in your own mind based on your own criteria for success, but being stuck in a short legibility loop certainly doesnât help.
If youâve learned to play an instrument, or studied painting under a mentor, you may have heard the advice âyou need to learn you trust in your own sense of aesthetics.â Think of the kid who, while learning the piano, expectantly looks to their parent after every key they press. Theyâre not learning to listen. Sort of like a GAN with a discriminator trusted so infrequently that it never learns anything. Training to both generate and discriminate within yourself, using your own observations, will be pretty embarrassing at first, but youâre running a much shorter feedback loop.
Maybe weâre talking about different timescales here? I definitely think researchers need to be able to make progress without checking in with the community at every step, and most people wonât do well to try and publish their progress to a broad group, say, weekly. For a typical researcher in an area with poor natural feedback loops Iâd guess the right frequency is something like:
Weekly: high-context peers (internal colleagues /â advisor /â manager)
Quarterly: medium-context peers (distant internal colleagues /â close external colleagues)
Yearly: low-context peers and the general world
(I think there are a lot of advantages to writing for these, including being able to go back later, though there are also big advantages to verbal interaction and discussion.)
I think Leverage was primarily short on (3); from the outside I donât know how much of (2) they were doing and I have the impression they were investing heavily in (1).
Roughly agreed. Although Iâd want to distinguish between feedback and legibility-requirement loops. One is optimised for making research progress, the other is optimised for being paid and respected.
When youâre talking to your weekly colleagues, you have enough shared context and trust that you can ramble about your incomplete intuitions and say âoops, hang onâ multiple times in an exposition. And medium-context peers are essential for sanity-checking. This is more about actually usefwl feedback than about paying a tax on speed to keep yourself legible to low-context funders.
Thank you for chatting with me! ^^
(Iâm only trying to talk about feedback here as it relates to research progress, not funding etc.)
Ah, but part of my point is that theyâre inextricably linkedâat least for pre-paradigmatic research that requires creativity and donât have cheap empirical-legible measures of progress. Shorter legibility loops puts a heavy tax on the speed of progress, at least for the top of the competence distribution. I canât make very general claims here given how different research fields and groups are, but I donât want us to be blind to important considerations.
There are deeper models behind this claim, but one point is that the âlegibility loopsâ you have to obey to receive funding requires you to compromise between optimisation criteria, and there are steeper invisible costs there than people realise.