The awakening of slumbering papers may be fundamentally un- predictable in part because science itself must advance before the implications of the discovery can unfold.
Except to the authors themselves, who may often have an inkling that their paper is important. E.g., I think Rosenblatt was incredibly excited/convinced about the insights in that sleeping beauty paper. (Small chance my memory is wrong about this, or that he changed his mind at some point.)
I don’t think this is just a nitpicky comment on the passage you quoted. I find it plausible that there’s some hard-to-study quantity around ‘research taste’ that predicts impact quite well. It’d be hard to study because the hypothesis is that only very few people have it. To tell who has it, you kind of need to have it a bit yourself. But one decent way to measure it is asking people who are universally regarded as ‘having it’ to comment on who else they think also has it. (I know this process would lead to unfair network effects and may result in false negatives and so on, but I’m advancing a descriptive observation here; I’m not advocating for a specific system on how to evaluate individuals.)
Related: I remember a comment (can’t find it anymore) somewhere by Liv Boeree or some other poker player familiar with EA. The commenter explained that monetary results aren’t the greatest metric for assessing the skill of top poker players. Instead, it’s best to go with assessments by expert peers. (I think this holds mostly for large-field tournaments, not online cash games.)
Related: I remember a comment (can’t find it anymore) somewhere by Liv Boeree or some other poker player familiar with EA. The commenter explained that monetary results aren’t the greatest metric for assessing the skill of top poker players. Instead, it’s best to go with assessments by expert peers. (I think this holds mostly for large-field tournaments, not online cash games.)
If I remember correctly, Linchuan Zhang made or referred to that comment somewhere on the Forum when saying that it was similar with assessing forecaster skill. (Or maybe it was you? :P)
I have indeed made that comment somewhere. It was one of the more insightful/memorable comments she made when I interviewed her, but tragically I didn’t end up writing down that question in the final document (maybe due to my own lack of researcher taste? :P)
That said, human memory is fallible etc so maybe it’d be worthwhile to circle back to Liv and ask if she still endorses this, and/or ask other poker players how much they agree with it.
I’ve been much less successful than LivB but would endorse it, though I’d note that there are substantially better objective metrics than cash prizes for many kinds of online play, and I’d have a harder time arguing that those were less reliable than subjective judgements of other good players. It somewhat depends on sample though, at the highest stakes the combination of v small playerpool and fairly small samples make this quite believable.
To give an example of what would go into research taste, consider the issue of reference class tennis (rationalist jargon for arguments on whether a given analogy has merit, or two people throwing widely different analogies at each other in an argument). That issue comes up a lot especially in preparadigmatic branches of science. Some people may have good intuitions about this sort of thing, while others may be hopelessly bad at it. Since arguments of that form feel notoriously intractable to outsiders, it would make sense if “being good at reference class tennis” were a skill that’s hard to evaluate.
Except to the authors themselves, who may often have an inkling that their paper is important. E.g., I think Rosenblatt was incredibly excited/convinced about the insights in that sleeping beauty paper. (Small chance my memory is wrong about this, or that he changed his mind at some point.)
I don’t think this is just a nitpicky comment on the passage you quoted. I find it plausible that there’s some hard-to-study quantity around ‘research taste’ that predicts impact quite well. It’d be hard to study because the hypothesis is that only very few people have it. To tell who has it, you kind of need to have it a bit yourself. But one decent way to measure it is asking people who are universally regarded as ‘having it’ to comment on who else they think also has it. (I know this process would lead to unfair network effects and may result in false negatives and so on, but I’m advancing a descriptive observation here; I’m not advocating for a specific system on how to evaluate individuals.)
Related: I remember a comment (can’t find it anymore) somewhere by Liv Boeree or some other poker player familiar with EA. The commenter explained that monetary results aren’t the greatest metric for assessing the skill of top poker players. Instead, it’s best to go with assessments by expert peers. (I think this holds mostly for large-field tournaments, not online cash games.)
If I remember correctly, Linchuan Zhang made or referred to that comment somewhere on the Forum when saying that it was similar with assessing forecaster skill. (Or maybe it was you? :P)
I have indeed made that comment somewhere. It was one of the more insightful/memorable comments she made when I interviewed her, but tragically I didn’t end up writing down that question in the final document (maybe due to my own lack of researcher taste? :P)
That said, human memory is fallible etc so maybe it’d be worthwhile to circle back to Liv and ask if she still endorses this, and/or ask other poker players how much they agree with it.
I’ve been much less successful than LivB but would endorse it, though I’d note that there are substantially better objective metrics than cash prizes for many kinds of online play, and I’d have a harder time arguing that those were less reliable than subjective judgements of other good players. It somewhat depends on sample though, at the highest stakes the combination of v small playerpool and fairly small samples make this quite believable.
To give an example of what would go into research taste, consider the issue of reference class tennis (rationalist jargon for arguments on whether a given analogy has merit, or two people throwing widely different analogies at each other in an argument). That issue comes up a lot especially in preparadigmatic branches of science. Some people may have good intuitions about this sort of thing, while others may be hopelessly bad at it. Since arguments of that form feel notoriously intractable to outsiders, it would make sense if “being good at reference class tennis” were a skill that’s hard to evaluate.