One possible synthesis comes from Turing award winner Richard Hamming’s book The Art of Doing Science and Engineering. He’s got chapters at the end on Creativity and Experts. The chapters are somewhat rambly and I’ve quoted passages below. My attempt to summarize Hamming’s position: Having a deep intellectual toolkit is valuable, but experts are often overconfident and resistant to new ideas.
Chapter 25: Creativity
...Do not be too hasty [in refining a problem], as you are likely to put the problem in the conventional form and find only the conventional solution...
...
...Wide acquaintance with various fields of knowledge is thus a help—provided you have the knowledge filed away so it is available when needed, rather than to be found only when led directly to it. This flexible access to pieces of knowledge seems to come from looking at knowledge while you are acquiring it from many different angles, turning over any new idea to see its many sides before filing it away. This implies effort on your part not to take the easy, immediately useful “memorizing the material” path, but prepare your mind for the future.
...
Over the years of watching and working with John Tukey I found many times he recalled the relevant information and I did not, until he pointed it out to me. Clearly his information retrieval system had many more “hooks” than mine did. At least more useful ones! How could this be? Probably because he was more in the
habit than I was of turning over new information again and again so his “hooks” for retrieval were more numerous and significantly better than mine were. Hence wishing I could similarly do what he did, I started to mull over new ideas, trying to make significant “hooks” to relevant information so when later I went fishing for an idea I had a better chance of finding an analogy. I can only advise you to do what I tried to do—when you learn something new think of other applications of it—ones which have not arisen in your past but which might in your future. How easy to say, but how hard to do! Yet, what else can I say about how to organize your mind so useful things will be recalled readily at the right time?
...
...Without self-confidence you are not likely to create great, new things. There is a thin line between having enough self-confidence and being over-confident. I suppose the difference is whether you succeed or fail; when you win you are strong willed, and when you lose you are stubborn!...
Chapter 26: Experts
...
In an argument between a specialist and a generalist the expert usually wins by simply: (1) using unintelligible jargon, and (2) citing their specialist results which are often completely irrelevant to the discussion. The expert is, therefore, a potent factor to be reckoned with in our society. Since experts are both necessary, and also at times do great harm in blocking significant progress, they need to be examined closely. All too often the expert misunderstands the problem at hand, but the generalist cannot carry though their side to completion. The person who thinks they understand the problem and does not is usually more of a curse (blockage) than the person who knows they do not understand the problem.
...
Experts in looking at something new always bring their expertise with them as well as their particular way of looking at things. Whatever does not fit into their frame of reference is dismissed, not seen, or forced to fit into their beliefs. Thus really new ideas seldom arise from the experts in the field. You can not blame them too much since it is more economical to try the old, successful ways before trying to find new ways of looking and thinking.
All things which are proved to be impossible must obviously rest on some assumptions, and when one or more of these assumptions are not true then the impossibility proof fails—but the expert seldom remembers to carefully inspect the assumptions before making their “impossible” statements. There is an old statement which covers this aspect of the expert. It goes as follows: “If an expert says something can be done he is probably correct, but if he says it is impossible then consider getting another opinion.”
...
...It appears most of the great innovations come from outside the field, and not from the insiders… examples occur in most fields of work, but the text books seldom, if ever, discuss this aspect.
...the expert faces the following dilemma. Outside the field there are a large number of genuine crackpots with their crazy ideas, but among them may also be the crackpot with the new, innovative idea which is going to triumph. What is a rational strategy for the expert to adopt? Most decide they will ignore, as best they can, all crackpots, thus ensuring they will not be part of the new paradigm, if and when it comes.
Those experts who do look for the possible innovative crackpot are likely to spend their lives in the futile pursuit of the elusive, rare crackpot with the right idea, the only idea which really matters in the long run. Obviously the strategy for you to adopt depends on how much you are willing to be merely one of those who served to advance things, vs. the desire to be one of the few who in the long run really matter. I cannot tell you which you should choose that is your choice. But I do say you should be conscious of making the choice as you pursue your career. Do not just drift along; think of what you want to be and how to get there. Do not automatically reject every crazy idea, the moment you hear of it, especially when it comes from outside the official circle of the insiders—it may be the great new approach which will change the paradigm of the field! But also you cannot afford to pursue every “crackpot” idea you hear about. I have been talking about paradigms of Science, but so far as I know the same applies to most fields of human thought, though I have not investigated them closely. And it probably happens for about the same reasons; the insiders are too sure of themselves, have too much invested in the accepted approaches, and are plain mentally lazy. Think of the history of modern technology you know!
...
...In some respects the expert is the curse of our society with their assurance they know everything, and without the decent humility to consider they might be wrong. Where the question looms so important I suggested to you long ago to use in an argument, “What would you accept as evidence you are wrong?” Ask yourself regularly, “Why do I believe whatever I do”. Especially in the areas where you are so sure you know; the area of the paradigms of your field.
Hamming shares a number of stories from the history of science to support his claims. He also says he has more stories which he didn’t include in the chapter, and that he looked for stories which went against his position too.
A couple takeaways:
Survivorship bias regarding stories of successful contrarians—most apparent crackpots actually are crackpots.
Paradigm shifts—if an apparent crackpot is not actually a crackpot, their idea has the potential to be extremely important. So shutting down all the apparent crackpots could have quite a high cost even if most are full of nonsense. As Jerome Friedman put it regarding the invention of bagging (coincidentally mentioned in the main post):
The first time I saw this—when would that have been, maybe the mid ’90s--
I knew a lot about the bootstrap. Actually, I was a student of Brad Efron, who
invented the bootstrap. And Brad and I wrote a book together on the bootstrap in
the early ’90s. And then when I saw the bag idea from Leo, I thought this
looks really crazy. Usually the bootstrap is used to get the idea of standard errors or bias, but Leo wants to use bootstrap to produce a whole bunch of trees and to average them, which sounded really crazy to me. And it was a reminder to me that you see an idea that looks really crazy, it’s got a reasonable chance of actually
being really good. If things look very familiar, they’re not likely to be big
steps forward. This was a big step forward, and took me and others a long
time to realize that.
However, even if one accepts the premise that apparent crackpots deliver surprisingly high expected value, it’s still not obvious how many we want on the Forum!
One possible synthesis comes from Turing award winner Richard Hamming’s book The Art of Doing Science and Engineering. He’s got chapters at the end on Creativity and Experts. The chapters are somewhat rambly and I’ve quoted passages below. My attempt to summarize Hamming’s position: Having a deep intellectual toolkit is valuable, but experts are often overconfident and resistant to new ideas.
Chapter 25: Creativity
Chapter 26: Experts
Hamming shares a number of stories from the history of science to support his claims. He also says he has more stories which he didn’t include in the chapter, and that he looked for stories which went against his position too.
A couple takeaways:
Survivorship bias regarding stories of successful contrarians—most apparent crackpots actually are crackpots.
Paradigm shifts—if an apparent crackpot is not actually a crackpot, their idea has the potential to be extremely important. So shutting down all the apparent crackpots could have quite a high cost even if most are full of nonsense. As Jerome Friedman put it regarding the invention of bagging (coincidentally mentioned in the main post):
However, even if one accepts the premise that apparent crackpots deliver surprisingly high expected value, it’s still not obvious how many we want on the Forum!