On hits-based research: I certainly agree there are other factors to consider in making a funding decision. I’m just saying that you should talk about those directly instead of criticizing the OP for looking at whether their research was good or not.
(In your response to OP you talk about a positive case for the work on simulators, SVD, and sparse coding—that’s the sort of thing that I would want to see, so I’m glad to see that discussion starting.)
On VCs: Your position seems reasonable to me (though so does the OP’s position).
On recommendations: Fwiw I also make unconditional recommendations in private. I don’t think this is unusual, e.g. I think many people make unconditional recommendations not to go into academia (though I don’t).
I don’t really buy that the burden of proof should be much higher in public. Reversing the position, do you think the burden of proof should be very high for anyone to publicly recommend working at lab X? If not, what’s the difference between a recommendation to work at org X vs an anti-recommendation (i.e. recommendation not to work at org X)? I think the three main considerations I’d point to are:
(Pro-recommendations) It’s rare for people to do things (relative to not doing things), so we differentially want recommendations vs anti-recommendations, so that it is easier for orgs to start up and do things.
(Anti-recommendations) There are strong incentives to recommend working at org X (obviously org X itself will do this), but no incentives to make the opposite recommendation (and in fact usually anti-incentives). Similarly I expect that inaccuracies in the case for the not-working recommendation will be pointed out (by org X), whereas inaccuracies in the case for working will not be pointed out. So we differentially want to encourage the opposite recommendations in order to get both sides of the story by lowering our “burden of proof”.
(Pro-recommendations) Recommendations have a nice effect of getting people excited and positive about the work done by the community, which can make people more motivated, whereas the same is not true of anti-recommendations.
Overall I think point 2 feels most important, and so I end up thinking that the burden of proof on critiques / anti-recommendations should be lower than the burden of proof on recommendations—and the burden of proof on recommendations is approximately zero. (E.g. if someone wrote a public post recommending Conjecture without any concrete details of why—just something along the lines of “it’s a great place doing great work”—I don’t think anyone would say that they were using their power irresponsibly.)
I would actually prefer a higher burden of proof on recommendations, but given the status quo if I’m only allowed to affect the burden of proof on anti-recommendations I’d probably want it to go down to ~zero. Certainly I’d want it to be well below the level that this post meets.
Hmm, yeah. I actually think you changed my mind on the recommendations. My new position is something like: 1. There should not be a higher burden on anti-recommendations than pro-recommendations. 2. Both pro- and anti-recommendations should come with caveats and conditionals whenever they make a difference to the target audience. 3. I’m now more convinced that the anti-recommendation of OP was appropriate. 4. I’d probably still phrase it differently than they did but my overall belief went from “this was unjustified” to “they should have used different wording” which is a substantial change in position. 5. In general, the context in which you make a recommendation still matters. For example, if you make a public comment saying “I’d probably not recommend working for X” the severity feels different than “I collected a lot of evidence and wrote this entire post and now recommend against working for X”. But I guess that just changes the effect size and not really the content of the recommendation.
On hits-based research: I certainly agree there are other factors to consider in making a funding decision. I’m just saying that you should talk about those directly instead of criticizing the OP for looking at whether their research was good or not.
(In your response to OP you talk about a positive case for the work on simulators, SVD, and sparse coding—that’s the sort of thing that I would want to see, so I’m glad to see that discussion starting.)
On VCs: Your position seems reasonable to me (though so does the OP’s position).
On recommendations: Fwiw I also make unconditional recommendations in private. I don’t think this is unusual, e.g. I think many people make unconditional recommendations not to go into academia (though I don’t).
I don’t really buy that the burden of proof should be much higher in public. Reversing the position, do you think the burden of proof should be very high for anyone to publicly recommend working at lab X? If not, what’s the difference between a recommendation to work at org X vs an anti-recommendation (i.e. recommendation not to work at org X)? I think the three main considerations I’d point to are:
(Pro-recommendations) It’s rare for people to do things (relative to not doing things), so we differentially want recommendations vs anti-recommendations, so that it is easier for orgs to start up and do things.
(Anti-recommendations) There are strong incentives to recommend working at org X (obviously org X itself will do this), but no incentives to make the opposite recommendation (and in fact usually anti-incentives). Similarly I expect that inaccuracies in the case for the not-working recommendation will be pointed out (by org X), whereas inaccuracies in the case for working will not be pointed out. So we differentially want to encourage the opposite recommendations in order to get both sides of the story by lowering our “burden of proof”.
(Pro-recommendations) Recommendations have a nice effect of getting people excited and positive about the work done by the community, which can make people more motivated, whereas the same is not true of anti-recommendations.
Overall I think point 2 feels most important, and so I end up thinking that the burden of proof on critiques / anti-recommendations should be lower than the burden of proof on recommendations—and the burden of proof on recommendations is approximately zero. (E.g. if someone wrote a public post recommending Conjecture without any concrete details of why—just something along the lines of “it’s a great place doing great work”—I don’t think anyone would say that they were using their power irresponsibly.)
I would actually prefer a higher burden of proof on recommendations, but given the status quo if I’m only allowed to affect the burden of proof on anti-recommendations I’d probably want it to go down to ~zero. Certainly I’d want it to be well below the level that this post meets.
Hmm, yeah. I actually think you changed my mind on the recommendations. My new position is something like:
1. There should not be a higher burden on anti-recommendations than pro-recommendations.
2. Both pro- and anti-recommendations should come with caveats and conditionals whenever they make a difference to the target audience.
3. I’m now more convinced that the anti-recommendation of OP was appropriate.
4. I’d probably still phrase it differently than they did but my overall belief went from “this was unjustified” to “they should have used different wording” which is a substantial change in position.
5. In general, the context in which you make a recommendation still matters. For example, if you make a public comment saying “I’d probably not recommend working for X” the severity feels different than “I collected a lot of evidence and wrote this entire post and now recommend against working for X”. But I guess that just changes the effect size and not really the content of the recommendation.
:) I’m glad we got to agreement!
(Or at least significantly closer, I’m sure there are still some minor differences.)