I have really enjoyed reading this report. Thanks for producing this public good! Here are some thoughts I had while reading it.
First, when should a sponsor use a contest and not some other mechanism? I agree with the report that a contest is e a good idea when marketplace incentives are weak, and the path to the innovation is not clear. I’d add that even in those circumstances, the sponsor could engage in contract research or provide a grant to a researcher. I would guess that contests are preferable to those alternatives when identifying suitable candidates for grants/research is difficult or moral hazard problems are particularly salient.
Second, what are the possible designs of innovation contests that could be used?
One important design decision is whether to have a fixed deadline when the participants are evaluated and the winners are determined (as in the 2005 DARPA Grand Challenge) or to have a fixed objective so that the first participant to achieve the objective wins (as in Clay Institute’s Millenium Prize Problems or the original Longitude Prize). The advantage of a fixed deadline is that the goal specification can be more vague, but runs the risk that no participant will have a satisfactory result. Fixed objective prizes can run (and provide incentives) for a long time, but require a very clear and objective goal specification.
With fixed deadline contests, the designer can also choose whether to have elimination rounds (like in NRG Cosia X Prize contest). I am not familiar with any work analyzing the effect of elimination rounds.
Whether to give one or multiple final prizes (which was discussed in the report) and whether to give interim prizes while the contest is ongoing (as was the case in the Netflix Prize contest).
What information to release to the participants during the contest, especially about their current standing relative to other participants? For example, whether to have a leaderboard or not. (Lemus and Marshall have some interesting results on this.)
Third, recognition prizes are cool, but I don’t see how they can be used to generate research aimed at solving a specific problem. As such, they can (likely?) only provide broad incentives to engage in research in some field.
Fourth, not all innovation contests need to generate a lot of publicity. I am actually hoping to see more of smaller, targeted innovation contests that aim to solve real but unglamorous problems people face (in particular, those in less developed countries). With such problems, monetary rewards may be more important than they were in, say, the Ansari Prize.
Thanks a lot for your elaborate and thoughtful comment! A quick reaction to your thoughts:
Unfortunately, the literature we reviewed did not seem to be very clear-cut on the question of when exactly to use prizes vs. grants (or other incentives). Intuitively, I’d agree that a prize makes sense (vs. a grant) when identifying a suitable candidate is difficult. To me, this point is already broadly covered by “when the goal is clear, but the path to achieving it is not”, as when you don’t know how to solve something, you may also not know who could solve it. Could you give an example of how moral hazard can come into play?
Thanks for pointing out more design issues! Our report is definitely not exhaustive with regard to how best to design a prize. I don’t fully remember why we did not include the specific design issues you mention, but it is likely because we didn’t find good (quasi-) experimental literature on them. Case studies might be useful here.
I agree that recognition prizes are likely less useful than inducement prizes when you have a very specific problem to be solved. I think recognition prizes are useful when you generally want to increase research and attention to a specific topic, which can help reveal new problems to be solved that you didn’t even think of in the first place.
I think I share your intuition here. I can definitely imagine that financial incentives might potentially be more important in smaller, unglamorous prizes. We focused on large innovation prizes in our report, so I am not sure what’s the most effective incentive structure for small prizes.
I have really enjoyed reading this report. Thanks for producing this public good! Here are some thoughts I had while reading it.
First, when should a sponsor use a contest and not some other mechanism? I agree with the report that a contest is e a good idea when marketplace incentives are weak, and the path to the innovation is not clear. I’d add that even in those circumstances, the sponsor could engage in contract research or provide a grant to a researcher. I would guess that contests are preferable to those alternatives when identifying suitable candidates for grants/research is difficult or moral hazard problems are particularly salient.
Second, what are the possible designs of innovation contests that could be used?
One important design decision is whether to have a fixed deadline when the participants are evaluated and the winners are determined (as in the 2005 DARPA Grand Challenge) or to have a fixed objective so that the first participant to achieve the objective wins (as in Clay Institute’s Millenium Prize Problems or the original Longitude Prize). The advantage of a fixed deadline is that the goal specification can be more vague, but runs the risk that no participant will have a satisfactory result. Fixed objective prizes can run (and provide incentives) for a long time, but require a very clear and objective goal specification.
With fixed deadline contests, the designer can also choose whether to have elimination rounds (like in NRG Cosia X Prize contest). I am not familiar with any work analyzing the effect of elimination rounds.
Whether to give one or multiple final prizes (which was discussed in the report) and whether to give interim prizes while the contest is ongoing (as was the case in the Netflix Prize contest).
What information to release to the participants during the contest, especially about their current standing relative to other participants? For example, whether to have a leaderboard or not. (Lemus and Marshall have some interesting results on this.)
Third, recognition prizes are cool, but I don’t see how they can be used to generate research aimed at solving a specific problem. As such, they can (likely?) only provide broad incentives to engage in research in some field.
Fourth, not all innovation contests need to generate a lot of publicity. I am actually hoping to see more of smaller, targeted innovation contests that aim to solve real but unglamorous problems people face (in particular, those in less developed countries). With such problems, monetary rewards may be more important than they were in, say, the Ansari Prize.
Thanks a lot for your elaborate and thoughtful comment! A quick reaction to your thoughts:
Unfortunately, the literature we reviewed did not seem to be very clear-cut on the question of when exactly to use prizes vs. grants (or other incentives). Intuitively, I’d agree that a prize makes sense (vs. a grant) when identifying a suitable candidate is difficult. To me, this point is already broadly covered by “when the goal is clear, but the path to achieving it is not”, as when you don’t know how to solve something, you may also not know who could solve it. Could you give an example of how moral hazard can come into play?
Thanks for pointing out more design issues! Our report is definitely not exhaustive with regard to how best to design a prize. I don’t fully remember why we did not include the specific design issues you mention, but it is likely because we didn’t find good (quasi-) experimental literature on them. Case studies might be useful here.
I agree that recognition prizes are likely less useful than inducement prizes when you have a very specific problem to be solved. I think recognition prizes are useful when you generally want to increase research and attention to a specific topic, which can help reveal new problems to be solved that you didn’t even think of in the first place.
I think I share your intuition here. I can definitely imagine that financial incentives might potentially be more important in smaller, unglamorous prizes. We focused on large innovation prizes in our report, so I am not sure what’s the most effective incentive structure for small prizes.