For your shorter prompt you ended up using, I think you might get different results by changing the questions to be similar to the domain, or to “cover more domain space”, or “loosen up the probability space”.
Your questions are sort of “dry” and 2 of the 3 questions cover geopolitical issues. If you expanded this to be a little more “dramatic”, or had the prompt “express skill”, I think you would see different results.
Temperature and other parameters matter a lot too. Related to this, I think you sort of have an “N of 1”? I need to think about this, but that might not give much information about GPT-3′s performance.
Also, there’s several comments on your prompt, that you might have thought of before:
As you noted, your first “long” prompt is long and in this case, this impedes GPT-3′s performance. In my own words, I would say that makes it harder for GPT to “construct” the framing involved. https://github.com/MperorM/gpt3-metaculus/blob/main/gpt_prompt.py
For your shorter prompt you ended up using, I think you might get different results by changing the questions to be similar to the domain, or to “cover more domain space”, or “loosen up the probability space”.
Your questions are sort of “dry” and 2 of the 3 questions cover geopolitical issues. If you expanded this to be a little more “dramatic”, or had the prompt “express skill”, I think you would see different results.
More tips from prompt design are from Andrew Mayne. https://andrewmayneblog.wordpress.com/
Temperature and other parameters matter a lot too. Related to this, I think you sort of have an “N of 1”? I need to think about this, but that might not give much information about GPT-3′s performance.