″ I am struggling to think of a way to test predictive skill nicely without using probability forecasts though.”
The list of forecasting methods that do not rely on subjective or unverifiable probability estimates includes:
in response to a yes/no forecasting question, given what you believe now about the current situation and what it will cause, answer with whether the questioned future scenario is what the current situation leads to.
in response to a multiple option (mutually exclusive choice) forecasting question, given what you believe now about the current situation and what it could cause, answer what alternative future scenarios the current situation could lead to.
in response to any forecasting question, if you lack sufficient knowledge of the facts of the current situation, then list the contingencies and contingent situations that you would match to better knowledge of the facts in order to determine the future.
in response to any forecasting question, if you lack sufficient knowledge of causal relations that lead to a scenario, backtrack necessary (and together sufficient) causes from the scenario (one of the alternative forecast answers) to as close to the present as you can. From there, decide whether the causal gap between the present situation and that near-future situation (backtracked from the forecast scenario) is plausible to causally bridge. In other words, is that near-future situation causally possible? If you don’t believe so, reject the scenario in your forecast. If you do believe so, you have a starting point for research to close the information gap corresponding to the causal gap.
The list of forecasting methods that do not provide a final forecast probability includes:
for a yes/no forecast, use whatever probability methods to reach a conclusion however you do. Then establish a probability floor below which the answer is “no”, a ceiling above which the answer is “yes”, and a range between the floor and ceiling an answer of “maybe”. Work to reduce your uncertainty enough to move your forecast probability below or above the maybe zone.
for an alternatives forecast,use whatever probability methods to reach a conclusion however you do. Then establish a floor of probability above which an option is selected. List all options above the floor as chosen alternatives (for example, “I forecast scenario A or B but not C or D.”). Work to reduce your uncertainty enough to concentrate forecast probability within fewer options (for example, “I forecast scenario B only, not A or C or D.”).
The definition of scenario in the domain of forecasting (as I use it here) includes:
an option among several that answer a forecast multiple-choice (single-answer) question.
a proposed future about which a yes/no forecast is made.
You wrote:
The list of forecasting methods that do not rely on subjective or unverifiable probability estimates includes:
in response to a yes/no forecasting question, given what you believe now about the current situation and what it will cause, answer with whether the questioned future scenario is what the current situation leads to.
in response to a multiple option (mutually exclusive choice) forecasting question, given what you believe now about the current situation and what it could cause, answer what alternative future scenarios the current situation could lead to.
in response to any forecasting question, if you lack sufficient knowledge of the facts of the current situation, then list the contingencies and contingent situations that you would match to better knowledge of the facts in order to determine the future.
in response to any forecasting question, if you lack sufficient knowledge of causal relations that lead to a scenario, backtrack necessary (and together sufficient) causes from the scenario (one of the alternative forecast answers) to as close to the present as you can. From there, decide whether the causal gap between the present situation and that near-future situation (backtracked from the forecast scenario) is plausible to causally bridge. In other words, is that near-future situation causally possible? If you don’t believe so, reject the scenario in your forecast. If you do believe so, you have a starting point for research to close the information gap corresponding to the causal gap.
The list of forecasting methods that do not provide a final forecast probability includes:
for a yes/no forecast, use whatever probability methods to reach a conclusion however you do. Then establish a probability floor below which the answer is “no”, a ceiling above which the answer is “yes”, and a range between the floor and ceiling an answer of “maybe”. Work to reduce your uncertainty enough to move your forecast probability below or above the maybe zone.
for an alternatives forecast,use whatever probability methods to reach a conclusion however you do. Then establish a floor of probability above which an option is selected. List all options above the floor as chosen alternatives (for example, “I forecast scenario A or B but not C or D.”). Work to reduce your uncertainty enough to concentrate forecast probability within fewer options (for example, “I forecast scenario B only, not A or C or D.”).
The definition of scenario in the domain of forecasting (as I use it here) includes:
an option among several that answer a forecast multiple-choice (single-answer) question.
a proposed future about which a yes/no forecast is made.