An anecdote: the US government is trying to convince a foreign government to sign an agreement with the United States but is repeatedly stymied by presidents from both parties for two decades. Let’s assume a forecast at that moment suggests a 10% change the law will be passed within a year. A creative new ambassador designs a creative new strategy that hasn’t been attempted before. Though the agreement would require executive signature, she’s decides instead to meet with every single member of parliament and tell them the United States would owe them if they came out publicly in favor of the deal. Fast forward a year, and the agreement is signed.
Another anecdote: the invention of the Apple computer.
Presumably you could use LLM+scaffold to generate a range of options and compare conditional forecasts of likelihood of success. But will it beat a human? I’m skeptical that an LLM is ever going to be able to “think” through the layers of contextual knowledge about a particular challenge (say nothing of prioritizing the correct challenge in the first place) to be able to generate winning solutions.
Metric: give forecasters a slate of decision options—some calculated by LLM, some by humans—and see who wins.
Another thought on metrics: calculate a “similarity score” between a decision option and previous at solving similar challenges. Almost like a metric that calculates “neglectedness” and “tractability”?
I imagine that some forms of human invention will be difficult to beat for some time. But I think there’s a lot of more generic strategic work that could be automated. Like what some hedge fund researchers do.
Forecasting systems now don’t even really try to come up with new ideas (they just forecast on existing ones), but they still can be useful.
An anecdote: the US government is trying to convince a foreign government to sign an agreement with the United States but is repeatedly stymied by presidents from both parties for two decades. Let’s assume a forecast at that moment suggests a 10% change the law will be passed within a year. A creative new ambassador designs a creative new strategy that hasn’t been attempted before. Though the agreement would require executive signature, she’s decides instead to meet with every single member of parliament and tell them the United States would owe them if they came out publicly in favor of the deal. Fast forward a year, and the agreement is signed.
Another anecdote: the invention of the Apple computer.
Presumably you could use LLM+scaffold to generate a range of options and compare conditional forecasts of likelihood of success. But will it beat a human? I’m skeptical that an LLM is ever going to be able to “think” through the layers of contextual knowledge about a particular challenge (say nothing of prioritizing the correct challenge in the first place) to be able to generate winning solutions.
Metric: give forecasters a slate of decision options—some calculated by LLM, some by humans—and see who wins.
Another thought on metrics: calculate a “similarity score” between a decision option and previous at solving similar challenges. Almost like a metric that calculates “neglectedness” and “tractability”?
I imagine that some forms of human invention will be difficult to beat for some time. But I think there’s a lot of more generic strategic work that could be automated. Like what some hedge fund researchers do.
Forecasting systems now don’t even really try to come up with new ideas (they just forecast on existing ones), but they still can be useful.