I was a participant and largely endorse this comment.
one contributor to a lack of convergence was attrition of effort and incentives. By the time there was superforecaster-expert exchange, we’d been at it for months, and there weren’t requirements for forum activity (unlike the first team stage)
As the origin of that comment i should say other reasons for non-convergence are stronger, but the attrition thing contributed. E.g. biases both for experts to over-rate and supers to under-rate. I wonder also about the structure of engagement with strong team identities fomenting tribal stubbornness for both...
I was also a participant and have my own intuitions from my limited experience. I’ve had lots of great conversations with people where we both learned new things and updated our beliefs… But I don’t know that I’ve ever had one in an asynchronous comment thread format. Especially given the complexity of the topics, I’m just not sure that format was up to the task. During the whole tournament I found myself wanting to create a Discord server and set up calls to dig deeper into assumptions and disagreements. I totally understand the logistical challenges something like that would impose, as well as making it much harder to analyze the communication between participants, but my biggest open question after the tournament was how much better our outputs could have been with a richer collaboration environment.
I asked the original question to try and get at the intuitions of the researchers, having seen all of the data. They outline possible causes and directions for investigation in the paper, which is the right thing to do, but I’m still interested in what they believe happened this time.
I was also a participant. I engaged less than I wanted mostly due to the amount of effort this demanded and losing more and more intrinsic motivation.
Some vague recollections:
Everything took more time than expected and that decreased my motivation a bunch
E.g. I just saw one note that one pandemic-related initial forecast took me ~90 minutes
I think making legible notes requires effort and I invested more time into this than others.
Also reading up on things takes a bunch of time if you’re new to a field (I think GPT-4 would’ve especially helped with making this faster)
Getting feedback from others took a while I think, IIRC most often more than a week? By the point I received feedback I basically forgot everything again and could only go by my own notes
It was effortful to read the notes from most others, I think they often were just written hastily
What could have caused me to engage more with others?
I think the idea of having experts just focus on questions they have some expertise in is a good idea to get me to try to think through other people’s vague and messy notes more, ask more questions, etc.
Probably also having smaller teams (like ~3-5 people) would’ve made the tournament feel more engaging, I basically didn’t develop anything close to a connection with anyone in my team because they were just a dozen of mostly anonymous usernames.
Why do you think participants largely didn’t change their minds?
I was a participant and largely endorse this comment.
As the origin of that comment i should say other reasons for non-convergence are stronger, but the attrition thing contributed. E.g. biases both for experts to over-rate and supers to under-rate. I wonder also about the structure of engagement with strong team identities fomenting tribal stubbornness for both...
I was also a participant and have my own intuitions from my limited experience. I’ve had lots of great conversations with people where we both learned new things and updated our beliefs… But I don’t know that I’ve ever had one in an asynchronous comment thread format. Especially given the complexity of the topics, I’m just not sure that format was up to the task. During the whole tournament I found myself wanting to create a Discord server and set up calls to dig deeper into assumptions and disagreements. I totally understand the logistical challenges something like that would impose, as well as making it much harder to analyze the communication between participants, but my biggest open question after the tournament was how much better our outputs could have been with a richer collaboration environment.
I asked the original question to try and get at the intuitions of the researchers, having seen all of the data. They outline possible causes and directions for investigation in the paper, which is the right thing to do, but I’m still interested in what they believe happened this time.
I was also a participant. I engaged less than I wanted mostly due to the amount of effort this demanded and losing more and more intrinsic motivation.
Some vague recollections:
Everything took more time than expected and that decreased my motivation a bunch
E.g. I just saw one note that one pandemic-related initial forecast took me ~90 minutes
I think making legible notes requires effort and I invested more time into this than others.
Also reading up on things takes a bunch of time if you’re new to a field (I think GPT-4 would’ve especially helped with making this faster)
Getting feedback from others took a while I think, IIRC most often more than a week? By the point I received feedback I basically forgot everything again and could only go by my own notes
It was effortful to read the notes from most others, I think they often were just written hastily
What could have caused me to engage more with others?
I think the idea of having experts just focus on questions they have some expertise in is a good idea to get me to try to think through other people’s vague and messy notes more, ask more questions, etc.
Probably also having smaller teams (like ~3-5 people) would’ve made the tournament feel more engaging, I basically didn’t develop anything close to a connection with anyone in my team because they were just a dozen of mostly anonymous usernames.