Hi Connacher! Thanks for the responses, makes sense.
On your question, one example I often miss from expert surveys is something like this open-ended question: “Do you have any other considerations that would help with understanding this topic?”
I generally agree that quantitative questions are intimately connected with identifying cruxes. Being quantitative about concrete events is a neat way of forcing the experts get more concrete and incentivize them to not get lost in a vague story, etc. But I suspect that often the individual insights from the experts might seem like cruxes to them, as they’re not used to think like that. So I think giving experts some prompts to just pour out their thoughts is often neglected. Furthermore, sometimes quantitative questions don’t fully capture all important angles of an issue and so it’s useful to give responders many chances to add additional comments.
Do you view this as separate from the rationale data we also collect? One low-burden way to do this is to just include something like your text in the rationale prompt.
It’s great that you already have a rationale prompt for each question. I would probably recommend having one prompt like this at the end, with “(Optional)” in front so experts can share all further thoughts they think might be useful.
Hi Connacher! Thanks for the responses, makes sense.
On your question, one example I often miss from expert surveys is something like this open-ended question: “Do you have any other considerations that would help with understanding this topic?”
I generally agree that quantitative questions are intimately connected with identifying cruxes. Being quantitative about concrete events is a neat way of forcing the experts get more concrete and incentivize them to not get lost in a vague story, etc. But I suspect that often the individual insights from the experts might seem like cruxes to them, as they’re not used to think like that. So I think giving experts some prompts to just pour out their thoughts is often neglected. Furthermore, sometimes quantitative questions don’t fully capture all important angles of an issue and so it’s useful to give responders many chances to add additional comments.
Do you view this as separate from the rationale data we also collect? One low-burden way to do this is to just include something like your text in the rationale prompt.
It’s great that you already have a rationale prompt for each question. I would probably recommend having one prompt like this at the end, with “(Optional)” in front so experts can share all further thoughts they think might be useful.