Thanks for writing this up, I found it very stimulating.
(One way that models can be useful without requiring any trust is giving you clues about where to look in building up your own models.)
Probably an edge case, but I wonder if an adversary could purposefully divert your attention away from important considerations. Thinking about it, I actually remember doing something like this in an adversarial board game, where I used „helpful“ clues to direct the attention of somebody who I‘m plotting against.
Another thought regarding incentives for good judgements came up: People seem to automatically develop good judgement in areas where they get feedback and have „skin in the game“, for example when deciding how to get a tasty meal or with whom to talk about a personal problem. So I wondered how much attention we should put on the surrounding incentives to develop good judgement. Will we get feedback, e.g. from peers that look at our reasoning, or from failed predictions because we develop the habit to make forecasts? Are we betting on our believes and making our track record public? There is probably much more to say how we could better incentivize good judgement.
Thanks for writing this up, I found it very stimulating.
Probably an edge case, but I wonder if an adversary could purposefully divert your attention away from important considerations. Thinking about it, I actually remember doing something like this in an adversarial board game, where I used „helpful“ clues to direct the attention of somebody who I‘m plotting against.
Another thought regarding incentives for good judgements came up: People seem to automatically develop good judgement in areas where they get feedback and have „skin in the game“, for example when deciding how to get a tasty meal or with whom to talk about a personal problem. So I wondered how much attention we should put on the surrounding incentives to develop good judgement. Will we get feedback, e.g. from peers that look at our reasoning, or from failed predictions because we develop the habit to make forecasts? Are we betting on our believes and making our track record public? There is probably much more to say how we could better incentivize good judgement.