Thank you very much for your response! I agree that there is a risk of capture and drift with any institution. I also agree with your point that the difference with institutions specifically mandated to deal with extreme risks is that those risks and metrics can be more precisely defined. As we say, I don’t think those lines of defence proposals are complete. It may also be possible, for example, to have external prediction markets or other external validation such as paid forecasters or ‘red teams’ to monitor their performance. I suspect we could perhaps define large buckets of risk with enough accuracy to allow that sort of external monitoring.
It’s much harder to envisage how that could be done for a much more general goal, and indeed the Bill does not attempt to do that in advance – but sadly I don’t have much confidence that its suggested process for selecting such goals would make it easy to create external institutions to accurately monitor performance.
Makes sense, precisely defining the risks does seem like it would help a lot.
How would external validators help (if relevant officials, voters, and legislators aren’t very concerned with these issues)? Is the idea that external validators would be useful after raising broader concerns?
It’s an excellent question. Yes, I think we agree. In general I think sunlight can be a great disinfectant. If external validators’ views were published then it would be easier to create pressure to improve the organisation that is doing bad work. We could also look at creating incentives for the organisation and individuals within the organisation to achieve better results as measured against the views of external validators.
Thank you very much for your response! I agree that there is a risk of capture and drift with any institution. I also agree with your point that the difference with institutions specifically mandated to deal with extreme risks is that those risks and metrics can be more precisely defined. As we say, I don’t think those lines of defence proposals are complete. It may also be possible, for example, to have external prediction markets or other external validation such as paid forecasters or ‘red teams’ to monitor their performance. I suspect we could perhaps define large buckets of risk with enough accuracy to allow that sort of external monitoring.
It’s much harder to envisage how that could be done for a much more general goal, and indeed the Bill does not attempt to do that in advance – but sadly I don’t have much confidence that its suggested process for selecting such goals would make it easy to create external institutions to accurately monitor performance.
Makes sense, precisely defining the risks does seem like it would help a lot.
How would external validators help (if relevant officials, voters, and legislators aren’t very concerned with these issues)? Is the idea that external validators would be useful after raising broader concerns?
It’s an excellent question. Yes, I think we agree. In general I think sunlight can be a great disinfectant. If external validators’ views were published then it would be easier to create pressure to improve the organisation that is doing bad work. We could also look at creating incentives for the organisation and individuals within the organisation to achieve better results as measured against the views of external validators.