For anyone interested I’ve posted a follow up to this post investigating these ideas and developing a black box procedure for improving our understanding of LLM accuracy: A Black-Box Procedure for LLM Confidence in Critical Applications — LessWrong
For anyone interested I’ve posted a follow up to this post investigating these ideas and developing a black box procedure for improving our understanding of LLM accuracy: A Black-Box Procedure for LLM Confidence in Critical Applications — LessWrong