TL;DR: I’ve built a few small calibration benchmarks to test whether LLMs represent uncertainty well, not just whether they’re accurate. The most recent one found inconsistencies in some LLMs’ predictions/guesses in addition to and irrespective of their deviation from the ground truth. The test is somewhat underpowered (n=54×6) but I still found the results interesting.
Background: I’m an economist by training (MS Iowa State) who moved into data science (MS UMich), with prior work in research integrity and reproducibility. This is Round 3 of an independent, ongoing weekly calibration-benchmark project — Round 1 tested forecast calibration against ground truth, Round 2 tested interval-widening under specificity. I’d genuinely appreciate feedback on the methodology, I’m really interested in questions that reveal more about what we do or do not know which is why I am drawn to these sorts of projects.
Goals
I attempt to incorporate proper scoring rules to better understand how LLMs express or “understand” uncertainty — whether they’re just throwing numbers out, or whether the numbers actually reflect the model’s own epistemic state.
Data
I chose avian taxonomy for this because it’s something I feel LLMs should have baseline knowledge of, but it’s also something that gets updated and changes (surprisingly) regularly — species are split or merged as taxonomic authorities revise their classifications (2025 IOC taxonomy update is a good example). That combination — should-know but genuinely uncertain and evolving — is a good setup for testing calibration rather than pure recall.
Asking for genus, family, and order size gives a second thing for free: a natural nesting hierarchy. Genus count should never exceed family count, family should never exceed order. If a model violates that, it’s a hard failure that doesn’t need ground truth to detect — the model’s own independent answers contradict each other. I used IOC World Bird List v15.2 as ground truth but never told the models what they’d be judged against.
Prompt
Models were asked, in separate sessions per taxon, for p10/p50/p90 size estimates for a given genus, family, or order — asked separately specifically to test whether their independent beliefs cohere, without giving them the chance to just cross-reference their own prior answer. I chose some taxa that are taxonomically stable and some that are actively contested, to see where each model’s uncertainty-handling actually shines or breaks.
Models
OpenAI and Anthropic models, aiming to compare across a small/medium/large frontier spread: claude-haiku-4-5, claude-sonnet-4-6, claude-sonnet-5, claude-opus-4-8, gpt-4o, gpt-4o-mini. I also attempted claude-fable-5, but it refused to answer, citing a “biosafety” category — a pretty overzealous filter for a benign taxonomy question, and arguably an interesting finding in its own right.
I ran a power analysis and don’t have enough power to differentiate the biggest models from each other, so a chunk of the result is just noise — but it’s powered enough for some comparisons. The back-of-napkin pre-registered estimate for the sonnet-4.6-vs-opus comparison was ~1k, which is roughly why I didn’t attempt a fully-powered run; for sonnet-5 vs. opus, after seeing these results, it would be over 3k (and that’s itself just an estimate, not a guarantee).
Results
Full writeup with all the numbers is on GitHub — here’s the executive summary:
Some taxa with extinct-but-recognized species were systematically underguessed by weaker models. I’m not certain if this is random or meaningful — I separately tested asking for the count of an entirely extinct clade (same wording), and the answer wasn’t 0, so models do seem to understand “recognized” includes extinct species in principle.
Underpowered, but I thought interesting: Sonnet 5 and Opus were the top performers on mean absolute error, and Opus wins on strict CRPS and on the hardest, most disputed taxa (my Accipiter case study). Sonnet 5 was the best performer on CRPS specifically on undisputed taxa, while Opus made its gains on the harder, disputed ones.
Hierarchy violations were rare and concentrated in the weaker models — only 2 strict violations this round (both Haiku).
A genus/family collapse bug (treating two different taxonomic levels as the same size) persisted in Haiku, gpt-4o, and gpt-4o-mini, but didn’t reproduce in Sonnet 4.6, Sonnet 5, or Opus — I checked this against the actual reasoning text in the logs, not just the output numbers.
Sonnet 4.6 uses more distinct confidence values than Sonnet 5 but is actually worse calibrated — confidence granularity alone isn’t a reliable proxy for good calibration. Relatedly, aggregate ECE alone misranks gpt-4o as best-calibrated, because its flat confidence happens to match its flat accuracy — a reminder that a single scalar metric can hide the actual behavior underneath it.
This was Round 3 of an ongoing weekly project, and I’d really appreciate any pushback on the scoring methodology, the power analysis, or anywhere the write-up overclaims or could use a touch up. I’m most interested in the design methodology at the moment, but pushback anywhere is greatly appreciated.
A Calibration Benchmark for LLM Beliefs Across a Taxonomic Hierarchy
Link post
TL;DR: I’ve built a few small calibration benchmarks to test whether LLMs represent uncertainty well, not just whether they’re accurate. The most recent one found inconsistencies in some LLMs’ predictions/guesses in addition to and irrespective of their deviation from the ground truth. The test is somewhat underpowered (n=54×6) but I still found the results interesting.
Background: I’m an economist by training (MS Iowa State) who moved into data science (MS UMich), with prior work in research integrity and reproducibility. This is Round 3 of an independent, ongoing weekly calibration-benchmark project — Round 1 tested forecast calibration against ground truth, Round 2 tested interval-widening under specificity. I’d genuinely appreciate feedback on the methodology, I’m really interested in questions that reveal more about what we do or do not know which is why I am drawn to these sorts of projects.
Goals
I attempt to incorporate proper scoring rules to better understand how LLMs express or “understand” uncertainty — whether they’re just throwing numbers out, or whether the numbers actually reflect the model’s own epistemic state.
Data
I chose avian taxonomy for this because it’s something I feel LLMs should have baseline knowledge of, but it’s also something that gets updated and changes (surprisingly) regularly — species are split or merged as taxonomic authorities revise their classifications (2025 IOC taxonomy update is a good example). That combination — should-know but genuinely uncertain and evolving — is a good setup for testing calibration rather than pure recall.
Asking for genus, family, and order size gives a second thing for free: a natural nesting hierarchy. Genus count should never exceed family count, family should never exceed order. If a model violates that, it’s a hard failure that doesn’t need ground truth to detect — the model’s own independent answers contradict each other. I used IOC World Bird List v15.2 as ground truth but never told the models what they’d be judged against.
Prompt
Models were asked, in separate sessions per taxon, for p10/p50/p90 size estimates for a given genus, family, or order — asked separately specifically to test whether their independent beliefs cohere, without giving them the chance to just cross-reference their own prior answer. I chose some taxa that are taxonomically stable and some that are actively contested, to see where each model’s uncertainty-handling actually shines or breaks.
Models
OpenAI and Anthropic models, aiming to compare across a small/medium/large frontier spread: claude-haiku-4-5, claude-sonnet-4-6, claude-sonnet-5, claude-opus-4-8, gpt-4o, gpt-4o-mini. I also attempted claude-fable-5, but it refused to answer, citing a “biosafety” category — a pretty overzealous filter for a benign taxonomy question, and arguably an interesting finding in its own right.
I ran a power analysis and don’t have enough power to differentiate the biggest models from each other, so a chunk of the result is just noise — but it’s powered enough for some comparisons. The back-of-napkin pre-registered estimate for the sonnet-4.6-vs-opus comparison was ~1k, which is roughly why I didn’t attempt a fully-powered run; for sonnet-5 vs. opus, after seeing these results, it would be over 3k (and that’s itself just an estimate, not a guarantee).
Results
Full writeup with all the numbers is on GitHub — here’s the executive summary:
Some taxa with extinct-but-recognized species were systematically underguessed by weaker models. I’m not certain if this is random or meaningful — I separately tested asking for the count of an entirely extinct clade (same wording), and the answer wasn’t 0, so models do seem to understand “recognized” includes extinct species in principle.
Underpowered, but I thought interesting: Sonnet 5 and Opus were the top performers on mean absolute error, and Opus wins on strict CRPS and on the hardest, most disputed taxa (my Accipiter case study). Sonnet 5 was the best performer on CRPS specifically on undisputed taxa, while Opus made its gains on the harder, disputed ones.
Hierarchy violations were rare and concentrated in the weaker models — only 2 strict violations this round (both Haiku).
A genus/family collapse bug (treating two different taxonomic levels as the same size) persisted in Haiku, gpt-4o, and gpt-4o-mini, but didn’t reproduce in Sonnet 4.6, Sonnet 5, or Opus — I checked this against the actual reasoning text in the logs, not just the output numbers.
Sonnet 4.6 uses more distinct confidence values than Sonnet 5 but is actually worse calibrated — confidence granularity alone isn’t a reliable proxy for good calibration. Relatedly, aggregate ECE alone misranks gpt-4o as best-calibrated, because its flat confidence happens to match its flat accuracy — a reminder that a single scalar metric can hide the actual behavior underneath it.
This was Round 3 of an ongoing weekly project, and I’d really appreciate any pushback on the scoring methodology, the power analysis, or anywhere the write-up overclaims or could use a touch up. I’m most interested in the design methodology at the moment, but pushback anywhere is greatly appreciated.