I find a lot of the challenge of making Fermi estimates is in creating early models / coming up with various ways to parameterize things. LLMs have been very good at this, in my opinion.
I wrote more in the “How good is it?” section of the Squiggle AI blog post.
We don’t yet have quantitative measures of output quality, partly due to the challenge of establishing ground-truth for cost-effectiveness estimates. However, we do have a variety of some qualitative results.
Early Use
As the primary user, I (Ozzie) have seen dramatic improvements in efficiency—model creation time has dropped from 2-3 hours to 10-30 minutes. For quick gut-checks, I often find the raw AI outputs informative enough to use without editing.
Our three Squiggle workshops (around 20 total attendees) have shown encouraging results, with participants strongly preferring Squiggle AI over manual code writing. Early adoption has been modest but promising—in recent months, 30 users outside our team have run 168 workflows total.
Accuracy Considerations
As with most LLM systems, Squiggle AI tends toward overconfidence and may miss crucial factors. We recommend treating its outputs as starting points rather than definitive analyses. The tool works best for quick sanity check and initial model drafts.
Current Limitations
Several technical constraints affect usage:
Code length soft-caps at 200 lines
Frequent workflow stalls from rate limits or API balance issues
Auto-generated documentation is decent but has gaps, particularly in outputting plots and diagrams
While slower and more expensive than single LLM queries, Squiggle AI provides more comprehensive and structured output, making it valuable for users who want detailed, adjustable, and documentable reasoning behind their estimates.
Do you have examples of LLMs improving Fermi estimates? I’ve found it hard to get any kind of credences at all out of them, let alone convincing ones.
I find a lot of the challenge of making Fermi estimates is in creating early models / coming up with various ways to parameterize things. LLMs have been very good at this, in my opinion.
I wrote more in the “How good is it?” section of the Squiggle AI blog post.
https://forum.effectivealtruism.org/posts/jJ4pn3qvBopkEvGXb/introducing-squiggle-ai#How_Good_Is_It_