I’m now over 20 minutes in and haven’t quite figured out what you’re looking for. Just to dump my thoughts—not necessarily looking for a response:
On the one hand it says “Our goal is to discover creative ways to use AI for Fermi estimation” but on the other hand it says “AI tools to generate said estimates aren’t required, but we expect them to help.”
From the Evaluation Rubric, “model quality” is only 20%, so it seems like the primary goal is neither to create a good “model” (which I understand to mean a particular method for making a Fermi estimate on a particular question) nor to see if AI tools can be used to create such models.
The largest score (40%) is whether the *result* of the model that is created (i.e. the actual estimate that the model spits out with the numbers put into it) is surprising or not, with more surprising being better. But it’s unclear to me if the estimate actually needs to be believed or not for it to be surprising. Extreme numbers could just mean that the output is bad or wrong and not that the output should be evidence of anything.
We’re just looking for a final Fermi model. You can use or not use AI to come up with this.
“Surprise” is important because that’s arguably what makes a model interesting. As in, if you have a big model about the expected impact of AI, and then it tells you the answer you started out expecting, then arguably it’s not an incredibly useful model.
The specific “Surprise” part of the rubric doesn’t require the model to be great, but the other parts of the rubric do weight that. So if you have a model that’s very surprising but otherwise poor, then it might do well on the “Surprise” measure, but won’t on the other measures, so on average will get a mediocre score.
”On the one hand it says “Our goal is to discover creative ways to use AI for Fermi estimation” but on the other hand it says “AI tools to generate said estimates aren’t required, but we expect them to help.”
-> We’re not forcing people to use AI, in part because it would be difficult to verify. But I expect that many people will do so, so I still expect this to be interesting.
I’m now over 20 minutes in and haven’t quite figured out what you’re looking for. Just to dump my thoughts—not necessarily looking for a response:
On the one hand it says “Our goal is to discover creative ways to use AI for Fermi estimation” but on the other hand it says “AI tools to generate said estimates aren’t required, but we expect them to help.”
From the Evaluation Rubric, “model quality” is only 20%, so it seems like the primary goal is neither to create a good “model” (which I understand to mean a particular method for making a Fermi estimate on a particular question) nor to see if AI tools can be used to create such models.
The largest score (40%) is whether the *result* of the model that is created (i.e. the actual estimate that the model spits out with the numbers put into it) is surprising or not, with more surprising being better. But it’s unclear to me if the estimate actually needs to be believed or not for it to be surprising. Extreme numbers could just mean that the output is bad or wrong and not that the output should be evidence of anything.
Thanks for the feedback!
We’re just looking for a final Fermi model. You can use or not use AI to come up with this.
“Surprise” is important because that’s arguably what makes a model interesting. As in, if you have a big model about the expected impact of AI, and then it tells you the answer you started out expecting, then arguably it’s not an incredibly useful model.
The specific “Surprise” part of the rubric doesn’t require the model to be great, but the other parts of the rubric do weight that. So if you have a model that’s very surprising but otherwise poor, then it might do well on the “Surprise” measure, but won’t on the other measures, so on average will get a mediocre score.
Note that there were a few submissions on LessWrong so far, those might make things clearer:
https://www.lesswrong.com/posts/AA8GJ7Qc6ndBtJxv7/usd300-fermi-model-competition#comments
”On the one hand it says “Our goal is to discover creative ways to use AI for Fermi estimation” but on the other hand it says “AI tools to generate said estimates aren’t required, but we expect them to help.”
-> We’re not forcing people to use AI, in part because it would be difficult to verify. But I expect that many people will do so, so I still expect this to be interesting.