Link post
I think this discussion was had on LW a few years ago (and probably sporadically since then). Just quickly some parameters off the top of my head. Pro:
Improves Forecasting
Necessary infrastructure for a variety of verification tech that will be needed for international treaties
Know when to sound alarm bells
Helps us know what type of defensive technologies we need to build
Cons:
Increases the speed of ai development
Unclear if US and China are even interested in coordinating
Unclear if a number from a eval will be enough to cause significant political pressure
Unsure
Depending on trajectory of benchmarking, builds/kills hype and reduces/increases investment.
[Question] Is benchmarking AI capabilities positive EV?
Link post
I think this discussion was had on LW a few years ago (and probably sporadically since then). Just quickly some parameters off the top of my head.
Pro:
Improves Forecasting
Necessary infrastructure for a variety of verification tech that will be needed for international treaties
Know when to sound alarm bells
Helps us know what type of defensive technologies we need to build
Cons:
Increases the speed of ai development
Unclear if US and China are even interested in coordinating
Unclear if a number from a eval will be enough to cause significant political pressure
Unsure
Depending on trajectory of benchmarking, builds/kills hype and reduces/increases investment.