Hello! I work on AI grantmaking at Open Philanthropy.
All posts in a personal capacity and do not reflect the views of my employer, unless otherwise stated.
cb
Karma: 103
I’d also be excited about projects aiming to do this.
One advantage that quantifying post-training variables on frontier models has over this idea is that you also get a better sense of what the upper bound of performance on some eval looks like, as well as some information about the returns from investing in post-training enhancements. I think if this were done responsibly on some well-chosen evals, it’d be helpful information to have. (Though my colleagues may disagree.)
If people outside of frontier labs were working on this, I’d be surprised if it significantly accelerated capabilities, though I can imagine it still making sense to keep the methodology private.
Evals projects I’d like to see, and a call to apply to OP’s evals RFP
Tractability + something-like-epistemic-humility feel like cruxes for me, I’m surprised they haven’t been discussed much; preventing extinction is good by most lights, specific interventions to improve the future are much less clearly good, and I feel much more confused about what would have lasting effects.
I wrote about mistakes I made as a uni group organiser here, inspired by this list!
Mistakes I made running an AI safety student group
(even larger disclaimer than usual: i don’t have much experience applying to EA orgs, i’m also not trying to give career advice and wouldn’t recommend taking career advice from me, ymmv)
Thanks for posting! I’m broadly sympathetic to this line of reasoning. One thing I wanted to note was that hiring processes seem pretty noisy, and lots of people seem pretty bad at estimating how good they are at things, so I think in practice there might not be that much difference between trying to get yourself hired vs. trying to get the best candidate hired. I think a reasonable heuristic is “try to do well at all the interviews/work tests, as you would for a normal job, but don’t rule yourself out in advance, and be very honest and transparent if you’re asked specific questions”.
Hi Søren,
Thanks for commenting. Some quick responses:
> The safety frameworks presented by the frontier labs are “safety-washing”, more appropriately considered roadmaps towards an unsurvivable future
I don’t see the labs as the main audience for evaluation results, and I don’t think voluntary safety frameworks should be how deployment and safeguard decisions are made in the long-term, so I don’t think the quality of lab safety frameworks is that relevant to this RFP.
> I’d like sources for your claim, please.
Sure, see e.g. the sources linked to in our RFP for this claim: What Are the Real Questions in AI? and What the AI debate is really about.
I’m surprised you think the disagreements are “performative” – in my experience, many sceptics of GCRs from AI really do sincerely hold their beliefs.
> No decision-relevant conclusions can be drawn from evaluations in the style of Cybench and Re-Bench.
I think Cybench and RE-Bench are useful, if imperfect, proxies for frontier model capabilities at cyberoffense and ML engineering respectively, and those capabilities are central to threats from cyberattacks and AI R&D. My claim isn’t that running these evals will tell you exactly what to do: it’s that these evaluations are being used as inputs into RSPs and governance proposals more broadly, and provide some evidence on the likelihood of GCRs from AI, but will need to be harder and more robust to be relied upon.
Improving capability evaluations for AI governance: Open Philanthropy’s new request for proposals
I found the tone of this piece pretty sneer-y and offputting. There’s a difference between “here’s an extreme example showing why this argument doesn’t work” and “look how stupid these people are” — I think this writing is veering towards the latter, and I’d prefer to see less of that on the Forum.
Seems false, unless he’s using “general agreement” and “foreseeable” in some very narrow sense?