I broadly agree with this, but at least with AI safety there’s a Goodharting issue: we don’t want AIS researchers optimising for legibly impressive ideas/results/writeups.
I assume there’s a similar-in-principle issue for most cause areas, but it does seem markedly worse for AIS. (given lack of meaningful feedback on the most important issues)
There’s a significant downside even in having some proportion of EA AIS researchers focus on more legible results: it gives a warped impression of useful AIS research to outsiders. This happens by default, since there are many incentives to pick a legibly impressive line of research, and there’ll be more engagement with more readable content.
None of this is to say that I know e.g. MIRI-style research to be the right approach. However, I do think we need to be careful not to optimise for the appearance of strong object level work.
I broadly agree with this, but at least with AI safety there’s a Goodharting issue: we don’t want AIS researchers optimising for legibly impressive ideas/results/writeups.
I assume there’s a similar-in-principle issue for most cause areas, but it does seem markedly worse for AIS. (given lack of meaningful feedback on the most important issues)
There’s a significant downside even in having some proportion of EA AIS researchers focus on more legible results: it gives a warped impression of useful AIS research to outsiders. This happens by default, since there are many incentives to pick a legibly impressive line of research, and there’ll be more engagement with more readable content.
None of this is to say that I know e.g. MIRI-style research to be the right approach.
However, I do think we need to be careful not to optimise for the appearance of strong object level work.