A benefit of some of the agency discourse, as I tried to articulate in this post, is that it can foster a culture of encouragement. I think EA is pretty cool for giving people the mindset to actually go out and try to improve things; tall poppy syndrome and ‘cheems mindsets’ are still very much the norm in many places!
I think a norm of encouragement is distinct from installing an individualistic sense of agency in everyone, though. The former should reduce the chances of Goodharting, since you’ll ideally be working out your goals iteratively with likeminded people (mitigating the risk of single-mindedly pursuing an underspecified goal). It’s great to have conviction — but conviction in everything you do by default could stop you from finding the things you really believe in.
I enjoyed reading these updated thoughts!
A benefit of some of the agency discourse, as I tried to articulate in this post, is that it can foster a culture of encouragement. I think EA is pretty cool for giving people the mindset to actually go out and try to improve things; tall poppy syndrome and ‘cheems mindsets’ are still very much the norm in many places!
I think a norm of encouragement is distinct from installing an individualistic sense of agency in everyone, though. The former should reduce the chances of Goodharting, since you’ll ideally be working out your goals iteratively with likeminded people (mitigating the risk of single-mindedly pursuing an underspecified goal). It’s great to have conviction — but conviction in everything you do by default could stop you from finding the things you really believe in.