Please provide feedback on AI-safety grant proposal, thanks!

Hello,

Disclaimer: This is my first post on the EA forum so please be gentle (but honest) with feedback :) In order to provide some accountability to myself that I actually post this, I am setting a 15 minute timer to write this post, and then a 5 minute timer to proof read & post. As such, if this post is a bit rough or lacking details I apologize! At least I posted it! (if you’re reading it) I’m happy to elaborate further in a follow up post, comments, or edits. Thank you!

My actual post:

I recently submitted a grant proposal for a AI-safety related experiment. The grant was not accepted but I was encouraged to keep iterating on the idea so here I am! I’m seeking any ideas for improvements that people can offer. Also if you know of any similar or related work being done I’d love a reference to learn more about it. Thanks in advance!

Here’s the grant proposal in a tweet: Code a toy reinforcement learning experiment in Python where an agent is tasked with a simple task but is incentivized to find a backdoor to that task, allowing it to achieve more “rewards” without accomplishing the main point of the task.

Here’s a more detailed breakdown:

The reinforcement agent will initially be trained to play the classic cart pole game. This is a common toy example many online ML tutorials use as an introductory task to develop a basic RL agent.

A twist will be added in a “difficulty” element will introduced. A certain amount of random jitter will be added to the pole which the agent will need to learn to cope with. If the agent consistently performs well with a certain level of jitter, the jitter will increase.

The “backdoor” will be that if the agent underperforms, the difficulty will actually be reduced, making the game easier. Theoretically the agent could learn to intentionally underperform in order to reduce the difficulty, allowing the game to get to a much easier state of difficulty, at which point the agent could then decide to start performing well and quickly rack up points before the difficulty is increased.

My intentions behind this experiment are mainly that I’d like to put a little bit of code down to explore some of the ideas I see people discussing in the AI-safety domain. Specifically, I think the concerns that AIs could find alternative ways to optimize for a given goal, as it is in this experiment, are a fairly big topic of discussion. I thought it would be cool to pair some of that discussion with an attempt to implement it. I realize this is a total toy example that has no implications for larger scale systems or more complex scenarios.

Anyways, that’s the gist of the idea and my 15 minute timer is almost up. As I said, I’m happy to elaborate further!

Thanks for reading!

Alex