People in EA end up optimizing for EA credentials so they can virtue signal to grantmakers, but grantmakers would probably like people to scope out non-EA opportunities because that allows us to introduce unknown people to the concerns we have
Heramb Podar
Karma: 187
166 States Vote to Adopt Lethal Autonomous Weapons Resolution at the UNGA
CAIDP Statement on Lethal Autonomous Weapons Systems
At some point, one has to ask “Am I in the cause area because I am an EA or am in EA because I am in the cause area?”
Reminds me of keeping your identity small by Paul Graham
At this point, we need an 80k page on “What to do after leaving Open AI”
Don’t start another AI safety lab
Good question, this can look very different for different people. My advice(and this might be not applicable to all) is do things that interest/excite you. The AI safety and governance field is very broad (especially with AI popping up in every field), so there’s tons of subfields to explore.
There might be issues that are more local/regional that are important to you, maybe there are things that worry you or your demographic and so on.
For policy recommendations, put forth things that actually build on or move the status quo.
For example, recommending a “National Youth Council” without a mandate can be a uniquely bad idea- instead of ignoring your (usually inactive) youth org, now, policymakers will ignore the Council of (usually inactive)youth orgs all while you(the actually proactive person), walk away with the false notion of a job well done.
Nice breakdown, I can see myself visiting this sometime in the future(hopefully not too soon)
My take on this is way too many people optimize for impact in the early rungs, which is bad. I think way too much of the messaging is impact-centric, which leads to people optimizing for the wrong end-goals when, in reality, hardly anyone will read/care about your shiny new fellowship paper.
For the past ~3 quarters, I have been optimizing for fun, and this gives me the right amount of kick to keep me going.
Additionally, for fields like policy, the lack of objective results is made up for by the higher requirements of social clout, which involves building a network that probably takes a lot of time(this is one of those pesky chicken and egg problems).
Technical Risks of (Lethal) Autonomous Weapons Systems
The problem with AI safety policy is that if we don’t specify and attempt to answer the technical concerns then someone else will and safety wash the concerns away.
CSOs need to understand what they themselves mean when they say “explainable” and “algorithmic transparency.”
(Comments from skimming the piece and general thoughts from the current state of AI legislation)
->If there is agreement, there should be a pause, building international trust for a pause is crucial- Current verification mechanisms are rather weak.
-> Current policy discourse rarely includes X-risks (coming from legislative drafts, frameworks, and National strategies countries are releasing). A very small minority of people in the broader CSO space seem concerned about X-risks. The recent UN AI Advisory Body report on AI also doesn’t really hone in on x-risks.
-> There might be strange observer effects wherein proposing the idea of a pause makes that party look weak and makes the tech seem even more important.
-> Personally, I am not sure if there is a well-defined end-point to the alignment problem. Any argument for a pause should come with what the “resume” conditions are going to be. In the current paradigm, there seems to be no good definition of acceptable/aligned behavior accepted across stakeholders.
Now,
→ Pausing is a really bad look for people in office. Without much precedent, they would be treading straight into the path of innovation while also angering the tech lobby. They need a good reason to show their constituents why they want to take a really extreme step, such as pausing progress/innovation in a hot area(this is why trigger events are a thing). This sets bad precedents and spooks other sectors as well(especially in the US where this is going to be painted as a Big Government move). Remember, policymakers have a much broader portfolio than just AI, and they do not necessarily think this is the most pressing problem.
→ Pausing hurts countries that stand to gain(or think that they do) the most from it (this tends to be Global South, AI For Good/SDGs folk).
-> Any arguments for pause will also have to consider the opportunity cost of delaying more capable AI.
-> Personally, I don’t update much on the US public being surveyed because of potential framing biases, little downside cost of agreeing, etc. I also don’t think the broader public understands the alignment problem well.
The real danger isn’t just from AI getting better- it’s from it getting good enough that humans start over-relying on it and offloading tasks to it.
Remember that Petrov had automatic detection systems, too; he just independently came to the conclusion not to fire nukes back.
It’s important to think about the policy space from a meta-level incentives/factors that might get in the way of having an impact, such as making AI safer.
One I heard today was that policy people thrive in moments of regulatory uncertainty, while this is bad for companies.
It’s interesting how we also have scope neglect of key historical events:
Death Toll: Siege of Athens in 86 BC ~ People killed in Hiroshima <<< Battle of Stalingrad
I feel like being exposed early on to longer form GovAI-type reports has made me set the bar high for writing my thoughts out in short form, which really sucks in terms of an output standpoint.
With open-source models being released and on ramps to downstream innovation lowering, the safety challenges may not be a single threshold but rather an ongoing, iterative cat-and-mouse game.
Just underscores the importance of people in the policy/safety field thinking far ahead
Updated away from this generally- there is a balance.
Good example for why I updated away is 28:27 from the video at:
Yeah, I wish someone had told me this earlier—it would have led me to apply a lot earlier and not “saving my chance.” There’s a couple of layers to this thought process in my opinion:
Talented people often feel like they are not the ideal candidates/ they don’t have the right qualifications.
The kind of people EA attracts generally have a track record of checking every box, so they carry this “trait” over into the EA space
In general, there’s a lot of uncertainty in fields like AI governance even among experts from what I can glean
Cultures particularly in the global south punish people for being uncertain, let alone quantifying uncertanity
Agreeing on building safe, trustworthy, and human-centric AI is akin to making an open call for a DIY definitions for different regulatory environments.