MichaelA

Karma: 12,401

I’m a Senior Research Manager in Rethink Priorities’ AI Governance and Strategy team. I’m also an advisor to organizations such as Training for Good and an affiliate of the Centre for the Governance of AI. My prior work includes positions at the Center on Long-Term Risk and the Future of Humanity Institute.

Opinions expressed are my own. You can give me anonymous feedback here.

See here for Rethink Priorities’ job openings or expression of interest forms, here for a list of EA-aligned funding sources you could apply to, and here for my top recommended resources for people interested in EA/longtermist research careers.

MichaelA 23 Aug 2023 14:49 UTC
2 points
0 ∶ 0
in reply to: constructive’s comment on: Collection of work on ‘Should you should focus on the EU if you’re interested in AI governance for longtermist/x-risk reasons?’
Oh wow, thanks for flagging that, fixed! Amazing that a whole extra word in the title itself survived a whole year, and survived me copy-pasting the title in various other places too 😬

MichaelA 30 Jul 2023 7:38 UTC
2 points
0 ∶ 0
on: US Policy Career Resources
Thanks for making this!
What do the asterisks before a given resource mean? (E.g. before “Act of Congress: How America’s Essential Institution Works, and How It Doesn’t”.) Maybe they mean you’re especially strongly recommending that?

MichaelA 29 Jul 2023 19:09 UTC
2 points
0 ∶ 0
on: List of EA funding opportunities
AI Safety Support have a list of funding opportunities. I’m pretty sure all of them are already in this post + comments section, but it’s plausible that’ll change in future.

MichaelA 24 Jul 2023 7:03 UTC
3 points
0 ∶ 0
in reply to: zeshen’s comment on: Survey on intermediate goals in AI governance
Yeah, the “About sharing information from this report” section attempts to explain this. Also, for what it’s worth, I approved all access requests, generally within 24 hours.
That said, FYI I’ve now switched to the folder being viewable by anyone with the link, rather than requiring requesting access, though we still have the policies in “About sharing information from this report”. (This switch was partly because my sense of the risks vs benefits has changed, and partly because we apparently hit the max number of people who can be individually shared on a folder.)

MichaelA 21 Jul 2023 10:16 UTC
8 points
0 ∶ 0
on: List of EA funding opportunities
AI Safety Impact Markets
Description provided to me by one of the organizers:
This is a public platform for AI safety projects where funders can find you. You shop around for donations from donors that already have a high donor score on the platform, and their donations will signal-boost your project so that more donors and funders will see it.

MichaelA 13 Jul 2023 18:04 UTC
2 points
0 ∶ 0
on: List of EA funding opportunities
See also An Overview of the AI Safety Funding Situation for indications of some additional non-EA funding opportunities relevant to AI safety (e.g. for people doing PhDs or further academic work).

MichaelA 13 Jul 2023 18:03 UTC
7 points
0 ∶ 0
on: An Overview of the AI Safety Funding Situation
FYI, if any readers want just a list of funding opportunities and to see some that aren’t in here, they could check out List of EA funding opportunities.
(But note that that includes some things not relevant to AI safety, and excludes some funding sources from outside the EA community.)

MichaelA 13 Jul 2023 18:02 UTC
2 points
0 ∶ 0
on: List of EA funding opportunities
$20 Million in NSF Grants for Safety Research
After a year of negotiation, the NSF has announced a $20 million request for proposals for empirical AI safety research.
Here is the detailed program description.
The request for proposals is broad, as is common for NSF RfPs. Many safety avenues, such as transparency and anomaly detection, are in scope

MichaelA 13 Jul 2023 18:00 UTC
2 points
0 ∶ 0
on: List of EA funding opportunities
Announcing Manifund Regrants
Manifund is launching a new regranting program! We will allocate ~$2 million over the next six months based on the recommendations of our regrantors. Grantees can apply for funding through our site; we’re also looking for additional regrantors and donors to join.

MichaelA 13 Jul 2023 8:18 UTC
11 points
0 ∶ 0
in reply to: Linch’s comment on: Linch’s Shortform
Yeah, this seems to me like an important question. I see it as one subquestion of the broader, seemingly important, and seemingly neglected questions “What fraction of importance-adjusted AI safety and governance work will be done or heavily boosted by AIs? What’s needed to enable that? What are the implications of that?”
I previously had a discussion focused on another subquestion of that, which is what the implications are for government funding programs in particular. I wrote notes from that conversation and will copy them below. (Some of this is also relevant to other questions in this vicinity.)
“Key takeaways
- Maybe in future most technical AI safety work will be done by AIs.
- Maybe that has important implications for whether & how to get government funding for technical AI safety work?
  - E.g., be less enthusiastic about getting government funding for more human AI safety researchers?
  - E.g., be more enthusiastic about laying the groundwork for gov funding for AI assistance for top AI safety researchers later?
    Such as by more strongly prioritizing having well-scoped research agendas, or ensuring top AI safety researchers (or their orgs) have enough credibility signals to potentially attract major government funding?
  - This is a subquestion of the broader question “What should we do to prep for a world where most technical AI safety work can be done by AIs?”, which also seems neglected as far as I can tell.
- Seems worth someone spending 1-20 hours doing distillation/research/writing on that topic, then sharing that with relevant people.
Additional object-level notes
- See [v. A] Introduction & summary – Survey on intermediate goals in AI governance for an indication of how excited AI risk folks are about “Increase US and/or UK government spending on AI reliability, robustness, verification, reward learning, interpretability, and explainability”.
  - Details of people’s views can be found in [v. B] Ratings & comments on goals related to government spending – Survey on intermediate goals in AI governance
  - (Feel free to request access, though it may not be granted.)
- But there may in future be a huge army of AI safety researchers in the form of AIs, or AI tools/systems that boost AI safety researchers in other ways. What does that imply, esp. for gov funding programs?
  - Reduced importance of funding for AI safety work, since it’ll be less bottlenecked by labor (which is costly) and more by a handful of good scalable ideas?
  - Funding for AI safety work is mostly important for getting top AI safety researchers to have huge compute budgets to run (and train?) all those AI assistance, rather than funding people themselves or other things?
    Perhaps this even increases the importance of funding, since we thought it’d be hard to scale the relevant labor via people but it may be easier to scale via lots of compute and hence AI assistance?
  - Increased importance of particular forms of “well-scoped” research agendas/questions? Or more specifically, focusing now on whatever work it’s hardest to hand off to AIs but that best sets things up for using AIs?
  - Make the best AI safety researchers, research agendas, and orgs more credible/legible to gov people so that they can absorb lots of funding to support AI assistants?
    What does that require?
    Might mean putting some of the best AI safety researchers in new or existing institutions that look credible? E.g. into academic labs, or merging a few safety projects into one org that we ensure has a great brand?
  - Start pushing the idea (in EA, to gov people, etc.) that gov should now/soon provide increasingly much funding for AI safety via compute support for relevant people?
  - Start pushing the idea that gov should be very choosy about who to support but then support them a lot? Like support just a few of the best AI safety researchers/orgs but providing them with a huge compute budget?
- That’s unusual and seems hard to make happen. Maybe that makes it worth actively laying groundwork for this?
Research proposal
- I think this seems worth a brief investigation of, then explicitly deciding whether or not to spend more time.
- Ideally this’d be done by someone with decent AI technical knowledge and/or gov funding program knowledge.
- If someone isn’t the ideal fit for working on this but has capacity and interest, they could:
  - spend 1-10 hours
  - aim to point out some somewhat-obvious-once-stated hypotheses, without properly vetting them or fleshing them out
  - Lean somewhat on conversations with relevant people or on sharing a rough doc with relevant people to elicit their thoughts
- Maybe the goals of an initial stab at this would be:
  - Increase the chance that someone who does have strong technical and/or gov knowledge does further thinking on this
  - Increase the chance that relevant technical AI safety people, leaders of technical AI safety orgs, and/or people in government bear this in mind and adjust their behavior in relevant ways”

RP’s AI Governance & Strategy team—June 2023 interim overview

MichaelA22 Jun 2023 13:45 UTC

67 points

1 comment7 min readEA link

MichaelA 11 Jun 2023 6:08 UTC
2 points
0 ∶ 0
in reply to: EA Opportunity Board’s comment on: Rethink Priorities is hiring a Compute Governance Researcher or Research Assistant
Thanks!

MichaelA 8 Jun 2023 14:36 UTC
2 points
0 ∶ 0
in reply to: MichaelA’s comment on: Database of orgs relevant to longtermist/x-risk work
Palisade Research
“At Palisade, our mission is to help humanity find the safest possible routes to powerful AI systems aligned with human values. Our current approach is to research offensive AI capabilities to better understand and communicate the threats posed by agentic AI systems.”
Jeffrey Ladish is the Executive Director.

MichaelA 8 Jun 2023 14:32 UTC
2 points
0 ∶ 0
in reply to: MichaelA’s comment on: Database of orgs relevant to longtermist/x-risk work
Admond
“Admond is an independent Danish think tank that works to promote the safe and beneficial development of artificial intelligence.”
“Artificial intelligence is going to change Denmark. Our mission is to ensure that this change happens safely and for the benefit of our democracy.”

MichaelA 8 Jun 2023 14:31 UTC
2 points
0 ∶ 0
in reply to: MichaelA’s comment on: Database of orgs relevant to longtermist/x-risk work
Senter for Langsiktig Politikk
“A politically independent organisation aimed at creating a better and safer future”
A think tank based in Norway.

Rethink Priorities is hiring a Compute Governance Researcher or Research Assistant

MichaelA7 Jun 2023 13:22 UTC

36 points

2 comments8 min readEA link

(careers.rethinkpriorities.org)

MichaelA 4 Jun 2023 15:40 UTC
4 points
0 ∶ 0
on: Announcement: you can now listen to all new EA Forum posts
Tentative suggestion: Maybe try to find a way include info about how much karma the post has near the start of the episode description, in the podcast feed?
Reasoning:
- This could help in deciding what to listen to, at least for the “all audio” feed. (E.g. I definitely don’t have time for even just all AI-related episodes in there.)
- It could also led to herd-like behavior or ignoring good content that didn’t get lots of karma right away. But I think that that is outweighed by the above benefit.
- OTOH this may just be infeasible to do in a non-misleading way, if you put things in the feed soon enough after they’re posted that the karma hasn’t really stabilized yet* and if it’s hard to automatically update the description to reflect karma scores later.
  - *My rough sense is that karma scores are pretty stable after something like 3-7 days—stable enough that something like “karma after 5 days was y” is useful info—but that if you can only show karma scores after e.g. 1 day then that wouldn’t be very informative.

MichaelA 4 Jun 2023 15:35 UTC
4 points
0 ∶ 0
on: Announcement: you can now listen to all new EA Forum posts
Thanks! This seems valuable.
One suggestion: Could the episode titles, or at least the start of the descriptions, say who the author is?
Reasoning:
- I think that’s often useful context for the post, and also useful info for deciding whether to read it (esp. for the feed where the bar is “just” >30 karma).
- I guess there are some upsides to nudging people to decide just based on topic or the start of the episode rather than based on the author’s identity. But I think that’s outweighed by the above points.

MichaelA 28 May 2023 15:07 UTC
2 points
0 ∶ 0
in reply to: MichaelA’s comment on: Database of orgs relevant to longtermist/x-risk work
Confido Institute
Hi, we are the Confido Institute and we believe in a world where decision makers (even outside the EA-rationalist bubble) can make important decisions with less overconfidence and more awareness of the uncertainties involved. We believe that almost all strategic decision-makers (or their advisors) can understand and use forecasting, quantified uncertainty and public forecasting platforms as valuable resources for making better and more informed decisions.
We design tools, workshops and materials to support this mission. This is the first in a series of multiple EA Forum posts. We will tell you more about our mission and our other projects in future articles.
In this post, we are pleased to announce that we have just released the Confido app, a web-based tool for tracking and sharing probabilistic predictions and estimates. You can use it in strategic decision making when you want a probabilistic estimate on a topic from different stakeholders, in meetings to avoid anchoring, to organize forecasting tournaments, or in calibration workshops and lectures. We offer very high data privacy, so it is used also in government setting. See our demo or request your Confido workspace for free.
The current version of Confido is already used by several organizations, including the Dutch government, several policy think tanks and EA organizations.
Confido is under active development and there is a lot more to come. We’d love to hear your feedback and feature requests. To see news, follow us on Twitter, Facebook or LinkedIn or collaborate with us on Discord. We are also looking for funding. [emphasis added]

MichaelA 28 May 2023 15:05 UTC
2 points
0 ∶ 0
in reply to: MichaelA’s comment on: Database of orgs relevant to longtermist/x-risk work
Epistea
We are announcing a new organization called Epistea. Epistea supports projects in the space of existential security, epistemics, rationality, and effective altruism. Some projects we initiate and run ourselves, and some projects we support by providing infrastructure, know-how, staff, operations, or fiscal sponsorship.
Our current projects are FIXED POINT, Prague Fall Season, and the Epistea Residency Program. We support ACS (Alignment of Complex Systems Research Group), PIBBSS (Principles of Intelligent Behavior in Biological and Social Systems), and HAAISS (Human Aligned AI Summer School).

MichaelA

“Key takeaways

Additional object-level notes

Research proposal

RP’s AI Gover­nance & Strat­egy team—June 2023 in­terim overview

Re­think Pri­ori­ties is hiring a Com­pute Gover­nance Re­searcher or Re­search Assistant

RP’s AI Governance & Strategy team—June 2023 interim overview

Rethink Priorities is hiring a Compute Governance Researcher or Research Assistant