MarkusAnderljung 8 Jun 2021 18:08 UTC
15 points
0 ∶ 0
in reply to: Misha_Yagudin’s comment on: Jade Leung: Why Companies Should be Leading on AI Governance
Some brief thoughts (just my quick takes. My guess is that others might disagree, including at GovAI):
- Overall, I think the situation is quite different compared to 2018, when I think the talk was recorded. AI governance / policy issues are much more prominent in the media, in politics, etc. The EU Commission has proposed some pretty comprehensive AI legislation. As such, there’s more pressure on companies as well as governments to take action. I think there’s also better understanding of what AI policy is sensible. All these things update me against 1 (insofar as we are still in the formative stages) and 2. They also update me in favour of thinking something like: governments will want to take a bunch of actions related to AI and so we should try to steer those actions in positive directions.
- I think the AI policy / governance field is mature enough at this point that it’s not that helpful to think of an AI governance regime as one unitary thing. I much prefer thinking about specific areas of AI governance. Depending on the area, I’d likely have different views on 1-3. For example, it seems likely that companies are best placed to help develop standards that may be used to inform legislation further down the line. I wouldn’t expect companies to be best placed to figure out what the US should do wrt updates to antitrust regulation.
- On 3, I think it’s true that companies have incentives in favour of acting prosocially and that we can boost these incentives. I’m not sure those incentives outweigh their other incentives, though. The view is not that e.g. Facebook, Amazon, Google, are all-things-considered going to act in the public interest. I also don’t think Jade-2018 held that view.

MarkusAnderljung 9 Jul 2021 12:00 UTC
14 points
0 ∶ 0
in reply to: Ofer’s comment on: The Centre for the Governance of AI is becoming a nonprofit
Thanks for the question. I agree that managing these kinds of issues is important and we aim to do so appropriately.

GovAI will continue to do research on regulation. To date, most of our work has been fairly foundational, though the past 1-2 years has seen an increase in research that may provide some fairly concrete advice to policymakers. This is primarily as the field is maturing, as policymakers are increasingly seeking to put in place AI regulation, and some folks at GovAI have had an interest in pursuing more policy-relevant work.

My view is that most of our policy work to date has been fairly (small c) conservative and has seldom passed judgment on whether there should be more or less regulation and praising specific actors. You can sample some of that previous work here:
We’re not yet decided on how we’ll manage potential conflicts of interest. Thoughts on what principles are welcome. Below is a subset of things that are likely to be put in place:
- We’re aiming for a board that does not have a majority of folks from any of: industry, policy, academia.
- Allan will be the co-lead of the organisation. We hope to be able to announce others soon.
- Whenever someone has a clear conflict of interest regarding a candidate or a piece of research – say we were to publish a ranking of how responsible various AI labs were being – we’ll have the person recuse themselves from the decision.
- For context, I expect most folks who collaborate with GovAI to not be directly paid by GovAI. Most folks will be employed elsewhere and not closely line managed by the organization.
What links here?
- Long-Term Future Fund: July 2021 grant recommendations by abergal (18 Jan 2022 8:49 UTC; 75 points)

MarkusAnderljung 31 May 2021 17:38 UTC
14 points
0 ∶ 0
on: Long-Term Future Fund: May 2021 grant recommendations
I’m really excited to see LTFF being in a position to review and make such a large number of grants. IIRC, you’re planning on writing up some reflections on how the scaling up has gone. I’m looking forward to reading them!

MarkusAnderljung 22 Sep 2020 15:13 UTC
14 points
0 ∶ 0
in reply to: jia’s comment on: AMA: Markus Anderljung (PM at GovAI, FHI)
Thanks, Jia!
Could you say more about the different skills and traits relevant to research project management?
Understanding the research: Probably the most important factor is that you’re able to understand the research. This entails knowing how it connects to adjacent questions / fields, having well thought-out models about the importance of the research. Ideally, the research manager is someone who could contribute, at least to some extent, to the research they’re helping manage. This often requires a decent amount of context on the research, often having spent a significant amount of time reading the relevant research and talking to the relevant people.
Common sense & wide expertise: One way in which you can help as a research manager is often to suggest how the research relates to work by others, and so having decently wide intellectual interests is useful. You also want to have a decent amount of common sense to help make decisions about things like where something should be published and what ways a research project could go wrong.
Relevant epistemic virtues: Just like a researcher, it seems important to have incorporated epistemic virtues like calibration, humility, and other truth-seeking behaviours. As a research manager, you might be the main person that needs to communicate these virtues to new potential researchers.
People skills: Seems very important. Being able to do things like helping people become better researchers by getting to know what motivates them, what tends to block them, etc. Also being able to deal with potential conflicts and sensitive situations that can arise in research collaborations.
Inclination: I think there’s a certain kind of inclination that’s helpful to do research management. You’re excited about dabbling in a lot of different questions, more so than really digging your head down and figuring out one question in depth. You’re perhaps better at providing ideas, structure, conceptual framing, feedback, than doing the nitty-gritty of producing all the research yourself. You also probably need to be fine with being more of a background figure, and let the researchers shine.

MarkusAnderljung 22 Sep 2020 9:15 UTC
14 points
0 ∶ 0
in reply to: Misha_Yagudin’s comment on: AMA: Markus Anderljung (PM at GovAI, FHI)
Thanks Misha!
Not sure I’ve developed any deep insights yet, but here are some things I find myself telling researchers (and myself) fairly often:
- Consider a wide range of research ideas. It’s easy to get stuck in a local optimum. We often have people write out at least 5 research ideas and rate them on criteria like “fit, importance, excitement, tractability”, e.g. when they join as a GovAI Fellow. You should also have a list of research ideas that you periodically look through.
- Think about what output you’re aiming at from the start. It will determine the style, what literature you need to read, length, etc. Reworking a piece from one style to another can often take up to 20 hours.
- Make outlines often and early. This will help you be clear about what your argument is. Also, start writing earlier than you feel comfortable with. Often, things feel much clearer in your head than they actually are.
- If you have a high barrier to start writing, find ways to lower it. You can write outlines, dictate, say it out loud to a friend, set a timer during which you’re just allowed to write, no editing.
- Discuss your ideas with others often. I find that a lot of good thinking, in particular on how to frame something, comes from discussion. Just the act of putting your thoughts into words helps.
- Clear writing is hugely valuable. A lot of research gets less attention than it deserves because it takes too much effort to parse. Trying to write clearly will also often highlight complexities or problems with your argument that can be hidden in unclear prose.
- Don’t reinvent the wheel. There’s been a lot of smart people trying to figure out things about the world before you. Make a proper effort to try to find their writings and learn from them.
- Quality over quantity. The impact of research is probably heavy-tailed, and we have okay means to tell the quality of a piece ex ante.
- If you’re just starting out, find ways to work closely with people you can learn from. To do this, it’s much easier to support some senior person’s current agenda, rather than convincing them to help you on your research ideas. That way, they will very clearly get value out of working with you, and you’ll get an inside view into their research intuitions. For example, I think working as a research assistant to a great researcher is one of the best ways you can learn the ropes.
- As a facilitator of research, there’s also a bunch of common sense things you can do: Set deadlines for people, organise talks, connect people with the right people.
- There’s a lot I could say about feedback, but I’ll just say some things on how to ensure you actually get the feedback you need. Often this is difficult as the people whose feedback is most valuable to you have a lot of things pulling at their time.
  - Make it easy. Give people instructions about what parts they should focus on, what questions you want them to think about etc. You can also pull out specific questions where you think they have expertise and just send those along.
  - Be accommodating. Some people find it much less costly to give feedback after having listened to you give a talk or prefer giving feedback over a call. Others prefer doing everything via text.

MarkusAnderljung 20 Dec 2021 13:45 UTC
12 points
0 ∶ 0
on: EU AI Act now has a section on general purpose AI systems
Overall, I think it’s not that surprising that this change is being proposed and I think it’s a fairly reasonable. However, I do think it should be complemented with duties to avoid e.g. AI systems being put to high-risk uses without going through a conformity assessment and that it should be made clear that certain parts of the conformity assessment will require changes on the part of the producer of a general system if that’s used to produce a system for a high-risk use.

In more detail, my view is that the following changes should be made: Goal 1: Avoid general systems being without the appropriate regulatory burdens kicking in. There are two kinds of cases one might worry about: (i) general systems might make it easier to produce a system that should either be covered by the transparency requirements (e.g. if your system is a chatbot, you need to tell the user that) or the high-risk requirements, leading to more such systems being put on the market without them being registered.

Proposed solution: Make it the case that providers of general systems must do certain checks on how their model is being used and whether it is being used for high risk uses without that AI system having been registered or having gone through the conformity assessment. Perhaps this would be done by giving the market surveillance authorities (MSAs) the right to ask providers of general models about certain information about how the model is being used. In practice, it could look as follows: the provider of the general system could have various ways to try to detect whether someone is using their system for something high risk (companies like OpenAI are already developing tools and systems to do this). If they detect such a use, they are required to check that against the database of high risk AI systems deployed on the EU market. If there’s a discrepancy, they must report it to the MSA and share some of the relevant information as evidence.

(ii) There’s a chance that individuals using general systems for high-risk uses without placing anything on the market will not be covered by the regulation. That is, as the regulation is currently designed, if a company where to use public CCTV footage to assess the number of women vs. men walking down a street, I believe that would be a high risk use. But if an individual does it, it might not count as a high risk use because nothing is placed on the market. This could end up being an issue, especially if word about these kinds of use cases spreads. Perhaps a more compelling example would be people starting to use large language models as personal chat bots. The proposed regulation wouldn’t require the provider of the LLM to add any warnings about how this is simply a chatbot, even if the user starts e.g. using it as a therapist or for medical advice.

Proposed solution: My guess is that the solution is that the provision suggested above is expanded to also look for individuals using the systems for high risk or limited risk uses and that they are required to stop such use.

Goal 2: (perhaps most important) Try to make it the case that crucial and appropriate parts of the conformity assessment will require changes on the part of the producer of the general system.

This could be done by e.g. making it the case that the technical documentation requires information that only the producer of the general model would have. It would plausibly already be the case with regards to the data requirements. It would also plausibly be the case regarding robustness. It seems worth making sure of those things. I don’t know if that’s a matter of changing the text of the legislation itself or about how the legislation will end up being interpreted.

One way to make sure that this is the case is to require that deployers only use general models that have gone through a certification process or that has also passed the conformity assessment (or perhaps a lighter version). I’m currently excited about the latter.

Why am I not excited about something more onerous on the part of the provider of the general system?
- I think we can get a lot of the benefits of providers of general systems needing to meet certain requirements without them having to go through the conformity assessment themselves. I expect there to be lots of changes that need to be made to the general model to allow the deployer to complete their conformity assessment. If I try to use GPT-3 to create a system that rates essays (ignoring for now that OpenAI currently prohibit this in their Terms of Use), I’ll need to make sure that the system meets certain robustness requirements, I need to be able to explain to some human overseer how it works, and so on. To meet those requirements, I think that’ll require changes on the part of the developer of the general system. As such, I think the legal requirements will have an effect on general AI systems produced by big tech companies. To illustrate the point, if EU car manufacturers are required to use less carbon-intensive steel, that would have a large impact on the carbon-intensity of steel production in the EU, even though the steel manufacturers weren’t directly targeted by the legislation.
- Introducing requirements on all general systems that can be used on the EU market seems hugely onerous to me. So much so that it would probably be a bad idea. I think that companies could fairly easily go from offering a general system on the EU market to offering a general-system-that-you’re-not-allowed-to-use-for-high-risk-uses. This could for example be done by adjusting the terms and conditions (OpenAI’s API usage guidelines already disallows most if not all high-risk uses as defined in the AI Act) or writing in big font somewhere “Not intended for high-risk uses as defined by the EU’s AI Act”. I worry that introducing requirements on general systems on masse would lead to that being the default response and that it wouldn’t deliver much benefit beyond what we’d get if the changes I gestured at above were made.

[link] Centre for the Governance of AI 2020 Annual Report

MarkusAnderljung14 Jan 2021 10:23 UTC

11 points

5 comments1 min readEA link

MarkusAnderljung 17 Jan 2020 10:07 UTC
11 points
0 ∶ 0
on: Five GCR grants from the Global Challenges Foundation
I have some familiarity with the org. Happy to chat to people who are considering applying.

MarkusAnderljung 4 Dec 2018 20:43 UTC
11 points
0 ∶ 0
in reply to: Jan_Kulveit’s comment on: So you want to do operations—which skills do you need?
I think what really filters down the number of candidates significantly is that most organizations want to fill ops-roles with people who are able to do their job very autonomously. This means that a premium is put on something like value-alignment and good judgment. These two factors significantly narrows down the talent pool.

MarkusAnderljung 4 Aug 2022 22:05 UTC
10 points
0 ∶ 0
on: 2022 AI expert survey results
Really excited to see this!
I noticed the survey featured the MIRI logo fairly prominently. Is there a way to tell whether that caused some self-selection bias?
In the post, you say “Zhang et al ran a followup survey in 2019 (published in 2022)1 however they reworded or altered many questions, including the definitions of HLMI, so much of their data is not directly comparable to that of the 2016 or 2022 surveys, especially in light of large potential for framing effects observed.” Just to make sure you haven’t missed this: we had the 2016 respondents who also responded to the 2019 survey receive the exact same question they were asked in 2016, including re HLMI and milestones. (I was part of the Zhang et al team)