I am a computer scientist (to degree level) and legal scholar (to PhD level) working at the intersection between technology and law. I currently work in a legislation role at a major technology company, and as a consultant to government and industry on AI Law, Policy, Governance, and Regulation.
CAISID
Karma: 265
At the risk of damaging my networks in EA, I am inclined to tentatively agree with some of your comment. Disclaimer here that I have very little interaction with forecasting for various reasons, so this is more of a general comment than anything else.
I think one of the major problems I see in EA as a whole is a fairly loose definition of ‘impact’. Often I see five or six groups using vast sums of money and talent to produce research or predictions that are shared and reviewed between each other and then hosted on the websites but never seem to actually be implemented anywhere. There’s no external (of EA) stakeholder participation, no follow-up to check for changed trends, no update on how this affects the real-world outside of EA circles.
I don’t always think paying clients are the best measurement system for impact, but I do think there needs to be a much higher focus on bridging the connection between high-quality forecasting and real-world decision-makers.
Obviously this doesn’t apply everywhere in EA, and there are lots and lots of exemptions, but I do think your comment has merit.
List of Good Beginner-friendly AI Law/Policy/Regulation Books
UK AI Bill Analysis & Opinion
Thank you for making this interesting post. It’s certainly something that pops up in forum discussions so it’s useful to see in a single post. Obviously without concrete examples it’s hard to delve into the details but I think it’s worth engaging on the discussion on an, ironically, more abstract level.
I think a lot of this comes down to how individual people define ‘impact’, which you do mention in your post. For some, increasing academic knowledge of a niche topic is impact. Other people might perceive citations as impact. For others, publishing a research paper that only gets read by other EA orgs but increases their social standing and therefore likelihood for further funding or work is impact. For some career capital is the intended impact. Some people measure impact only by the frontline change it elicits. This seems like the focus of your post unless I am mistaken, so it sounds like your post boils down to ‘EA-centric research doesn’t cause real-world, measurable change often enough’.
If that is the measure of impact you think is important, I think your post has some merit. That’s not to say the other two are any lesser, or deserve less attention, but I think you are correct that there’s an ‘impact gap’ near the end of the research-to-change pipeline.I can only speak to AI Governance as that is my niche. As fortune would have it, my career is in AI Governance within organisational change—that is to say my role is to enter private or public sector organisations to a greater or lesser extent and then help create new AI governance and policy on either a project or org-wide basis. So my feedback/thoughts here come with that experience but also that bias. I’ll also take the opportunity to point out that AI governance isn’t just about lobbying politicians but there’s lots of wider organisational work there too, though I understand the oversight was likely word-count related.
Generally I think the problem you describe isn’t so much one within EA as it is one within wider academia. During my PhD I got declined travel funding to present my research to government decision-makers at a government-sponsored event because it wasn’t an ‘academic conference’ and therefore failed their ‘impact’ criteria. I was accepted by the same fund the year previous to that to present a (hardly ground-breaking) poster at a 35-person conference. I was very upset at the time because I had to miss that meeting and that opportunity passed me by, and I was frustrated that they gave me money to attend a conference that changed nothing and didn’t give me the money I needed to make a big impact the year later. It was only later that I realised they just wanted different outcomes than me.
The problem there was that the university’s definition of ‘impact’ differed from mine, so by their metric presenting a poster at an academic conference to 35 people was more impactful to their criteria than my meeting with government officials to show my research. It’s a handy example of the fact that impact maps to goals.
So I think what it boils down to is how much this concept of goal-related impact bleeds into EA.
There is also a difference between research that you think should change the behaviour of decision makers, and what will actually influence them. While it might be clear to you that your research on some obscure form of decision theory has implications for the actions that key decision makers should take, if there is a negligible chance of them seeing this research or taking this on board then this research has very little value.
This point features partly in a post I am currently writing for Draft Amnesty Week, but essentially I think you’re correct that in my work in more ‘frontline’ AI governance I’ve found that anecdotally roughly 0% of decision-makers read academic research. Or know where it is published. Or how to access it. That’s a real problem when it comes to using academic research as an influence lever. That’s not to say the research is pointless, it’s just that there’s extra steps between research and impact that are woefully neglected. If end-user change is something that is important to you as a researcher, it would be understandably frustrating for this hurdle to reduce that impact.This isn’t an EA issue but a field issue. There’s plenty of fantastic non-EA AI governance research which lands like a pin-drop, far from the ears of decision-makers, because it wasn’t put in the right hands at the right time. The problem is many decision-makers where it counts (particularly in industry) get their knowledge from staff, consultants, dedicated third-party summary organisations, or field-relevant newsletters/conferences. Not directly from academia.
One caveat here is that some fields, like Law, have a much greater overlap of ‘people reading/publishing’ and ‘decision-makers’. This is partly because publishing and work in many legal sectors are designed for impact in this way. So the above isn’t always ironclad, but it largely tracks for general decision-making and AI governance. I find the best EA orgs at generating real-world impact are the orgs in the legal/policy because of the larger than normal amount of legal and policy researchers there coupled with the fact they are more likely to measure their success by policy change.
A further complicating factor that I think contributes to the way you feel is that unfortunately some AI Governance research is undertaken and published by people who don’t always have lots of experience in large organisations. Perhaps they spent their entire career in academia, or have worked only in start-ups, or via different paths, but that’s where you see different ‘paths to impact’ which don’t translate well to larger-scale impact like the type you describe in your post. Again the reason here is that each of these spheres have their own definition of what constitutes ‘impact’ and it doesn’t always translate well.
As a partial result of this I’ve seen some really good AI governance ideas pitched really badly, and to the wrong gatekeeper. Knowing how to pitch research to an organisation is a skillset curated by experience, and the modern academic pathway doesn’t give people the opportunity to gain much of that experience. Personally, I just learned it by failing really hard a lot of times early in my career. For what it’s worth, I’d 100% recommend that strategy if there’s any early careers folks reading this.
I will disagree with you on one point here:
Soon after the initial ChatGPT launch probably wasn’t the right time for governments to regulate AI, but given the amount of funding that has gone into AI governance research it seems like a bad sign that there weren’t many (if any) viable AI governance proposals that were ready for policymakers to take off-the-shelf and implement.
I’ll be pedantic and point out that governments already do regulate AI, just to different extents than some would like, and that off-the-shelf governance proposals don’t really exist because of how law and policy works. So not sure this is a good metric to use for your wider point. Law and policy of AI is literally my career and I couldn’t create an off-the-shelf policy that was workable just because of how many factors are required to be considered.
It seems like EA think tanks are becoming more savvy and gradually moving in the direction of action-guiding research and focusing on communicating to decision makers, especially in AI governance.
Taking a leaf from your vagueness book, I’ll say that in my experience some of the EA or EA-adjacent AI governance orgs are really good at engaging external stakeholders, and some are less good. I say this as an outsider because I don’t work for and nor have I ever worked for an EA org, but I do follow their research. So take this with appropriate pinches of salt.I think part of the disparity is that some orgs recruit people with experience in how internal government decision-making works—ie people who have worked in the public sector or have legal or policy backgrounds. Some others don’t. I think that translates largely to their goal. It’s not random that some are good at it and some not so much, it’s just some value that and some don’t—therefore effort is invested in change impact or it isn’t.
If an EA research org defines ‘impact’ as increasing research standing within EA, or amount of publications per year, or amount of conferences attended, then why would they make effort to create organisational change? Likewise, I don’t publish that much because it’s just not directly related to how effective my measurements of my own impact are. Neither is better, it just relates to how goals are measured.
If, as I think your post details, your criticism is that EA research doesn’t create more frontline change often enough, then I think that there are some relatively simple fixes.
EA research has something of a neglect of involving external stakeholders which I think links back to the issues you explore in your post. Stakeholder engagement can be quite easily and well integrated into AI Governance research as this example shows, and that’s quite an easy (and often non-costly) methodology to pick up that can result in frontline impact.
Stakeholder-involved research must always be done carefully, so I don’t blame EA funding orgs or think tanks for being very careful in approaching it, but they need to cultivate the right talent for this kind of work and use it because it’s very important.
I think a solution would be to offer grants or groups for this specific kind of work. Even workshops for people might work. I’d volunteer some of my experience for that, if asked to do so. Just something to give researchers who want the kind of impact you describe, but don’t know how to do it, a head-start.
I think impact-centric conferences would also be a good idea. Theoretical researchers do fantastic work, and many of us more involved in the change side of things couldn’t do our jobs without them, so creating a space where those groups can exchange ideas would be awesome. EAGs are good for that, I find. I often get a lot of 1-1s booked, and I get a lot from them too.
AUKUS Military AI Trial
This is a really good post, and something that is both complex and important. Thank you for taking the time to make it.
I think from some of the language used that this is a US-centric lean. I’m UK-based, but given the mutual common law system I think there’s some common ground to comment from. Given the jurisdictional difference none of these are criticisms or feedback, but are comparative explorations of your post with our own circumstance.
One thing I think is key from a liability perspective is the lack of IP protections for algorithms used as evidence. It may not be the case in the US, but in England & Wales there are very few (if any) intellectual property protections for algorithms once they are out in the open. This makes developers and retailers absurdly reluctant to disclose algorithms in court even if they are in the right. I think that is a niggle when it comes to using liability as a lever, and something that needs sorted.
The UK also doesn’t really have much of a punitive system in most cases, and it’s almost entirely based on “putting the individual back where they were”, with the exception of violated safety standards etc. I think using tort law is still feasible though, if we can address the IP issue and link harms more to ECHR, data law, and similar.
I don’t think there needs to be a worry about imminent existential risk to use tort law as a lever. Nuclear regulation has existential risk, but the safety standards still have plenty of punitive element there, as well as a tort law aspect. It links in with your strict liability point, but I do wonder how we would define ‘unpredictable and uncontrollable’ in terms of AI systems—though likely that is one for the guidance notes.
I do think there would be merit in compulsory insurance, and a portion of that insurance paying into a pot from which significant uninsured harms are compensated from. The UK does that with motor insurance, but other countries likely have similar systems.
Really enjoyed this post, and hope to see more from you soon.
I’m not entirely sure why you’re being karma-bombed for this. I’ve done what I can to bring your score up towards 0.
Agree or disagree, I don’t think your comment breaks any rules or norms, and was well written with reasoning provided. Don’t take the weird scoring to heart.
This is a useful list, thank you for writing it.
In terms of:UK specific questions
Could the UK establish a new regulator for AI (similar to the Financial Conduct Authority or Environment Agency)? What structure should such an institution have? This question may be especially important because the UK civil service tends to hire generalists, in a way which could plausibly make UK AI policy substantially worse.
I wrote some coverage here of this bill which seeks to do this, which may be useful for people exploring the above. Also well worth watching and not particularly well covered right now is how AUKUS will affect AI Governance internationally. I’m currently preparing a deeper dive on this as a post, but for people researching UK-specific governance it’s a good head start to look at these areas as ones where not a lot of people are directing effort.
This was a really interesting read. Anecdotally, I’d put in that since having my two children my dedication to the future has increased dramatically. I genuinely care about, and I will butcher this saying, planting trees the shade of which I will never sit under.
Maybe I was a bad, amoral person and now I have a personal interest in making a better world. Maybe this works only for me. But knowing the world I help build will be the one my children have to live in has certainly made me choose better career paths and projects, as well as adding rocket fuel to my motivation.As always, mileage will vary significantly between people.
Great post as always, Nathan :)
Defence dominates government technical research
Reasonable balance of in-house and contracting. No obvious pattern in which is more successful.
It’s a weird one because the actual military has extraordinary problems attracting talent in these areas because they can’t compete with the salary, perks, or lifestyle of the private sector in any respect at all. So the military itself doesn’t do much technical R&D in a large scale that isn’t led by government or private enterprise.However, the defence industry (the private entities who manufacture much of the gear) has the funds and the network to hire top-tier specialists, and due to the extremely high levels of regulatory oversight and compliance evidence required, they generally have really good safety systems in place. As a result it’s an attractive place to work for the upskilling of the technical staff. If you spend 25% of your workday talking to legal and governance, you pick things up.
The downside for many government and almost all defence roles though is the significant amounts of (very personally invasive) clearance required to work on projects. An academic can’t just collaborate with those industries on a whim. Even most defence fellowships (like RAND) require at least SC level of clearance. It’s pretty much why there’s such a quality divide, in that someone who knows they can pass a clearance will do so for the double salary and long-term career benefits. This means though that hiring someone ‘fresh’ will take 6 months to a year, which is a long runway. But most of these projects have very long runways. There’s lots of IT staff now trying to gain clearances just to get into that industry on a contractor basis, but the vetting is very much a bottleneck for them.
An exception to the high quality I’d say is criminal or national intelligence where the actual intelligence agencies have super-high levels of qualified individuals top in their field, whereas policing itself is the opposite in that they’re like the military itself where they just can’t compete for talent. When working together it’s not a problem, but solo technical endeavours by police forces (at least in the UK) have a pretty spotty hit rate. Some great. Some not so great.
One thing to consider is that the line between government and not-government can get super blurry in technical research, or R&D in general.
No, no additional clarity needed at all—it was obvious. I just didn’t want it to come off like I was criticising rather than saying how this could work on our side of the pond :) I’ll be sure to give that a read tonight!
This is a very interesting read. I have some feedback (mostly neutral) to help further shape these ideas. Mostly just to prompt and direct some interest extra thought directions. It’s not a list of criticisms, but myself just spitballing ideas on top of your foundations:
A product that carries large negative externalities
You mention aerospace and nuclear as good demonstrations of regulations and to an extent I agree, but a potential weakness here is the strength of regulation comes a lot from the fact that these developers are often either monopsonies or close. That is, there are very few builders of these systems and they have either very few or singular customers, as well as access to top-tier legal talent. AI development is much more diverse, and I think this makes regulation harder. Not saying your idea on this element is bad—it’s very good—but it’s something to bear in mind. This would be an interesting governance category to maybe split into subcategories.
Innovation policy
This is a good idea again, and the food-for-thought relates to the above quite closely too again with the nuclear mention. One thing I’d look into is positive influence on procurement—make it more rewarding for an organisation to buy (or make) safer AI than the financial reward is for not doing that. Policing in England and Wales is experiencing a subtle shift like this right now which has actually been very impactful.
A national security risk
Obviously it’s hard to get detailed in an overview post, but WMDs are regulated in a specific way which doesn’t necessarily marry well to NatSec. There’s some great research right now on how NatSec related algorithms and transparency threats are beginning to be regulated, with some recent trials of regulations.
Preventing competitive dynamics
Not much to say here, as this is outside my expertise area. I’ll leave that for others.As an instrument of great power conflict
This was an interesting point. One thing I’d highlight is though most of my work is in AI regulation, I’ve done a bunch of Space regulation too and a thing to bear in mind is that space law has aged horribly and is stagnant. One of the main issues is that it was written when there were three space powers (mainly US and USSR, with UK as a US-aligned third space power), and the regulation was written with the idea of a major tech bottleneck to space and the ability for two nations to ‘police’ it all. This is more so true of the Outer Space Treaty than some of the treaties that followed and built on it. Obviously the modern day bears very little resemblance to that which makes things more difficult eg private entities exploring space, modern technology allowing autonomy. Worth thinking about how we would avoid this when we don’t know what the world will look like in 10, 25, 50 years.
Improving consumer welfare
This was a great point. Not much further direction to add here, other than looking at how some successful laws have been future-proofed against AI changes vs how some have been less successful.
Political economy
I’ve done a bunch of work in this area, but haven’t really got anything to add beyond what you’ve put.Military Technology
One of the major bottlenecks with this category is that most new projects happen far behind closed doors for decades. EA currently lacks much of a MilTech presence, which is a missed opportunity IMO.
All in all this is an interesting way to group AI governance areas, but I think some additional attention could be paid to how the markets behave there and how that affects regulation. Perhaps an extra category for monopsonies or large market suppliers at opposite ends of a spectrum?
Obviously it isn’t always perfect, and good posts with niche audiences aren’t always given much karma.
This was actually a surprisingly useful thing to hear, and I’m glad you included it. It can be quite disheartening (especially for first or second time posters) to spend weeks on a post for it to receive single digit karma after it goes live. I’d hate to think that ever puts anyone off. It’s just something that happens. Things like amnesty are good for overcoming that (I assume), especially people who are afraid to make posts for fear of harsh criticism.
I may have a crack at taking part in the draft amnesty, if any ‘would like to see’ suggestions match my research areas.
Looking forward to seeing what else this generates!
This looks like it produced a lot of really beneficial research and made a professional difference for people too. I also really like how this post is laid out. It’s a good example for similar reports. I’ve signed up for updates from PIBBSS—looking forward to seeing what is next!
I would like to see how many people read to the end. I have an 18 min read post, and I can see that many people read for 1m 30s so probably read the tl;dr or summary, but it does tell me that a number of people did read for much longer. I’d like to know how many read it to the end so I can tell if maybe posts need to be much shorter.
May not be possible, but wishlist idea would be to know a bit more about who these users are in terms of interest demographic. If my post is AI Safety but most of those who click away fast have animal welfare listed as their primary interest then that’s not an issue. But if my post is AI Safety and lots of AI Safety people didn’t read or engage, then that’s an issue I need to fix. This may not be feasible to implement however.
All in all I like it, and it feels like it gives insight into developing better engagement habits rather than encouraging clickbait.
I’m going to stick to forum norms and assume good faith in this question. There is also every chance I am misinterpreting it (it’s very early!).
I’m curious as to what has prompted “my guess is suicide hotline volunteers aren’t that stellar”, and the assumption that for whatever reason people in this forum would be any better? It seems the entire premise here rests on those two unevidenced assertions, which potentially explains why this path isn’t recommended more.
Compliance Monitoring as an Impactful Mechanism of AI Safety Policy
One of the things I think is important to remember when it comes to Defence is that the idea of boundaries between military technology and civilian technology hasn’t really existed since the 1970s. A vast amount of defence technology now is dual-use, meaning that even people working in (for example) the video games or automotive industry are, in a potentially unaware manner, designing hardware and software for the defence industry. And funnily enough vice-versa. So that line gets fuzzy fast. It sounds like your work is dual-use so it might be a bit complex for you to work through, in terms of ethics.
As for the hard ethics there, it depends on your own ethics and what you want to accomplish with the work. If the finance is the main draw, then that’s it’s own thing for only you to answer. If you want to make ‘wider impact’ in a positive way, then that’s a whole other thing that again I guess falls to you and relates largely to the role. There’s plenty of people work with stakeholders they aren’t exactly stoked about in order to achieve a larger goal.
I asked myself a similar question the first time I had the opportunity to do AI Governance with a police force, as someone who was from a background which often has friction with police. Some mixed feelings there. I eventually decided that the chance to make positive impact was worth it, but plenty of other people might feel otherwise.
In my job search until this point I have refused to apply to jobs at defense contractors and have turned down interviews from recruiters because it just seemed icky
I would end by saying that if something makes you feel ‘icky’ it might not be worth doing it, no matter what the more neutral ethics say. I’m happy with the lines I have drawn, and it’s important that you are as well. Not sure any of us can help with that :)
This was really interesting, and thank you for being so open. This is a really useful post.
Unfortunately you can plan perfectly, have the resources you need, put in all the effort, and a project still doesn’t work due to a variety of factors out of your control. There’s no shame in that, it happens to many of us, and I hope this doesn’t put you off doing similar work in future.
I wouldn’t be disheartened. I have considerable experience in AI safety and my current role has me advising decision-makers in the topic in major tech organisations. I’ve had my work cited by politicians in parliament twice last year.
I’ve also been rejected for every single AI Safety fellowship or scholarship that I’ve ever applied for. That’s every advertised one, every single year, for at least 5 years. My last rejection, actually, was on March 4th (so a week ago!). A 0% success rate, baby!
Rejected doesn’t mean you’re bad. It’s just that there’s maybe a dozen places for well over a thousand people, and remember these places have a certain goal in mind so you could be the perfect candidate but at the wrong career stage, or location, or suchlike.
I’d say keep applying, but also apply outside the EA sphere. Don’t pigeonhole yourself. As others mentioned, keep developing skills but I’d also add that you may never get accepted and that’s okay. It’s not a linear progression where you have to get one of these opportunities before you make impact. Check out other branches.
Inbox me if you feel you need more personal direction, happy to help :)