I am a computer scientist (to degree level) and legal scholar (to PhD level) working at the intersection between technology and law. I currently work in a legislation role at a major technology company, and as a consultant to government and industry on AI Law, Policy, Governance, and Regulation.
CAISID
Karma: 265
You have some interesting questions here. I am a computer scientist and a legal scholar, and I work a lot with organisations on AI policy as well as helping to create policy too. I can sympathise with a lot of the struggles here from experience. I’ll focus in some some of the more concrete answers I can give in the hopes that they are the most useful. Note that this explanation isn’t from your jurisdiction (which I assume from the FBI comment is USA) but instead from England & Wales, but as they’re both Common Law systems there’s a lot of overlap and many key themes are the same.
For example, one problem is: How do you even define what “AGI” or “trying to write an AGI” is?This is actually a really big problem. There’s been a few times we’ve trialled new policies with a range of organisations and found that how those organisations interpret the term ‘AI’ makes a massive difference to how they interpret, understand, and adhere to the policy itself. This isn’t even a case of bad faith, more just people trying to attach meaning to a vague term and then doing their best but ultimately doing so in different directions. A real struggle is that when you try to get more specific, it can actually end up being less clear because the further you zoom in, the more you accidentally exclude. It’s a really difficult balancing act—so yes, you’re right. That’s a big problem.
I’m wondering how much this is actually a problem, though. As a layman, as far as I know there could be existing government policies that are somewhat comparably difficult to evaluate.
Oh, tons. In different industries, in a variety of forms. Law and policy can be famously hard to interpret. Words like ‘autonomous’, ‘harm’, and ‘intend’ are regular prickly customers.
Many judicial decisions related to crimes, as I vaguely understand it, depend on intentionality and belief——e.g. for a killing to be a murder, the killer must have intended to kill and must not have believed on reasonable grounds that zer life was imminently unjustifiedly threatened by the victim.
This is true to an extent. So in law you often have the actus reus (what actually happened) and the mens rea (what the person intended to happen). The law tends to weigh the mens rea quite heavily. Yes, intent is very important—but more so provable intent. Lots of murder cases get downgraded to manslaughter for a better chance at a conviction. Though to answer your question yes at a basic level criminal law often relates to intention and belief. Most of the time this is the objective belief of the average person, but there are some cases (such as self-defence in your example) where the intent is measured against the subjective belief of that individual in those particular circumstances.What are some crimes that are defined by mental states that are even more difficult to evaluate? Insider trading? (The problem is still very hairy, because e.g. you have to define “AGI” broadly enough that it includes “generalist scientist tool-AI”, even though that phrase gives some plausible deniability like “we’re trying to make a thing which is bad at agentic stuff, and only good at thinky stuff”. Can you ban “unbounded algorithmic search”?)
Theft and assault of the everyday variety are actually some of the most difficult to evaluate really, since both require intent to be criminal and yet intent can be super difficult to prove. In the context of what you’re asking, ‘plausible deniability’ is often a strategy chosen when accused of a crime (i.e making the prosecution prove something non-provable which is an uphill battle) but ultimately it would come down to a court to decide. You can ban whatever you want, but the actual interpretation could only really be tested in that matter. In terms of broad language the definitions of words is often a core point of contention in court cases so likely it would be resolved there, but honestly from experience the overwhelming majority of issues never reach court. Neither side wants to take the risk, so usually the company or organisation backs off and negotiates a settlement. The only times things really go ‘to the hilt’ is for criminal breaches which require a very severe stepping over the mark.
Bans on computer programs. E.g. bans on hacking private computer systems. How much do these bans work? Presumably fewer people hack their school’s grades database than would without whatever laws there are; on the other hand, there’s tons of piracy.
In the UK the Computer Misuse Act 1990 is actually one of the oldest bits of computer-specific legislation and is still effective today after a few amendments. It’s mostly due to the broadness of the law and the fact that evidence is fairly easy to come by and that intent with those is fairly easy to prove. It’s beginning to struggle in the new digital era though, thanks to totally unforeseen technologies like generative AI and blockchain.
Some bits of legislation have been really good at maintaining bans though. England and Wales have a few laws against CSAM which included the term ‘pseudo-photography’ which actually applies to generative AI and so someone who launched an AI for that purpose would still be guilty of an offence. It depends what you mean by ‘ban’ as a ban in legislation can often function much differently than a ban from, for example, a regulator.
Bans on conspiracies with illegal long-term goals. E.g. hopefully-presumably you can’t in real life create the Let’s Build A Nuclear Bomb, Inc. company and hire a bunch of nuclear scientists and engineers with the express goal of blowing up a city. And hopefully-presumably your nuke company gets shut down well before you actually try to smuggle some uranium, even though “you were just doing theoretical math research on a whiteboard”. How specifically is this regulated? Could the same mechanism apply to AGI research?Nuclear regulation is made up of a whole load of different laws and policy types too broad to really go into here, but essentially what you’re describing there is less about the technology and more about the goal. That’s terrorism and conspiracy to commit murder just to start off with, no matter whether you use a nuke or an AGI or a spatula. If your question centres more on ‘how do we dictate who is allowed access to dangerous knowledge and materials’ that’s usually a licensing issue. In theory you could have a licensing system around AGIs, but that would probably only work for a little while and would be really hard to implement without buy-in internationally.
If you’re specifically interested in how this example is regulated, I can’t help you in terms of US law beyond this actually quite funny example of a guy who attempted a home-built nuclear reactor and narrowly escaped criminal charges—however some UK-based laws include the Nuclear Installations Act 1965 and much of the policy from the Office for Nuclear Regulation (ONR).
Hopefully some of this response is useful!
Yeah, that’s fixed for me :)
No worries :)
This is a useful list, thank you for writing it.
In terms of:UK specific questions
Could the UK establish a new regulator for AI (similar to the Financial Conduct Authority or Environment Agency)? What structure should such an institution have? This question may be especially important because the UK civil service tends to hire generalists, in a way which could plausibly make UK AI policy substantially worse.
I wrote some coverage here of this bill which seeks to do this, which may be useful for people exploring the above. Also well worth watching and not particularly well covered right now is how AUKUS will affect AI Governance internationally. I’m currently preparing a deeper dive on this as a post, but for people researching UK-specific governance it’s a good head start to look at these areas as ones where not a lot of people are directing effort.
This is really interesting, thank you. As an aside, am I the only one getting an unsecured network warning for nonlinear.org/network?
I wouldn’t be disheartened. I have considerable experience in AI safety and my current role has me advising decision-makers in the topic in major tech organisations. I’ve had my work cited by politicians in parliament twice last year.
I’ve also been rejected for every single AI Safety fellowship or scholarship that I’ve ever applied for. That’s every advertised one, every single year, for at least 5 years. My last rejection, actually, was on March 4th (so a week ago!). A 0% success rate, baby!
Rejected doesn’t mean you’re bad. It’s just that there’s maybe a dozen places for well over a thousand people, and remember these places have a certain goal in mind so you could be the perfect candidate but at the wrong career stage, or location, or suchlike.I’d say keep applying, but also apply outside the EA sphere. Don’t pigeonhole yourself. As others mentioned, keep developing skills but I’d also add that you may never get accepted and that’s okay. It’s not a linear progression where you have to get one of these opportunities before you make impact. Check out other branches.
Inbox me if you feel you need more personal direction, happy to help :)
This won’t be the answer you’re looking for but honestly, time permitting, I just take a day or three off. I find when I’m relaxing, giving myself space to breathe and think without force, that’s when creativity starts to flow again and ideas come in. Obviously this isn’t deadline-friendly!
This is a really interesting podcast—particularly the section with the discussion on foundation models and cost analysis. You mention a difficulty on exploring this. If you ever want to explore it, I’m happy to give some insight via inbox because I’ve done a bit of work in industry in this area that I can share.
“What makes you stop posting?” could be reframed as “What makes you post in the first place?”, and “What might make it easier?” could be reframed as “What might make you publish posts that were more challenging for you (practically or emotionally)?”
The quality of many forum posts is very high, including from people who are not paid by a research org to write them and have no direct connection to the community (such as these two). So even if you only factor in the time cost, you would still have to suppose some pretty large benefits to explain why people write them.
This was a really good point, and it made me think for quite a while. I’ve posted on the forum a lot since re-entering the EA community (to the point I’ve consciously tried to do it less so it’s not spammy!), but I’ve never really thought about why I put so much effort into posts or, indeed, my comments. There’s not much of a difference between the two, really, since one of my recent comments on someone’s post was 1,207 words long haha. All in good faith, though!
I don’t gain anything from posting. I have a good job outside of EA, I’m not part of any EA groups, and I don’t particularly want or need anything from anyone in EA. So there’s nothing concrete there. I’ve never really thought about it, but it boils down to sharing knowledge. If the things I know about can help someone else somewhere do good better, or address a problem, or whatever then I like the idea that maybe my posts are useful to people. My specialist area is also kind of niche and difficult to enter, so I like the idea of making it more understandable and approachable.
I never get any karma really, or even high reads, but I do get high retention so people (~50%) tend to read my posts all the way through which I really like. So that ties in with what I think my core motivation is.
Obviously I think it’s good to make sure criticism is of ideas and not people/their values, and to be polite in a common sense way such as trying to give criticism as a compliment sandwich.
I’m a huge fan of this. It’s rare, but if ever I really disagree with someone’s post I’ll always highlight what I liked about it too. In my experience aside from being polite, it also results in better conversation.
Edit: Grammar
My PhD was in this area so I’d be super interested in hearing more about your thoughts on this. Looking forward to seeing this post if you decide on it :)
I think this would be an interesting post to read. I’m often surprised that existing AI disasters with considerable death/harm counts aren’t covered in more detail in people’s concerns. There’s been a few times where AI systems have, via acting in a manner which was not anticipated by its creators, killed or otherwise injured very large numbers of people which exceed any air disaster or similar we’ve ever seen. So the pollution aspect is quite interesting to me.
Posts about any of the knock-on or tertiary effects of AI would be interesting.
One of the things I think is important to remember when it comes to Defence is that the idea of boundaries between military technology and civilian technology hasn’t really existed since the 1970s. A vast amount of defence technology now is dual-use, meaning that even people working in (for example) the video games or automotive industry are, in a potentially unaware manner, designing hardware and software for the defence industry. And funnily enough vice-versa. So that line gets fuzzy fast. It sounds like your work is dual-use so it might be a bit complex for you to work through, in terms of ethics.
As for the hard ethics there, it depends on your own ethics and what you want to accomplish with the work. If the finance is the main draw, then that’s it’s own thing for only you to answer. If you want to make ‘wider impact’ in a positive way, then that’s a whole other thing that again I guess falls to you and relates largely to the role. There’s plenty of people work with stakeholders they aren’t exactly stoked about in order to achieve a larger goal.
I asked myself a similar question the first time I had the opportunity to do AI Governance with a police force, as someone who was from a background which often has friction with police. Some mixed feelings there. I eventually decided that the chance to make positive impact was worth it, but plenty of other people might feel otherwise.
In my job search until this point I have refused to apply to jobs at defense contractors and have turned down interviews from recruiters because it just seemed icky
I would end by saying that if something makes you feel ‘icky’ it might not be worth doing it, no matter what the more neutral ethics say. I’m happy with the lines I have drawn, and it’s important that you are as well. Not sure any of us can help with that :)
I have a few ideas mulling in my head that I’m yet to decide if it would be useful. I’m unsure about posting them as not sure how popular the ‘listicle’ format would be as opposed to my normally very long and detailed posts/comments. These are:
Title: Top 5 Lessons Working in Frontline AI Governance
Summary: Things I’ve picked up regarding AI, risk, harms etc working in-industry in an AI governance role. Might end up being 10 things. Or 3. Depending how it goes.Title: Top 5 Tips for Early Careers AI Governance Researchers
Summary: Similar to the above, things I wish I had known when I was an ECR.
Title: Why non-AGI/ASI systems pose the greatest longterm risks
Summary: Using a comparison to other technologies, some of my ideas as to why more ‘normal’ AI systems will always carry more risk and capacity for harm, and why EA’s focus on ‘super-AI’ is potentially missing the wood for the trees.
Title: 5 Examples of Low-Hanging Policy Fruit to Reduce AI Risk
Summary: Again, a similar listicle-style of good research areas or angles that would be impactful ‘wins’ compared to the resources invested.
Thank you for making this interesting post. It’s certainly something that pops up in forum discussions so it’s useful to see in a single post. Obviously without concrete examples it’s hard to delve into the details but I think it’s worth engaging on the discussion on an, ironically, more abstract level.
I think a lot of this comes down to how individual people define ‘impact’, which you do mention in your post. For some, increasing academic knowledge of a niche topic is impact. Other people might perceive citations as impact. For others, publishing a research paper that only gets read by other EA orgs but increases their social standing and therefore likelihood for further funding or work is impact. For some career capital is the intended impact. Some people measure impact only by the frontline change it elicits. This seems like the focus of your post unless I am mistaken, so it sounds like your post boils down to ‘EA-centric research doesn’t cause real-world, measurable change often enough’.
If that is the measure of impact you think is important, I think your post has some merit. That’s not to say the other two are any lesser, or deserve less attention, but I think you are correct that there’s an ‘impact gap’ near the end of the research-to-change pipeline.I can only speak to AI Governance as that is my niche. As fortune would have it, my career is in AI Governance within organisational change—that is to say my role is to enter private or public sector organisations to a greater or lesser extent and then help create new AI governance and policy on either a project or org-wide basis. So my feedback/thoughts here come with that experience but also that bias. I’ll also take the opportunity to point out that AI governance isn’t just about lobbying politicians but there’s lots of wider organisational work there too, though I understand the oversight was likely word-count related.
Generally I think the problem you describe isn’t so much one within EA as it is one within wider academia. During my PhD I got declined travel funding to present my research to government decision-makers at a government-sponsored event because it wasn’t an ‘academic conference’ and therefore failed their ‘impact’ criteria. I was accepted by the same fund the year previous to that to present a (hardly ground-breaking) poster at a 35-person conference. I was very upset at the time because I had to miss that meeting and that opportunity passed me by, and I was frustrated that they gave me money to attend a conference that changed nothing and didn’t give me the money I needed to make a big impact the year later. It was only later that I realised they just wanted different outcomes than me.
The problem there was that the university’s definition of ‘impact’ differed from mine, so by their metric presenting a poster at an academic conference to 35 people was more impactful to their criteria than my meeting with government officials to show my research. It’s a handy example of the fact that impact maps to goals.
So I think what it boils down to is how much this concept of goal-related impact bleeds into EA.
There is also a difference between research that you think should change the behaviour of decision makers, and what will actually influence them. While it might be clear to you that your research on some obscure form of decision theory has implications for the actions that key decision makers should take, if there is a negligible chance of them seeing this research or taking this on board then this research has very little value.
This point features partly in a post I am currently writing for Draft Amnesty Week, but essentially I think you’re correct that in my work in more ‘frontline’ AI governance I’ve found that anecdotally roughly 0% of decision-makers read academic research. Or know where it is published. Or how to access it. That’s a real problem when it comes to using academic research as an influence lever. That’s not to say the research is pointless, it’s just that there’s extra steps between research and impact that are woefully neglected. If end-user change is something that is important to you as a researcher, it would be understandably frustrating for this hurdle to reduce that impact.This isn’t an EA issue but a field issue. There’s plenty of fantastic non-EA AI governance research which lands like a pin-drop, far from the ears of decision-makers, because it wasn’t put in the right hands at the right time. The problem is many decision-makers where it counts (particularly in industry) get their knowledge from staff, consultants, dedicated third-party summary organisations, or field-relevant newsletters/conferences. Not directly from academia.
One caveat here is that some fields, like Law, have a much greater overlap of ‘people reading/publishing’ and ‘decision-makers’. This is partly because publishing and work in many legal sectors are designed for impact in this way. So the above isn’t always ironclad, but it largely tracks for general decision-making and AI governance. I find the best EA orgs at generating real-world impact are the orgs in the legal/policy because of the larger than normal amount of legal and policy researchers there coupled with the fact they are more likely to measure their success by policy change.
A further complicating factor that I think contributes to the way you feel is that unfortunately some AI Governance research is undertaken and published by people who don’t always have lots of experience in large organisations. Perhaps they spent their entire career in academia, or have worked only in start-ups, or via different paths, but that’s where you see different ‘paths to impact’ which don’t translate well to larger-scale impact like the type you describe in your post. Again the reason here is that each of these spheres have their own definition of what constitutes ‘impact’ and it doesn’t always translate well.
As a partial result of this I’ve seen some really good AI governance ideas pitched really badly, and to the wrong gatekeeper. Knowing how to pitch research to an organisation is a skillset curated by experience, and the modern academic pathway doesn’t give people the opportunity to gain much of that experience. Personally, I just learned it by failing really hard a lot of times early in my career. For what it’s worth, I’d 100% recommend that strategy if there’s any early careers folks reading this.
I will disagree with you on one point here:
Soon after the initial ChatGPT launch probably wasn’t the right time for governments to regulate AI, but given the amount of funding that has gone into AI governance research it seems like a bad sign that there weren’t many (if any) viable AI governance proposals that were ready for policymakers to take off-the-shelf and implement.
I’ll be pedantic and point out that governments already do regulate AI, just to different extents than some would like, and that off-the-shelf governance proposals don’t really exist because of how law and policy works. So not sure this is a good metric to use for your wider point. Law and policy of AI is literally my career and I couldn’t create an off-the-shelf policy that was workable just because of how many factors are required to be considered.
It seems like EA think tanks are becoming more savvy and gradually moving in the direction of action-guiding research and focusing on communicating to decision makers, especially in AI governance.
Taking a leaf from your vagueness book, I’ll say that in my experience some of the EA or EA-adjacent AI governance orgs are really good at engaging external stakeholders, and some are less good. I say this as an outsider because I don’t work for and nor have I ever worked for an EA org, but I do follow their research. So take this with appropriate pinches of salt.I think part of the disparity is that some orgs recruit people with experience in how internal government decision-making works—ie people who have worked in the public sector or have legal or policy backgrounds. Some others don’t. I think that translates largely to their goal. It’s not random that some are good at it and some not so much, it’s just some value that and some don’t—therefore effort is invested in change impact or it isn’t.
If an EA research org defines ‘impact’ as increasing research standing within EA, or amount of publications per year, or amount of conferences attended, then why would they make effort to create organisational change? Likewise, I don’t publish that much because it’s just not directly related to how effective my measurements of my own impact are. Neither is better, it just relates to how goals are measured.
If, as I think your post details, your criticism is that EA research doesn’t create more frontline change often enough, then I think that there are some relatively simple fixes.
EA research has something of a neglect of involving external stakeholders which I think links back to the issues you explore in your post. Stakeholder engagement can be quite easily and well integrated into AI Governance research as this example shows, and that’s quite an easy (and often non-costly) methodology to pick up that can result in frontline impact.
Stakeholder-involved research must always be done carefully, so I don’t blame EA funding orgs or think tanks for being very careful in approaching it, but they need to cultivate the right talent for this kind of work and use it because it’s very important.
I think a solution would be to offer grants or groups for this specific kind of work. Even workshops for people might work. I’d volunteer some of my experience for that, if asked to do so. Just something to give researchers who want the kind of impact you describe, but don’t know how to do it, a head-start.
I think impact-centric conferences would also be a good idea. Theoretical researchers do fantastic work, and many of us more involved in the change side of things couldn’t do our jobs without them, so creating a space where those groups can exchange ideas would be awesome. EAGs are good for that, I find. I often get a lot of 1-1s booked, and I get a lot from them too.
This is a really interesting post, thank you for making it. I’ve written about similar internal safety methods before, as well as writing some longer comments on this topic on other people’s posts elsewhere, but I am very interested in the different angle of approach detailed in your post.
I wonder what windfalls would look like across various organisational types? A big division would be between private sector and public sector, and I would also be interested in seeing how this is different for monopsonies or more complex market sectors.This research showed that public sector organisations (at least in some sub-types) had a very specific set of concerns and desires for AI tools compared to what we’ve seen elsewhere. In a more anecdotal vein, in my work across both private and public sectors I have seen this pattern quite strongly represented. I would be interested to hear your thoughts about ways that alignment windfalls/taxes as described in your post could impact or be impacted by that side of things—the difference in what each sector considers taxes and windfalls when profit is not the main objective?
I guess the first question is do you like academia? Or do you like research? The two aren’t always the same. Personally, you couldn’t get me to do a job in academia if you paid me—which is funny because they barely pay anyone! I love research, I love complex cutting-edge stuff, but I hated hated hated academia. I never thrived there. The corporate environment, however, has been my natural space. It may be the same for you—have you tried different types of organisation? I can tell you from experience that research-centric corporate life is much slower paced and more secure than academia, with the downside of less creative freedom during work hours (until you get to mid level).
Honestly more than anything it sounds like you just maybe need a rest. People think a rest is a career-killer but it absolutely isn’t. And even if it was, it’d be much less of one than a burnout is. Maybe take a couple months to decompress—though I understand that financing such a rest is rarely easy.
I do a lot of hiring outside of academia and honestly a lot of your major worries are something I don’t even look at in a CV anyway. Don’t feel like you’d sabotaged or blown anything because you haven’t. Even within Academia, we all have our stuff. You’d be surprised how many well-known academics have had ‘wobbles’ that aren’t public knowledge.
Feel free to inbox me if you need more detailed advice. I’m not US based so unlikely to be useful for specifics, but always happy to hear spitballed ideas :)
Is there a particular way this relates to current cause areas? Or wider EA? The linked article is paywalled so apologies if I’ve missed an obvious link :)
Thank you for this post Matthew, it is just as thoughtful and detailed as your last one. I am excited to see more posts from you in future!
I have some thoughts and comments as someone with experience in this area. Apologies in advance if this comment ends up being long—I prefer to mirror the effort of the original post creator in my replies and you have set a very high bar!Risk Assessments – How should frontier AI organisations design their risk assessment procedures in order to sufficiently acknowledge – and prepare for – the breadth, severity and complexity of risks associated with developing frontier AI models?
This is a really great first area of focus, and if I may arrogantly share a self-plug, I recently posted something along this specific theme here. Clearly it has been field-changing, achieving a whopping 3 karma in the month since posting. I am truly a beacon of our field!
Jest aside, I agree this is an important area and one that is hugely neglected. A major issue is that academia is not good at understanding how this actually works in practice. Much more industry-academia partnership is needed but that can be difficult to arrange where it really counts—which is something you successfully allude to in your post.
Senior leadership of firms operate with limited information. Members of senior management of large companies themselves cannot know of everything that goes on in the firm. Therefore, strong communication channels and systems of oversight are needed to effectively manage risks.
This is a fantastic point, and one that is frequently a problem. Not long ago I was having a chat with the head of a major government organisation who quite confidently stated that his department did not use a specific type of AI system. I had the uncomfortable moral duty to inform that it did, because I had helped advise on risk mitigation for that system only some weeks earlier. It’s a fun story, but the higher up the chain you are in large organisations the harder it can be. Another good, recent example is also Nottinghamshire Police publicly claiming that they do not use and do not plan to use AFR in an FOI request—seemingly unaware their force revealed a new AFR tool to the media earlier that week.
Although much can be learned from practices in other industries, there are a number of unique challenges in implementing good corporate governance in AI firms. One such challenge is the immaturity of the field and technology. This makes it difficult currently to define standardised risk frameworks for the development and deployment of systems. It also means that many of the firms conducting cutting edge research are still relatively small; even the largest still in many ways operate with “start-up” cultures. These are cultures that are fantastic for innovation, but terrible for safety and careful action.
This is such a fantastic point, and to back this up it’s the source of I reckon about 75% of the risk scenarios I’ve advised on in the past year. Although I don’t think ‘AI firms’ is a good focus term because many major corporations are making AI as part of their coverage but are not themselves “AI Firms”, your point still stands well in the face of evidence because a major problem right now is AI startups selling immature, untested, ungoverned tools to major organisations who don’t know better and don’t know how to question what they’re buying. This isn’t just a problem with corporations but with government, too. It’s such a huge risk vector.
For Sections 2 and 3, engineering and energy are fantastic industries to draw from in terms of their processes for risk and incident reporting. They’re certainly amongst the strictest I’ve had experience of working alongside.Ethics committees take a key role in decision making that may have particularly large negative impacts on society. For frontier AI labs, such committees will have their work cut out for them. Work should be done to consider the full list of processes ethics committees should have input in, but it will likely include decisions around:
Model training, including
Appropriate data usage
The dangers of expected capabilities
Model deployments
Research approval
This is an area that’s seen a lot of really good outcomes in AI in high-risk industries. I would advise reading this research which covers a fantastic use-case in detail. There are also some really good ones in the process of getting the correct approvals which I’m not entirely sure I can post here yet but if you want kept updated shoot me an inbox and I’ll keep you informed.
The challenge for frontier AI firms by comparison is that many of the severe risks posed by AI are of a more esoteric nature, with much current uncertainty about how failure modes may present themselves. One potential area of study is the development of more general forms of risk awareness training, e.g. training for developing a “scout mindset” or to improve awareness of black swan events.
This is actually one of the few sections I disagree with you on. Of all the high-risk AI systems I’ve worked with in a governance capacity, exceptionally few have had esoteric risks. Many times AI systems interact with the world via existing processes which themselves are fairly risk scoped. Exceptions if you meant far-future AI systems which obviously would be currently unpredictable. For contemporary and near-future AI systems though the risk landscape is quite well explored.
7 – Open Research QuestionsThese are fantastic questions, and I’m glad to see some of these are covered by a recent grant application I made. Hopefully the grant decision-makers read these forums! I actually have something of a research group forming in this precise area, so feel free to drop me a message if there’s likely to be any overlap there and I’m happy to share research directions etc :)
There are huge technical research questions that must be answered to avoid tragedy, including important advancements in technical AI safety, evaluations and regulation. It is the author’s opinion that corporate governance should sit alongside these fields, with a few questions requiring particular priority and focus:
One final point of input that may be valuable is that in most of my experience of hiring people for risk management / compliance / governance roles in high-risk AI systems is the best candidates in the long run seem to be people with an interdisciplinary STEM and social studies background. It is tremendously hard to find these people. There needs to be much, much more effort put towards sharing of skills and knowledge between the socio-legal and STEM spheres, but a glance at my profile might show a bit of bias in this statement! Still, for these type of roles that kind of balance is important. I understand that many European universities now offer such interdisciplinary courses, but no degrees yet. Perhaps the winds will change.
Apologies if this comment was overly long! This is a very important area of AI governance and it was worth taking the time to put some thoughts on your fantastic post together. Looking forward to seeing your future posts—particularly in this area!
This was a really interesting read. Anecdotally, I’d put in that since having my two children my dedication to the future has increased dramatically. I genuinely care about, and I will butcher this saying, planting trees the shade of which I will never sit under.
Maybe I was a bad, amoral person and now I have a personal interest in making a better world. Maybe this works only for me. But knowing the world I help build will be the one my children have to live in has certainly made me choose better career paths and projects, as well as adding rocket fuel to my motivation.As always, mileage will vary significantly between people.
Hm. The closest things I can think of would either be things like inciting racial hatred or hate speech (ie not physical, no intent for crime, but illegal). In terms of research, most research isn’t illegal but is usually tightly regulated by participating stakeholders, ethics panels, and industry regulations. Lots of it is stakeholder management too. I removed some information from my PhD thesis at the request of a government stakeholder, even though I didn’t have to. But it was a good idea to ensure future participation and I could see the value in the reasoning. I’m not sure there was anything they could do legally if I had refused, as it wasn’t illegal per se.
The closest thing I can think of to your example is perhaps weapons research. There’s nothing specifically making weapons research illegal, but it would be an absolute quagmire in terms of not breaking the law. For example sharing the research could well fall under anti-terrorism legislation, and creating a prototype would obviously be illegal without the right permits. So realistically you could come up with a fantastic new idea for a weapon but you’d need to partner with a licensing authority very, very early on or risk doing all of your research by post at His Majesty’s pleasure for the next few decades.
I have in the past worked in some quite heavily regulated areas with AI, but always working with a stakeholder who had all the licenses etc so I’m not terribly sure how all that works behind the scenes.