JamesN

Karma: 195

Executive Director of the Swift Centre for Applied Forecasting (led projects with U.K. Gov., Google DeepMind, and on AI security and capability risks).

Co-founder of ‘Looking for Growth’ - a political movement for growth in the U.K.

CTO of Praxis—a AI led assessment platform for schools

Former Head of Policy at ControlAI (co-authored ‘A Narrow Path’)

Former Director of Impactful Government Careers

Former Head of Development Policy at HM Treasury

Former Head of Strategy at the Centre for Data Ethics and Innovation

Former Senior Policy Advisor at HM Treasury, leading on the economic and financial response to the war in Ukraine, and the modelling and allocation of the UK’s ‘Official Development Assistance’ budget.

MSc in Cognitive and Decision Sciences from UCL, my dissertation was an experimental study using Bayesian reasoning to improve predictive reasoning and forecasting in U.K. public policy officials and analysts

JamesN 6 Feb 2026 10:33 UTC
5 points
2 ∶ 0
on: JamesN’s Quick takes
Mental health support for those working on AI risks and policy?
During the numerous projects I work on relating to AI risks, policies, and future threats/scenarios, I speak to a lot of people who bring exposed to issues of catastrophic and existential nature for the first time (or grappling with them for the first time in detail). This combined with the likelihood that things will get worse before they better, makes me frequently wonder: are we doing enough around mental health support?
Things that I don’t know exist but feel they should. Some may sound OTT but I expect you could fund all of these for c.$300k, which relative to the amount being spent in the sector as a whole, is tiny in exchange for resilience of the talent we’re building.
- Structured proactive therapy or mental health resilience sessions tied into Fellowship programs.
- Regular in-built mental health support within organisations dealing with AI risks etc., particularly around helping to anti-catastrophise (there are several threat models which are only catastrophically bad if many, many, many things go down and happen counter to regular incentives—but are very worrying to people (especially new entrants to the field) - it feels support to help prioritise the risks would help both outcomes and mental health.
- Free to use services for mental health support for those in the field.

JamesN 28 Jan 2026 10:29 UTC
1 point
0 ∶ 0
on: JamesN’s Quick takes
I’ll be in D.C. on the 12th February and morning of the 13th, before heading to EAG. If people are around and want to meet, feel free to drop me a DM!

JamesN 27 Dec 2025 17:05 UTC
3 points
0 ∶ 0
in reply to: WillPearson’s comment on: WillPearson’s Quick takes
I think that’s quite a broad remit. What’s the focus of improving the decisions? Better problem identification/specification? Better data analysis and evidence base? Better predictive accuracy? Better efficiency/Adaptiveness/robustness?

JamesN 27 Dec 2025 8:12 UTC
3 points
1 ∶ 0
on: JamesN’s Quick takes
Much of the community’s focus is rightly on technical alignment and governance. However, there seems to be a significant blind spot regarding societal adaptation, specifically, how we raise and educate the next generation.
Our current education model is predicated on a learn skills to provide economic value loop. When transformative AI disrupts this model, we risk creating a generation that is not only economically displaced but fundamentally disenfranchised and without a clear sense of purpose. Historically, large populations of disenfranchised young people have been a primary driver of societal collapse and political volatility.
If the transition to a post-AGI world is chaotic due to human unrest, our ability to manage technical safety drops significantly. Is anyone seriously funding or working on how education/raising children needs to change to fit with an AGI era? It seems like ensuring the next generation is psychologically and philosophically prepared for a world of transformative AI is a necessary prerequisite for a stable transition.

JamesN 20 Jan 2025 11:25 UTC
5 points
1 ∶ 0
on: JamesN’s Quick takes
I’m interesting in chatting to any civil servants, ideally in the UK, who are keen on improving decision making in their teams/area—potentially through forecasting techniques and similar methods. If you’d be interested in chatting, please DM me!

JamesN 25 Nov 2024 22:46 UTC
1 point
1 ∶ 0
in reply to: Joseph’s comment on: Joseph Lemien’s Shortform
Is this just a combination of anchoring with confirmation bias? Or have I misunderstood.

JamesN 3 Oct 2024 10:37 UTC
1 point
2 ∶ 3
in reply to: Catherine Low🔸’s comment on: Making it easier for decision-makers to give feedback
Thank you for responding Catherine! It’s very much appreciated.

This should therefore be easily transferable into feedback to the grantee.

I think this is where we disagree—this written information often isn’t in a good shape to be shared with applicants and would need significant work before sharing.

I think this is my fundamental concern. Reasoning transparency and systematic processes to record grant maker’s judgments and show how they are updating their position should be intrinsic to how they are evaluating the applications. Otherwise they can’t have much confidence in the quality of their decisions or hope to learn from what judgment errors they make when determining which grants to fund (as they have no clear way to track back why they made a grant and whether or not that was a predictor for its success/failure).

JamesN 1 Oct 2024 22:36 UTC
3 points
1 ∶ 0
in reply to: Ozzie Gooen’s comment on: Ozzie Gooen’s Shortform
I am not so aware of the assumption you make up front, and would agree with you that anyone making such an assumption is being naive. Not least because humans on average (and even supers under many conditions) are objectively inaccurate at forecasting—even if relatively good given we don’t have anything better yet.

I think the more interesting and important when it comes to AI forecasting and claiming they are “good”, is to look at the reasoning process that they undertaken to do that. How are they forming reference classes, how are they integrating specific information, how are they updating their posterior to form an accurate inference and likelihood of the event occurring? Right now, they can sort of do (1), but from my experience don’t do well at all at integration, updating, and making a probabilistic judgment. In fairness, humans often don’t either. But we do it more consistently than current AI.

For your post, this suggests to me that AI could be used to help base rate/reference class creation, and maybe loosely support integration.

JamesN 30 Sep 2024 17:10 UTC
14 points
2 ∶ 1
on: Making it easier for decision-makers to give feedback
Thank you for posting this, is it definitely nice to get a funders perspective on this!
From the other side (someone who has applied for grants and received little to no feedback on them), and having been involved in very large scale grant making through my governmental role, I fear your point (1) below is likely to be the greatest influence on grantmakers not providing feedback. Unfortunately, this I find (and did when I was a grantmaker and was prevented/unable to provide feedback) is often a cover for a lack of transparent and good reasoning practice in the grant decision process.
The vast majority of EAs are aware of reasoning transparency and good Bayesian reasoning practices. I’d hope, as I assume many members of the EA community do, that EA grant makers have a defined method to record the grantmakers judgments and what is updating their view of a grants potential impact and likelihood of success. Not least because this would allow them to identify errors and any systematic biases that grantmakers may have, and thus improve as necessary. This should therefore be easily transferable into feedback to the grantee.
The fact this isn’t done raises questions for me. Are there such systematic processes? If not, how do grantmakers have confidence in their decision making a priori? If there are such processes to record reasoning, why can’t they be summarised and provided for feedback?
The post you linked by Linch and the concern he raises that by being transparent about the reasons for not making a grant may risk applicants overupdating on the feedback seems unfounded/unevidenced. I also question how relevant given they weren’t funded anyway, so why would you be concerned they’d over update? If you don’t tell them they were a near miss and what changes may change your mind, then instead the risk is they either update randomly or the project is just completely canned—which feels worse for edge cases.

There might have been a lot of factors that led to a decision.

Sometimes there are multiple small reasons to be hesitant about a person doing a particular project – none of which are “deal breakers”. There is often so much uncertainty and decision-makers realistically have so little information such that the only option is to rely on small bits of information to update their view on the person/project. Each of these factors on their own might just be a small grain of sand on a scale, or be felt with low confidence, but together they might build up to tip the scale.

JamesN 11 Sep 2024 10:10 UTC
2 points
1 ∶ 0
on: Reconsidering the Celebration of Project Cancellations: Have We Updated Too Far?
I tend to agree with you, though would rather people were more on the “close early” side of the coin than the “hold out”. Simply because the sunk cost fallacy and confirmation bias in your own idea is incredibly strong and I see no compelling reason for how current funders in the EA space help counteract these (beyond maybe being aware of them more than the average funder).

In an ideal system the funders should be driving most of these decisions by requiring clear milestones and evaluation processes for who they fund. If the funder did this they would be able to identify predictive signals of success and help avoid early or late closures (e.g. “we see on average policy advocacy groups that have been successful have met fewer/more comparable milestones and recommend continued/stopping funding”). This can still allow the organisation to pitch for why they are outside of the average, but the funder should be in the best position to know what is signalling success and what isn’t.

Unfortunately I don’t see such a system and I fear the incentives aren’t aligned in the EA ecosystem to create it. The organisations getting funded enjoy the looser, less funder involved setup. And funders de-risk their reputational risk by not properly evaluating what is working and why, and they can continue funding projects they are personally interested in but have questionable causal impact chains. *noting I think EA GHD has much less of this issue mainly because funders anchor on GiveWell assessments which is to a large degree delivering the mechanism I outline above.

JamesN 9 Sep 2024 21:44 UTC
45 points
15 ∶ 0
on: AI forecasting bots incoming
The science underpinning this study is unfortunately incredibly limited. For instance, there isn’t even basic significance testing provided. Furthermore, the use of historic events to check forecasting accuracy, and the limited reporting of proper measures to prevent the model utilising knowledge not available to the human forecasters (with only a brief mention of the researchers providing the search link with pre-approved) is also very poor.

I’m all for AI tools improving decision making and have undertaken several side-projects myself on this. But studies like this should be highlighted for their lack of scientific standards and thus, we should be sceptical of how much we use them to update our judgments of how good AI should be at forecasting currently (which to me is quite low given they struggle to causally reason)

JamesN 14 Aug 2024 13:28 UTC
7 points
0 ∶ 0
on: JamesN’s Quick takes
What are some of the best (relatively) small podcasts on AI and/or policy that people would recommend? I know of all the big ones, but keen to see if there are any nascent ones that are worth sharing with others.

JamesN 5 Aug 2024 11:36 UTC
5 points
0 ∶ 0
on: Jacob Steinhardt’s forecasting course lecture notes
Thanks for posting this. Sounds like an exciting project.

There have been a number of experimental studies done on improving individual and group forecasting, and a couple of organisations currently provide training in a classroom/workshop setting. From my knowledge the framework being proposing is quite different to these (albeit I respect it can only explain it at a high level here). What’s the main evidence base guiding this approach and what’s the expected increase in accuracy attendees can expect the course to have?

JamesN 31 Jul 2024 19:43 UTC
3 points
2 ∶ 0
on: Thoughts on EA Epistemics Projects
I think this is a really important area, and it is great someone is thinking whether EAIF could expand more into this sort of area.
To provide some thoughts of my own having explored working with a few EA entities through our consultancy (https://www.daymark-di.com/) that works to help organisations improve their decision making processes, along with discussions I’ve had with others on similar endeavours:
1. Funding is a key bottleneck, which isn’t surpising. I think there is naturally an aversion to consultancy-type support in EA organisations, mostly driven by lack of funds to pay for it and partly because I think they are concerned how it’ll look in progress reports if they spend money on consultants.
  1. EAIF funding could make this easier, as it’ll remove the entire (or a large part) of the cost above.
2. There does appear a fairly frequent assumption that EA organisations suffer less from poor epistemics and decision making practices, which my experience suggests is somewhat true but unfortunately not entirely. I want to repeat what Jona from cFactual commented below that there are lots of actions EA organisations take that are very positive, such as BOTECs, likely failure models, and using decision matrices. This should be praised as many organisations don’t even do these. However, the simple existence of these is too often assumed to mean good analysis and judgment will naturally occur and the systems/processes that are needed to make them useful are often lacking. To be more concrete, as an example few BOTECs incorporate second order probability/confidence correctly (or it is conflated with first order probability) and they subsequently fail to properly account for the uncertainty of the calculation and the accurate comparisons that can be made between options.
  It has been surprising to observe the difference between some EA organisations and non-EA institutions when it comes to interest in improving their epistemics/decision making. With large institutions (including governmental) being more receptive and proactive in trying to improve—often those institutions are mostly being constrained by their slow procurement processes as opposed appetite.
When it comes to future projects, my recommendations of those with highest value add would be:
1. Projects that help organisations improve their accountability, incentive, and prioritisation mechanisms. In particular helping to identify and implement internal processes that properly link workstreams/actions/decisions/judgments to the goals of the organisation and that role’s objectives. This is something which is most useful to larger organisations (and especially those that make funding/granting decisions or recommendations), but can also be something which smaller organisations would benefit from.
2. Projects that help organisations improve their processes to more accurately and efficiently reason under uncertainty, namely assisting with defining the prediction/diagnostic they are trying to make a judgment on, identify the causal chain and predictors (and their relative importance) that underpins their judgment, and provide them with a framework to communicate their uncertainty and reasoning in a transparent and consistent way.
3. Projects that provide external scrutiny and advanced decision modelling capabilities. I think at a low level there are some relatively easy wins that can be gained from having an entity/service which red teams and provides external assessment of big decisions that organisations are looking at making (e.g. new requests for new funding). At a greater level there should be more advanced modelling (including using tools such as Bayesian models) which can provide transparent and updatable analysis which can expose conditional relationships that we can’t properly account for in our minds.
  1. I think funding entities like EA Funds could utilise such an entity/entities to inform their grant making decisions (e.g. if such analysis was instrumental in the decision EA Funds made on a grant request, they’d pay half of the cost of the analysis).

JamesN 5 Jun 2024 16:05 UTC
11 points
1 ∶ 0
on: JamesN’s Quick takes
I’m considering my future donation options, either directly to charities or through a fund. I know EA Funds is still somewhat cash constrained but I’m also a little concerned with the natural variance in grant quality.

I’d be interested in why others have or have not chosen to donate to EA Funds, and if so, would they again in the future?

I respect people may prefer to answer this by DM, so please do feel free to drop me a message there if posting here feels uncomfortable.

JamesN 5 Jun 2024 14:29 UTC
5 points
0 ∶ 0
on: I bet Greg Colbourn 10 k€ that AI will not kill us all by the end of 2027
There seems to be quite a few people who are keen to take up the bet against extinction in 2027… are there many who would be willing to take the opposite bet (on equal terms (i.e. bet + inflation) as opposed to the 200% loss that Greg is on the hook for)?
Do people also mind where their bet goes? In this case I see the money went to PauseAI, would people be willing to make the bet if the money went to that person for them to just spend as they want? I could see someone who believed p(doom) by 2027 was 90%+ might just want more money to go on holiday before the end if they doubt any intervention will succeed. This is obviously hypothetical for interest sake as a real trade would need some sort of guarantee the money would be repaid etc. etc.

JamesN 5 Jun 2024 12:46 UTC
3 points
2 ∶ 1
on: Pay to get AI safety info from behind NDA wall?
My prior on this is that the opportunity cost of what the money could’ve been spent on is excessively high and there are much better uses it could go towards.
Obviously the more novel and actionable the information is, the lower that trade-off becomes. However, I expect that most of the time the information they can provide would be valued too high due to personal intrigue as opposed to it actually being anything that meaningfully moves the dial. Equally, if the information they have was especially ground-breaking then I’d hope the person under the NDA would sacrifice personal wealth to expose that, then retrospectively they may get support with any legal costs etc. A reactive system as opposed proactive would also help prevent weird incentives being created where people would try to hold out on blowing the whistle until they had funds confirmed.

JamesN 25 May 2024 10:43 UTC
1 point
0 ∶ 0
on: Project idea: AI for epistemics
I’m a strong believer that AI can be of massive assistance in this area, especially in areas such as improving forecasting ability where the science is fairly well understood/evidenced (I.e. improve reasoning process to improve forecasting and prediction).

My point of caution would be that exploration here, if not done with sufficient scientific rigour, can result in superficially useful tools that add to AI misuse risks and/or worse decision making. For more info do see the below research paper: https://arxiv.org/abs/2402.01743#:~:text=This report examines a novel,many critical real world problems.

JamesN 25 May 2024 10:23 UTC
3 points
1 ∶ 0
in reply to: Habryka [Deactivated]’s comment on: Project idea: AI for epistemics
The bottleneck on funding is the biggest issue in unlocking the potential in this space imo, and on trying to improve decision making more broadly.

I find the void between the importance placed on improving reasoning and decision making on cause priority researchers such as 80k and the EA community as a whole, and the appetite from funders to invest is quite huge. A colleague and I have struggled for even relatively small amounts of funding, even when having existing users/clients from which funding would allow us to scale from.

That’s not a complaint—funders are free to determine what they want to fund. But it seems a consistent challenge people who want to improve IIDM face.

It increases my view that such endeavours, though incredibly important, should be focused on earning a profit as the ability to scale as a non-profit will be limited.

JamesN 11 Jan 2024 13:46 UTC
6 points
1 ∶ 0
on: JamesN’s Quick takes
As my toddler continues to grow, my wife and I are reaching the point that all parents do (if not already done) where we have to decide what to do about his education. Obviously education and life aren’t fixed so there is always a necessary amount of flexibility to any decision.
My issue (UK focus): Broadly there are three stages in the UK − 1) nursery, 2) primary school, 3) secondary/high school. With the world becoming increasingly uncertain and complex, I have very little confidence that any current institution (in stage 2 and 3) is set up and run well enough to prepare a child for the future—beyond the basic reading and writing skills. As a small example for instance, there is a shocking lack of investment in both resources and capability on key topics such as computer science, maths, engineering, rationality, and political science. However, given this is a quick post I won’t go into the depth of my reasoning of why I feel there is a structural problem with the education system that would be almost impractical (or at least, too politically costly) to resolve. To add some potential credence, this isn’t just wandering thoughts but they have been generated through the experiences of multiple family (including my wife) and friends who work/worked in primary or secondary schools.
My question: What are others’ thoughts on this (both parents and non-parents)? Do you have similar concerns? Do you not? If you do, what are your intended actions to mitigate?