JamesN

Karma: 162

I have particular expertise in:
- Developing and implementing policy
- Improving decision making within organisations, mainly focused on improving people’s reasoning process (e.g. predictive/forecasting) that underpins how they make and communicate judgments.
- AI Safety policy

This has been achieved through being:

1) Director of Daymark Decision Insights, a company which provides consultative services and tailor made workshops related to improving decision making and reasoning processes to high-impact organisations (https://www.daymark-di.com/). More recently I’ve provided specific consultative services on policy development and advocacy to a large-scale AI safety organisation.

2) Director of Impactful Government Careers—an organisation focused on helping individuals find, secure, and excel in high impact civil service roles.

3) I spent 5 years working in the heart of the UK Government, with 4 of those at HM Treasury. With roles including:

- Head of Development Policy, HM Treasury
- Head of Strategy, Centre for Data Ethics and Innovation
- Senior Policy Advisor, Strategy and Spending for Official Development Assistance, HM Treasury

These roles have involved: advising UK Ministers on policy, spending, and strategy issues relating to international development; assessing the value for money of proposed high-value development projects; developing the 2021 CDEI Strategy and leading the organisational change related to this.

4) I’ve completed an MSc in Cognitive and Decision Sciences at UCL, where I have focused my research on probabilistic reasoning and improving individual and group decision-making processes. My final research project involved an experimental study into whether a short course (2 hours) on Bayesian reasoning could improve individual’s single-shot accuracy when forecasting geopolitical events.

JamesN 11 Sep 2024 10:10 UTC
2 points
1 ∶ 0
on: Reconsidering the Celebration of Project Cancellations: Have We Updated Too Far?
I tend to agree with you, though would rather people were more on the “close early” side of the coin than the “hold out”. Simply because the sunk cost fallacy and confirmation bias in your own idea is incredibly strong and I see no compelling reason for how current funders in the EA space help counteract these (beyond maybe being aware of them more than the average funder).

In an ideal system the funders should be driving most of these decisions by requiring clear milestones and evaluation processes for who they fund. If the funder did this they would be able to identify predictive signals of success and help avoid early or late closures (e.g. “we see on average policy advocacy groups that have been successful have met fewer/more comparable milestones and recommend continued/stopping funding”). This can still allow the organisation to pitch for why they are outside of the average, but the funder should be in the best position to know what is signalling success and what isn’t.

Unfortunately I don’t see such a system and I fear the incentives aren’t aligned in the EA ecosystem to create it. The organisations getting funded enjoy the looser, less funder involved setup. And funders de-risk their reputational risk by not properly evaluating what is working and why, and they can continue funding projects they are personally interested in but have questionable causal impact chains. *noting I think EA GHD has much less of this issue mainly because funders anchor on GiveWell assessments which is to a large degree delivering the mechanism I outline above.

JamesN 9 Sep 2024 21:44 UTC
43 points
14 ∶ 0
on: AI forecasting bots incoming
The science underpinning this study is unfortunately incredibly limited. For instance, there isn’t even basic significance testing provided. Furthermore, the use of historic events to check forecasting accuracy, and the limited reporting of proper measures to prevent the model utilising knowledge not available to the human forecasters (with only a brief mention of the researchers providing the search link with pre-approved) is also very poor.

I’m all for AI tools improving decision making and have undertaken several side-projects myself on this. But studies like this should be highlighted for their lack of scientific standards and thus, we should be sceptical of how much we use them to update our judgments of how good AI should be at forecasting currently (which to me is quite low given they struggle to causally reason)

JamesN 14 Aug 2024 13:28 UTC
7 points
0 ∶ 0
on: JamesN’s Quick takes
What are some of the best (relatively) small podcasts on AI and/or policy that people would recommend? I know of all the big ones, but keen to see if there are any nascent ones that are worth sharing with others.

JamesN 5 Aug 2024 11:36 UTC
3 points
0 ∶ 0
on: Jacob Steinhardt’s forecasting course lecture notes
Thanks for posting this. Sounds like an exciting project.

There have been a number of experimental studies done on improving individual and group forecasting, and a couple of organisations currently provide training in a classroom/workshop setting. From my knowledge the framework being proposing is quite different to these (albeit I respect it can only explain it at a high level here). What’s the main evidence base guiding this approach and what’s the expected increase in accuracy attendees can expect the course to have?

JamesN 31 Jul 2024 19:43 UTC
3 points
2 ∶ 0
on: Thoughts on EA Epistemics Projects
I think this is a really important area, and it is great someone is thinking whether EAIF could expand more into this sort of area.
To provide some thoughts of my own having explored working with a few EA entities through our consultancy (https://www.daymark-di.com/) that works to help organisations improve their decision making processes, along with discussions I’ve had with others on similar endeavours:
1. Funding is a key bottleneck, which isn’t surpising. I think there is naturally an aversion to consultancy-type support in EA organisations, mostly driven by lack of funds to pay for it and partly because I think they are concerned how it’ll look in progress reports if they spend money on consultants.
  1. EAIF funding could make this easier, as it’ll remove the entire (or a large part) of the cost above.
2. There does appear a fairly frequent assumption that EA organisations suffer less from poor epistemics and decision making practices, which my experience suggests is somewhat true but unfortunately not entirely. I want to repeat what Jona from cFactual commented below that there are lots of actions EA organisations take that are very positive, such as BOTECs, likely failure models, and using decision matrices. This should be praised as many organisations don’t even do these. However, the simple existence of these is too often assumed to mean good analysis and judgment will naturally occur and the systems/processes that are needed to make them useful are often lacking. To be more concrete, as an example few BOTECs incorporate second order probability/confidence correctly (or it is conflated with first order probability) and they subsequently fail to properly account for the uncertainty of the calculation and the accurate comparisons that can be made between options.
  It has been surprising to observe the difference between some EA organisations and non-EA institutions when it comes to interest in improving their epistemics/decision making. With large institutions (including governmental) being more receptive and proactive in trying to improve—often those institutions are mostly being constrained by their slow procurement processes as opposed appetite.
When it comes to future projects, my recommendations of those with highest value add would be:
1. Projects that help organisations improve their accountability, incentive, and prioritisation mechanisms. In particular helping to identify and implement internal processes that properly link workstreams/actions/decisions/judgments to the goals of the organisation and that role’s objectives. This is something which is most useful to larger organisations (and especially those that make funding/granting decisions or recommendations), but can also be something which smaller organisations would benefit from.
2. Projects that help organisations improve their processes to more accurately and efficiently reason under uncertainty, namely assisting with defining the prediction/diagnostic they are trying to make a judgment on, identify the causal chain and predictors (and their relative importance) that underpins their judgment, and provide them with a framework to communicate their uncertainty and reasoning in a transparent and consistent way.
3. Projects that provide external scrutiny and advanced decision modelling capabilities. I think at a low level there are some relatively easy wins that can be gained from having an entity/service which red teams and provides external assessment of big decisions that organisations are looking at making (e.g. new requests for new funding). At a greater level there should be more advanced modelling (including using tools such as Bayesian models) which can provide transparent and updatable analysis which can expose conditional relationships that we can’t properly account for in our minds.
  1. I think funding entities like EA Funds could utilise such an entity/entities to inform their grant making decisions (e.g. if such analysis was instrumental in the decision EA Funds made on a grant request, they’d pay half of the cost of the analysis).

JamesN 5 Jun 2024 16:05 UTC
11 points
1 ∶ 0
on: JamesN’s Quick takes
I’m considering my future donation options, either directly to charities or through a fund. I know EA Funds is still somewhat cash constrained but I’m also a little concerned with the natural variance in grant quality.

I’d be interested in why others have or have not chosen to donate to EA Funds, and if so, would they again in the future?

I respect people may prefer to answer this by DM, so please do feel free to drop me a message there if posting here feels uncomfortable.

JamesN 5 Jun 2024 14:29 UTC
5 points
0 ∶ 0
on: I bet Greg Colbourn 10 k€ that AI will not kill us all by the end of 2027
There seems to be quite a few people who are keen to take up the bet against extinction in 2027… are there many who would be willing to take the opposite bet (on equal terms (i.e. bet + inflation) as opposed to the 200% loss that Greg is on the hook for)?
Do people also mind where their bet goes? In this case I see the money went to PauseAI, would people be willing to make the bet if the money went to that person for them to just spend as they want? I could see someone who believed p(doom) by 2027 was 90%+ might just want more money to go on holiday before the end if they doubt any intervention will succeed. This is obviously hypothetical for interest sake as a real trade would need some sort of guarantee the money would be repaid etc. etc.

JamesN 5 Jun 2024 12:46 UTC
3 points
2 ∶ 1
on: Pay to get AI safety info from behind NDA wall?
My prior on this is that the opportunity cost of what the money could’ve been spent on is excessively high and there are much better uses it could go towards.
Obviously the more novel and actionable the information is, the lower that trade-off becomes. However, I expect that most of the time the information they can provide would be valued too high due to personal intrigue as opposed to it actually being anything that meaningfully moves the dial. Equally, if the information they have was especially ground-breaking then I’d hope the person under the NDA would sacrifice personal wealth to expose that, then retrospectively they may get support with any legal costs etc. A reactive system as opposed proactive would also help prevent weird incentives being created where people would try to hold out on blowing the whistle until they had funds confirmed.

JamesN 25 May 2024 10:43 UTC
1 point
0 ∶ 0
on: Project idea: AI for epistemics
I’m a strong believer that AI can be of massive assistance in this area, especially in areas such as improving forecasting ability where the science is fairly well understood/evidenced (I.e. improve reasoning process to improve forecasting and prediction).

My point of caution would be that exploration here, if not done with sufficient scientific rigour, can result in superficially useful tools that add to AI misuse risks and/or worse decision making. For more info do see the below research paper: https://arxiv.org/abs/2402.01743#:~:text=This report examines a novel,many critical real world problems.

JamesN 25 May 2024 10:23 UTC
3 points
1 ∶ 0
in reply to: Habryka’s comment on: Project idea: AI for epistemics
The bottleneck on funding is the biggest issue in unlocking the potential in this space imo, and on trying to improve decision making more broadly.

I find the void between the importance placed on improving reasoning and decision making on cause priority researchers such as 80k and the EA community as a whole, and the appetite from funders to invest is quite huge. A colleague and I have struggled for even relatively small amounts of funding, even when having existing users/clients from which funding would allow us to scale from.

That’s not a complaint—funders are free to determine what they want to fund. But it seems a consistent challenge people who want to improve IIDM face.

It increases my view that such endeavours, though incredibly important, should be focused on earning a profit as the ability to scale as a non-profit will be limited.

JamesN 11 Jan 2024 13:46 UTC
6 points
1 ∶ 0
on: JamesN’s Quick takes
As my toddler continues to grow, my wife and I are reaching the point that all parents do (if not already done) where we have to decide what to do about his education. Obviously education and life aren’t fixed so there is always a necessary amount of flexibility to any decision.
My issue (UK focus): Broadly there are three stages in the UK − 1) nursery, 2) primary school, 3) secondary/high school. With the world becoming increasingly uncertain and complex, I have very little confidence that any current institution (in stage 2 and 3) is set up and run well enough to prepare a child for the future—beyond the basic reading and writing skills. As a small example for instance, there is a shocking lack of investment in both resources and capability on key topics such as computer science, maths, engineering, rationality, and political science. However, given this is a quick post I won’t go into the depth of my reasoning of why I feel there is a structural problem with the education system that would be almost impractical (or at least, too politically costly) to resolve. To add some potential credence, this isn’t just wandering thoughts but they have been generated through the experiences of multiple family (including my wife) and friends who work/worked in primary or secondary schools.
My question: What are others’ thoughts on this (both parents and non-parents)? Do you have similar concerns? Do you not? If you do, what are your intended actions to mitigate?

JamesN 13 Dec 2023 8:54 UTC
2 points
0 ∶ 0
on: New Meta-Analysis: Exercise is a Highly Effective (But Neglected) Treatment for Depression
Nice post, and I like the short snappy sections for quick reading—thanks!
My main reflection is whether this is actually a neglected? Exercise is a huge industry, and I feel the psychological benefits as well as the physical are quite often utilised in marketing and advertising (though I may be biased given I go to the gym quite regularly). Perhaps the specific link to depression is leveraged less, but I feel its there by proxy in a lot of cases.
I guess a fairer question would be: what would need to exist, that doesn’t currently, for this to be assessed as not a neglected issue?

JamesN 13 Dec 2023 8:42 UTC
1 point
0 ∶ 0
in reply to: Karthik Tadepalli’s comment on: Karthik Tadepalli’s Quick takes
This is an important reflection, and one I’ve found myself querying when seeing various programs claim they are hyper effective. Incredibly well performing interventions are rare, but we might expect to see a higher number of them to be showcased on this forum given there is already a selection bias from the membership/readership here.
However, I do feel the community naturally creates an incentive to inflate (conciously or not) the CEA of interventions—afterall, if you aren’t working on something which can compete with AMF, then why take money away from that? The fix to this being you live in the ambiguity of your intervention and argue that under certain assumptions, your program could be better.
As you effectively note, the problem is could (a priori) judgments are riddled with reasoning risks and errors, which is why I feel the community could do more to better support and also challenge reasoning methods (cognitive and computational). For example, lots of posts mention key uncertainties people have on their interventions, but they often don’t state the second order probabilities of them (not even GiveWell does this consistently) along with how much that uncertainty fundamentally underpins the intervention. A relatively simple fix, which could be a community norm.

JamesN 29 Nov 2023 19:11 UTC
1 point
1 ∶ 0
in reply to: JordanStone’s comment on: Effective Selfishism?
I agree—though not so much on everything gets funded anyway point.
I think there is also a wider meta question which is what is the best use of EA’s marginal time/energy/money. My (highly unjustified) judgement would be that people donating for such causes aren’t motivated by effectiveness, or at least are motivated much more by emotion. So the likelihood of changing their donation based on an argument around effectiveness may be quite hard to achieve.
I’m also not sure on the scale of difference between the worst and best charities for such causes (i.e. is the best cancer charity 100x better than the worst)? It’d be great to know, but assuming not, this would also reduce the benefit of any success.
A more effective solution achieve the same goal by proxy would seem to be just influencing the existing major funds or initiatives to focus more on the marginal impact of every £ they receive.

JamesN 29 Nov 2023 12:54 UTC
8 points
0 ∶ 0
on: The Odyssean Process
Thanks for posting this!
I want to highlight up front that I am a big supporter of any work that aims to improve institutional decision making. I believe it’s a highly impactful area with unparalleled potential given the decision power (in both terms of spending and benefit potential) of large institutions is immense. I personally feel there’s a big moral and EA argument in supporting solutions that could practically deliver benefits (even small returns given the scale).
In terms of cleaner questions upfront which get to the heart of my uncertainties:
1. How much will the elements combined improve the quality of decisions that are made? Tied to this—which elements could be cut if needed for time without undermining the benefits you’d expect?
2. Are there examples of previous decisions that have been made that have been run through this process, to show what different outcomes would have been generated?
3. Given the time investment needed to implement this process, why is it advantageous over existing solutions that have been shown to provide substantial improvements in decision making quality (under experimentation) but often face complaints over needing significant time and expertise investment (e.g. training on and aggregating Bayesian models)?
Further reflections if interested/useful
Having read your paper, I have some concerns over how the solution can be implemented at a beneficial scale. I raise this particularly as a number of the problems you’ve mentioned in the White Paper (e.g. unstructured/limited consultations with experts or limited analysis of the problem space) are driven more by time constraints rather than a clear framework of how to do it. This is an important consideration as planning for catastrophic risks is only half of the problem—we can’t consistently (or at all) predict black swan events and thus decision making at speed in crises is incredibly (if not more) important, as Covid showed us.
Given decision science research, I query the heavy reliance on expert judgment as a key node to improve the predictive accuracy, as there’s a healthy body of evidence that suggests quality of reasoning as opposed to domain expertise is a better predictor for such accuracy. Your White Paper actually seems to account for this by proxy when it highlights specific reasoning methods to drive improved accuracy (e.g. IDEA framework).
In addition, I’m less sure how beneficial the democratic/deliberation process with citizens is for the risks you are targeting. The examples you note (such as abortion and LGBTQ+ issues) are primarily social issues which lend themselves well to citizen assemblies given they are moral in nature. On the other side, planning policy is quite heavily democratised in the UK and arguably has led to very bad outcomes given wider economic or societal benefits from construction are less tangible than personal concerns around changes to the local area. These externalities aren’t always accurately priced into people’s incentives and thus their judgements aren’t necessarily what’s best for society. Do you see a similar issue for catastrophic risks/how will you mitigate if so?

[Question] How much do you spend on M&E?

JamesN19 Nov 2023 14:33 UTC

2 points

0 comments1 min readEA link

JamesN 16 Nov 2023 15:11 UTC
6 points
0 ∶ 0
on: JamesN’s Quick takes
With a number of charity evaluators recommendations coming out over the last few days/weeks, has there been any further development on AI safety/GCR evaluator(s)? A need that was raised in the post below (I don’t know if best EA forum practice is to ressurrect an old thread or not, so I apologies if it’s better I just comment in there).
https://forum.effectivealtruism.org/posts/QahDdRzpPuat8ujx2/why-isn-t-there-a-charity-evaluator-for-longtermist-projects

JamesN 15 Nov 2023 15:44 UTC
4 points
0 ∶ 0
on: HearMeOut—Networking While Funding Charities (Looking for a founder and beta users)
A unique and interesting concept.

I can see the logic but I’d suspect the payments for meetings would need to be quite low in most instances as I wonder whether you create weird incentives where time strapped high-in-demand people slap high fees on meetings and people who have limited funds have yet another barrier to meeting such people. Could also make some very moral conscious people maximise the charge they put on the meeting slot given the time:cost benefit is now not just their own time, but also the opportunity cost of what someone else was willing to pay.

How do you see mitigating the risk that it’ll no longer just be your idea that gets you through the door, but the depth of your pockets?

Separately, am I correct that there is no funds to create the MVP? It feels like something which a few thousand £/$’s could build from a reputable programmer? Would that be useful and possible to get the concept off the ground and able to be tested?

JamesN 13 Nov 2023 16:23 UTC
2 points
0 ∶ 0
in reply to: Yarrow B.’s comment on: JamesN’s Quick takes
Very fair identification of some sloppy wording by me there with “increasing”. Apologies, my main focus was on the relatively high risk. Though as you’ve noted the WSJ survey had a median probability of 50%, and Forbes (link below) notes the NY Fed recession probability indicator is at 56% (albeit decreased from its previous 66%) - an unusually accurate indicator. Assessments which should be making central banks and economic ministries a little nervous.
Even with some forecasters reducing their probabilities, the relative risk and high level of uncertainties underpinning them would suggest to me it’d be a good time to review plans to weather any downturn.
https://www.forbes.com/advisor/investing/is-a-recession-coming/

JamesN 9 Nov 2023 9:37 UTC
10 points
1 ∶ 0
on: JamesN’s Quick takes
Do EA funding organisations have contingency plans (I use the term loosely) to manage downside risks to cash-flow in the eventuality of a recession? With the possibility of a recession in the coming 6-18 months seemingly increasing, it would be prudent to model the scenarios and think through how to smooth the funding profile if needed. Given cash is already constrained post the collapse of “that will shall not be named”, it feels a recession (and thus implications to donations/value of investments etc.) could compound existing funding issues. I don’t expect to get a financial plan from any of the big funders, but I thought worth raising the question in the low probability scenario it isn’t already part of their modelling.