RSS

Mo­ral uncertainty

TagLast edit: Aug 2, 2023, 9:04 AM by Leo

Moral uncertainty is uncertainty about how to act given lack of certainty in any one moral theory, as well as the study of how we ought to act given this uncertainty.

We are sometimes uncertain about empirical facts, such as whether it will rain tomorrow. But we can also be uncertain about moral facts, such as whether it’s wrong to steal, or how we should value the well-being of animals. Uncertainty about whether it’s wrong to steal is uncertainty about moral or normative facts, while uncertainty about how we should value the well-being of animals is uncertainty about axiological or value facts.

Can moral uncertainty be rational, and what should we do in response to it? We might think that it can never be rational to be uncertain about normative or axiological facts, because such facts are, like mathematical facts, knowable a priori. Nevertheless it seems that agents like ourselves are uncertain about non-trivial mathematical facts, and that we are also uncertain about normative and axiological facts. Given this, it seems necessary to develop some account of how we ought to act under moral uncertainty.

Several such accounts have been formulated in recent years:

Two important problems facing the view that moral uncertainty can affect how we ought to act are the regress problem—the problem that we will be uncertain not only about typical moral questions, but also about which approach to moral uncertainty is correct, and so on ad infinitum[5] - and the problem of intertheoretic comparisons—the problem that there seems to be no principled ways to make comparisons between different moral theories.[6]

Further reading

MacAskill, William, Krister Bykvist, & Toby Ord (2020) Moral Uncertainty, Oxford: Oxford University Press.

Related entries

altruistic wager | decision-theoretic uncertainty | ethics of existential risk | expected value | fanaticism | moral trade | normative uncertainty

  1. ^

    Weatherson, Brian (2014) Running risks morally, Philosophical Studies, 167: 141–63.

  2. ^

    Gustafsson, Johan E. & Olle Torpman (2014) In defence of my favourite theory, Pacific Philosophical Quarterly, 95(2): 159-174.

  3. ^

    MacAskill, William (2014) Normative uncertainty, Doctoral dissertation, University of Oxford.

  4. ^

    Bostrom, Nick (2009) Moral uncertainty—towards a solution?, Overcoming Bias, January 1.

  5. ^
  6. ^

    Nissan-Rozen, Ittay (2015) Against moral hedging, Economics and Philosophy, 31(3): 349-369.

New book: Mo­ral Uncer­tainty by MacAskill, Ord & Bykvist

frankieawSep 10, 2020, 4:22 PM
91 points
2 comments1 min readEA link

Wheel­ing and deal­ing: An in­ter­nal bar­gain­ing ap­proach to moral uncertainty

MichaelPlantJul 10, 2022, 12:04 PM
56 points
18 comments23 min readEA link

A dilemma for Max­i­mize Ex­pected Choice­wor­thi­ness (MEC)

Calvin_BakerSep 1, 2022, 2:46 PM
33 points
7 comments10 min readEA link

Mo­ral Uncer­tainty and Mo­ral Real­ism Are in Tension

Lukas_GloorJan 25, 2022, 11:07 AM
37 points
4 comments12 min readEA link

Re­solv­ing moral un­cer­tainty with randomization

Bob JacobsMar 29, 2023, 10:10 AM
29 points
3 comments10 min readEA link

Mak­ing de­ci­sions un­der moral uncertainty

MichaelA🔸Jan 1, 2020, 1:02 PM
44 points
8 comments17 min readEA link

The prop­erty rights ap­proach to moral uncertainty

Harry R. LloydOct 14, 2022, 4:49 PM
31 points
14 comments2 min readEA link
(www.happierlivesinstitute.org)

A philo­soph­i­cal re­view of Open Philan­thropy’s Cause Pri­ori­ti­sa­tion Framework

MichaelPlantJul 15, 2022, 8:21 AM
160 points
11 comments29 min readEA link
(www.happierlivesinstitute.org)

Hedg­ing against deep and moral uncertainty

MichaelStJulesSep 12, 2020, 11:44 PM
83 points
16 comments9 min readEA link

The moral am­bi­guity of fish­ing on wild aquatic an­i­mal populations

MichaelStJulesMay 6, 2024, 11:03 PM
76 points
2 comments4 min readEA link

A longter­mist cri­tique of “The ex­pected value of ex­tinc­tion risk re­duc­tion is pos­i­tive”

Anthony DiGiovanniJul 1, 2021, 9:01 PM
145 points
10 comments32 min readEA link

Mo­ral Anti-Real­ism: In­tro­duc­tion & Summary

Lukas_GloorApr 2, 2022, 2:32 PM
52 points
1 comment7 min readEA link

The “Mo­ral Uncer­tainty” Rab­bit Hole, Fully Excavated

Lukas_GloorApr 1, 2022, 1:14 PM
23 points
2 comments34 min readEA link

Cause pri­ori­ti­za­tion for down­side-fo­cused value systems

Lukas_GloorJan 31, 2018, 2:47 PM
76 points
11 comments48 min readEA link

If Con­trac­tu­al­ism, Then AMF

Bob FischerOct 13, 2023, 6:03 PM
62 points
54 comments24 min readEA link

A Case for Vol­un­tary Abor­tion Reduction

Ariel Simnegar 🔸Dec 20, 2022, 1:23 PM
39 points
150 comments13 min readEA link

The Pos­si­bil­ity of an On­go­ing Mo­ral Catas­tro­phe (Sum­mary)

LinchAug 2, 2019, 9:55 PM
148 points
17 comments4 min readEA link
(docs.google.com)

Wor­ld­view uncertainty

Ronja_LutzMay 16, 2018, 2:02 PM
15 points
5 comments8 min readEA link

Ar­gu­ments for moral indefinability

richard_ngoFeb 8, 2019, 11:09 AM
34 points
8 comments7 min readEA link
(thinkingcomplete.blogspot.com)

The op­tion value ar­gu­ment doesn’t work when it’s most needed

WinstonOct 24, 2023, 7:40 PM
131 points
6 comments6 min readEA link

[Question] How to es­ti­mate the prob­a­bil­ity of moral as­ser­tions?

lamparitaFeb 10, 2025, 2:36 PM
12 points
3 comments1 min readEA link

Ex­pected value un­der nor­ma­tive uncertainty

Gordon Seidoh WorleyJun 8, 2020, 3:45 PM
16 points
1 comment1 min readEA link
(philpapers.org)

Wrong by Induction

Ales_FlidrSep 6, 2018, 10:00 PM
20 points
2 comments6 min readEA link

[Pod­cast] Ajeya Co­tra on wor­ld­view di­ver­sifi­ca­tion and how big the fu­ture could be

Eevee🔹Jan 22, 2021, 11:57 PM
57 points
20 comments1 min readEA link
(80000hours.org)

Up­date on the Global Pri­ori­ties In­sti­tute’s (GPI) activities

Global Priorities InstituteDec 24, 2019, 1:51 PM
23 points
0 comments7 min readEA link

Why I Don’t Ac­count for Mo­ral Uncertainty

[deleted]Jan 11, 2015, 3:52 AM
25 points
16 comments2 min readEA link

Peter Singer on the 80K podcast

Jon_BeharDec 18, 2019, 5:17 PM
31 points
0 comments1 min readEA link
(80000hours.org)

Will MacAskill on nor­ma­tive uncertainty

lukeprogApr 9, 2014, 3:27 PM
2 points
0 comments8 min readEA link

Mo­ral plu­ral­ism and longter­mism | Sunyshore

Eevee🔹Apr 17, 2021, 12:14 AM
26 points
0 comments5 min readEA link
(sunyshore.substack.com)

Deon­tol­ogy and virtue ethics as “effec­tive the­o­ries” of con­se­quen­tial­ist ethics

Jan_KulveitNov 17, 2022, 9:20 AM
55 points
12 comments10 min readEA link

Amanda Askell: The moral value of information

EA GlobalJun 2, 2017, 8:48 AM
19 points
0 comments13 min readEA link
(www.youtube.com)

Shar­ing the World with Digi­tal Minds

Aaron Gertler 🔸Dec 1, 2020, 8:00 AM
12 points
1 comment1 min readEA link
(www.nickbostrom.com)

Ideal gov­er­nance (for com­pa­nies, coun­tries and more)

Holden KarnofskyApr 7, 2022, 4:54 PM
80 points
19 comments14 min readEA link

Me­taeth­i­cal Fa­nat­i­cism (Dialogue)

Lukas_GloorJun 17, 2020, 12:33 PM
35 points
10 comments15 min readEA link

Quotes about the long reflection

MichaelA🔸Mar 5, 2020, 7:48 AM
55 points
14 comments13 min readEA link

The ex­pected value of ex­tinc­tion risk re­duc­tion is positive

JanBDec 9, 2018, 8:00 AM
66 points
22 comments61 min readEA link

Chris­tian Tarsney on fu­ture bias and a pos­si­ble solu­tion to moral fanaticism

80000_HoursMay 5, 2021, 7:38 PM
7 points
0 comments113 min readEA link

Vive la Différence? Struc­tural Diver­sity as a Challenge for Me­tanor­ma­tive Theories

Gordon Seidoh WorleyMay 26, 2020, 12:45 AM
17 points
0 comments1 min readEA link
(philpapers.org)

Mo­ral­ity vs re­lated concepts

MichaelA🔸Feb 10, 2020, 8:02 AM
19 points
2 comments8 min readEA link

How might we be wildly wrong?

William_MacAskillSep 4, 2013, 7:19 PM
3 points
0 comments2 min readEA link

Might stop­ping mis­car­riages be a very im­por­tant cause area?

SaraAzubuikeDec 5, 2021, 5:18 PM
16 points
13 comments2 min readEA link

EA read­ing list: moral un­cer­tainty, moral co­op­er­a­tion, and val­ues spreading

richard_ngoAug 3, 2020, 9:21 AM
15 points
0 comments1 min readEA link

Mak­ing de­ci­sions when both morally and em­piri­cally uncertain

MichaelA🔸Jan 2, 2020, 7:08 AM
22 points
4 comments20 min readEA link

“Ex­is­ten­tial Risk” is badly named and leads to nar­row fo­cus on as­tro­nom­i­cal waste

freedomandutilityAug 22, 2022, 8:25 PM
39 points
2 comments2 min readEA link

Light­ning talks from EA Global: San Fran­cisco (2018)

EA GlobalJun 8, 2018, 8:51 PM
7 points
0 comments11 min readEA link
(www.youtube.com)

New re­search on moral weights

GiveWellDec 4, 2019, 11:57 AM
42 points
4 comments2 min readEA link
(blog.givewell.org)

EA read­ing list: clue­less­ness and epistemic modesty

richard_ngoAug 3, 2020, 9:23 AM
29 points
3 comments1 min readEA link

Stan­ley the Uncer­tain [Creative Writ­ing Con­test]

AkashOct 28, 2021, 8:12 PM
33 points
1 comment3 min readEA link

Even Allo­ca­tion Strat­egy un­der High Model Ambiguity

MichaelStJulesDec 31, 2020, 9:10 AM
15 points
3 comments2 min readEA link

SoGive’s moral weights—please take part!

SanjayApr 5, 2021, 10:47 PM
26 points
2 comments2 min readEA link

Brian To­masik – Rea­sons to Be Nice to Other Value Systems

Tessa A 🔸Jan 17, 2014, 12:28 AM
24 points
0 comments1 min readEA link
(longtermrisk.org)

Why the Irre­ducible Nor­ma­tivity Wager (Mostly) Fails

Lukas_GloorJun 14, 2020, 1:33 PM
25 points
13 comments10 min readEA link

Seek­ing feed­back on new EA-al­igned eco­nomics paper

jhOct 21, 2021, 9:19 PM
35 points
13 comments1 min readEA link

Against the nor­ma­tive re­al­ist’s wager

Joe_CarlsmithOct 13, 2022, 4:36 PM
27 points
5 comments23 min readEA link

Nor­ma­tive Uncer­tainty and the Depen­dence Problem

Gordon Seidoh WorleyMar 23, 2020, 5:29 PM
15 points
4 comments2 min readEA link
(philpapers.org)

Nor­ma­tive Uncer­tainty and Prob­a­bil­is­tic Mo­ral Knowledge

Gordon Seidoh WorleyNov 11, 2019, 8:26 PM
13 points
0 comments1 min readEA link
(philpapers.org)

Chris­tian Tarsney on fu­ture bias and a pos­si­ble solu­tion to moral fanaticism

PabloMay 6, 2021, 10:39 AM
26 points
6 comments1 min readEA link
(80000hours.org)

Prac­ti­cal ethics given moral uncertainty

William_MacAskillJan 31, 2012, 5:00 AM
23 points
5 comments3 min readEA link

Han­dling Mo­ral Uncer­tainty with Aver­age vs. To­tal Utili­tar­i­anism: One Method That Ap­par­ently *Doesn’t* Work (But Seemed Like it Should)

Marcel DJan 5, 2023, 10:18 PM
10 points
0 comments8 min readEA link

On “fringe” ideas

Kelsey PiperAug 19, 2019, 7:00 AM
261 points
6 comments5 min readEA link
(theunitofcaring.tumblr.com)

“Aligned with who?” Re­sults of sur­vey­ing 1,000 US par­ti­ci­pants on AI values

Holly MorganMar 21, 2023, 10:07 PM
41 points
0 comments2 min readEA link
(www.lesswrong.com)

Tak­ing Uncer­tainty Se­ri­ously (or, Why Tools Mat­ter)

Bob FischerJul 19, 2024, 10:30 AM
115 points
4 comments8 min readEA link

Causes and Uncer­tainty: Re­think­ing Value in Expectation

Bob FischerOct 11, 2023, 9:15 AM
220 points
29 comments15 min readEA link

[Linkpost] My moral view: Re­duc­ing suffer­ing, ‘how to be’ as fun­da­men­tal to moral­ity, no pos­i­tive value, cons of grand the­ory, and more—By Si­mon Knutsson

Alistair WebsterAug 25, 2023, 12:53 PM
35 points
2 comments2 min readEA link
(centerforreducingsuffering.org)

Some thoughts on mod­er­a­tion in do­ing good

Vasco Grilo🔸Jan 20, 2024, 9:28 AM
34 points
2 comments10 min readEA link
(80000hours.org)

Mo­ral Trade, Im­pact Distri­bu­tions and Large Worlds

LarksSep 20, 2024, 3:45 AM
25 points
1 comment4 min readEA link

Solu­tion to the two en­velopes prob­lem for moral weights

MichaelStJulesFeb 19, 2024, 12:15 AM
66 points
26 comments27 min readEA link

Gra­da­tions of moral weight

MichaelStJulesFeb 29, 2024, 11:08 PM
13 points
0 comments10 min readEA link

An In­tro­duc­tion to the CRAFT Sequence

Bob FischerJul 8, 2024, 10:27 AM
103 points
1 comment4 min readEA link

Se­quence overview: Welfare and moral weights

MichaelStJulesAug 1, 2024, 4:51 PM
34 points
0 comments1 min readEA link

Re­think Pri­ori­ties’ Mo­ral Par­li­a­ment Tool

Derek ShillerJul 17, 2024, 12:34 PM
184 points
12 comments12 min readEA link

Meth­ods for im­prov­ing un­cer­tainty anal­y­sis in EA cost-effec­tive­ness models

FroolowAug 29, 2022, 6:07 PM
127 points
8 comments37 min readEA link

Why is it so hard to know if you’re helping?

Vasco Grilo🔸Oct 16, 2024, 4:08 PM
59 points
6 comments12 min readEA link
(www.theintrinsicperspective.com)

Cri­tique of MacAskill’s “Is It Good to Make Happy Peo­ple?”

Magnus VindingAug 23, 2022, 9:21 AM
225 points
115 comments8 min readEA link

Non-util­i­tar­ian effec­tive altruism

keir bradwellJan 29, 2023, 6:07 AM
42 points
10 comments17 min readEA link
(keirbradwell.substack.com)

[Question] Does any thor­ough dis­cus­sion of moral par­li­a­ments ex­ist?

richard_ngoSep 6, 2019, 3:33 PM
36 points
10 comments1 min readEA link

Pod­cast: Krister Bykvist on moral un­cer­tainty, ra­tio­nal­ity, metaethics, AI and fu­ture pop­u­la­tions

Gus DockerOct 21, 2021, 3:17 PM
8 points
0 comments1 min readEA link
(www.utilitarianpodcast.com)

A brief the­ory of why we think things are good or bad

David JohnstonOct 20, 2024, 12:15 PM
5 points
1 comment4 min readEA link

Power Laws of Value

tylermjohnMar 17, 2025, 10:10 AM
36 points
15 comments13 min readEA link

Ar­ti­cle on Value Plu­ral­ism, Mak­ing Trade Offs and Policymaking

Danny WardleMar 17, 2025, 1:26 PM
1 point
0 comments1 min readEA link
(sagacitymagazine.com.au)

Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

Jobst Heitzig (vodle.it)Sep 2, 2023, 11:32 PM
9 points
6 comments3 min readEA link

[Question] What should we ac­tu­ally do in re­sponse to moral un­cer­tainty?

John Bridge 🔸May 30, 2022, 9:48 AM
4 points
5 comments1 min readEA link

Bar­gain­ing among worldviews

Hayley ClatterbuckOct 18, 2024, 6:32 PM
57 points
5 comments12 min readEA link

Le idee “es­treme”

EA ItalyDec 31, 2022, 4:15 AM
1 point
0 comments5 min readEA link

[Opz­ionale] La pos­si­bil­ità di una Catas­trofe Mo­rale in atto (Ri­as­sunto)

EA ItalyDec 31, 2022, 4:17 AM
1 point
0 comments5 min readEA link

The Wages of North-At­lantic Bias

Sach WryAug 19, 2022, 12:34 PM
8 points
2 comments17 min readEA link

Stu­art Rus­sell Hu­man Com­pat­i­ble AI Roundtable with Allan Dafoe, Rob Re­ich, & Ma­ri­etje Schaake

Mahendra PrasadFeb 11, 2021, 7:43 AM
16 points
0 comments1 min readEA link

Sor­ti­tion Model of Mo­ral Uncertainty

Bob JacobsOct 4, 2020, 8:58 PM
33 points
5 comments2 min readEA link

Re­think­ing the Value of Work­ing on AI Safety

JohanEAJan 9, 2025, 2:15 PM
46 points
21 comments10 min readEA link

EA thoughts from Is­rael-Ha­mas war

ezrahDec 6, 2023, 8:53 AM
271 points
64 comments9 min readEA link

Large episte­molog­i­cal con­cerns I should maybe have about EA a priori

LuiseJun 7, 2023, 2:11 PM
115 points
16 comments8 min readEA link

Inescapably Value-Laden Ex­pe­rience—Can We Make Mo­ral­ity Ra­tion­al­is­able?

Non-zero-sum JamesJan 15, 2025, 12:13 AM
2 points
7 comments2 min readEA link
(nonzerosum.games)

Com­plex­ity of value but not dis­value im­plies more fo­cus on s-risk. Mo­ral un­cer­tainty and prefer­ence util­i­tar­i­anism also do.

ChiFeb 13, 2024, 10:24 PM
95 points
7 comments2 min readEA link

Mo­ral mountaineering

finmJan 18, 2025, 3:42 PM
20 points
5 comments6 min readEA link
(finmoorhouse.substack.com)

Uncer­tainty and Wild An­i­mal Welfare

abrahamroweJul 19, 2019, 1:33 PM
24 points
0 comments8 min readEA link

Fair Col­lec­tive Effec­tive Altruism

Jobst Heitzig (vodle.it)Nov 28, 2022, 1:35 PM
6 points
1 comment5 min readEA link
(www.lesswrong.com)

Ajeya Co­tra: Im­ple­ment­ing cause pri­ori­ti­sa­tion at Open Philanthropy

EA GlobalNov 3, 2017, 7:43 AM
14 points
0 comments1 min readEA link
(www.youtube.com)

Don’t treat prob­a­bil­ities less than 0.5 as if they’re 0

MichaelDickensFeb 26, 2025, 5:14 AM
36 points
3 comments1 min readEA link

[Question] De­ci­sion mak­ing un­der model am­bi­guity, moral un­cer­tainty, and other agents with free will?

Jobst Heitzig (vodle.it)Nov 2, 2022, 12:52 PM
16 points
1 comment8 min readEA link

Balanc­ing safety and waste

Daniel_FriedrichMar 17, 2024, 10:57 AM
6 points
0 comments7 min readEA link

Re­view and Sum­mary of ‘Mo­ral Uncer­tainty’

finmOct 2, 2020, 3:41 PM
37 points
3 comments1 min readEA link
(www.finmoorhouse.com)

Is­sues in (Meta) Nor­ma­tive Ethics

Noah BirnbaumFeb 5, 2025, 3:34 AM
14 points
5 comments11 min readEA link
(irrationalitycommunity.substack.com)

A bar­gain­ing-the­o­retic ap­proach to moral uncertainty

Global Priorities InstituteAug 8, 2019, 1:41 PM
3 points
0 comments3 min readEA link
(globalprioritiesinstitute.org)

Non-Con­se­quen­tial­ist Con­sid­er­a­tions For Cause-Pri­oritza­tion Part 1

Parker_WhitfillNov 28, 2018, 10:22 PM
14 points
2 comments15 min readEA link

Con­struc­tive Crit­i­cism of Mo­ral Uncer­tainty (book)

mariushobbhahnJun 4, 2021, 6:04 AM
28 points
3 comments5 min readEA link

What you pri­ori­tise is mostly moral intuition

JamesÖz 🔸Dec 24, 2022, 12:06 PM
76 points
8 comments12 min readEA link

The Mo­ral Two En­velopes Prob­lem and the Mo­ral Weights Project

Derek ShillerOct 15, 2024, 6:12 PM
92 points
20 comments14 min readEA link

[Linkpost] Leif We­nar’s The Deaths of Effec­tive Altruism

EarthlingMar 27, 2024, 3:35 PM
46 points
58 comments1 min readEA link
(www.wired.com)

Mo­ral Con­sid­er­a­tions In De­sign­ing AI Systems

Hans GundlachJul 5, 2024, 6:13 PM
8 points
1 comment7 min readEA link

Con­se­quen­tial­ism as an ex­per­i­men­tal discipline

vandermudeNov 29, 2022, 6:07 AM
9 points
2 comments3 min readEA link

Refin­ing My Mo­ral Values Us­ing Steel­man Solitaire

Evan LaForgeFeb 2, 2024, 10:07 AM
19 points
3 comments9 min readEA link

The Mo­ral Illu­sion: What Hap­pens When We Real­ize Ethics Was Never Real?

Aaron ClarkeMar 1, 2025, 8:35 PM
2 points
5 comments24 min readEA link

[Question] Is it eth­i­cal to work in AI “con­tent eval­u­a­tion”?

anon_databoy555Jan 30, 2025, 1:27 PM
10 points
3 comments1 min readEA link

Are we as rigor­ous in ad­dress­ing Utili­tar­i­anism’s the­o­ret­i­cal challenges as we are with its prac­ti­cal im­pli­ca­tions?

SilvanJul 10, 2024, 1:08 PM
15 points
5 comments5 min readEA link

Three tricky bi­ases: hu­man bias, ex­is­tence bias, and happy bias

Steven RoukFeb 2, 2024, 2:20 PM
54 points
1 comment11 min readEA link

Is Our Uni­verse A New­comb’s Para­dox Si­mu­la­tion?

Jordan ArelMay 15, 2022, 7:28 AM
16 points
8 comments2 min readEA link

Reflec­tive Equil­ibria and the Hunt for a For­mal­ized Pragmatism

BenjaminCaulfieldJun 7, 2023, 10:55 PM
4 points
1 comment12 min readEA link

Th­e­sis Sum­mary and Open Ques­tions: Nor­ma­tive Uncer­tainty and In­for­ma­tion Value

rileyharrisApr 27, 2022, 8:46 AM
32 points
0 comments3 min readEA link

Mo­ral un­cer­tainty and pub­lic jus­tifi­ca­tion (Bar­ret and Sch­midt, 2021)

Global Priorities InstituteOct 22, 2021, 9:57 AM
4 points
3 comments3 min readEA link
(globalprioritiesinstitute.org)

Re­duc­ing x-risk might be ac­tively harmful

MountainPathNov 18, 2024, 2:18 PM
22 points
9 comments1 min readEA link
No comments.