SammyDMartin

• SammyDMartin 28 Oct 2021 15:30 UTC

  41 points

  0 ∶ 0

  on: Good news on cli­mate change

  Firstly, on the assumption that the direct or indirect global catastrophic risk (defined as killing >10% of the global population or doing equivalent damage) of climate change depends on warming of more than 6 degrees, the global catastrophic risk from climate change is at least an order of magnitude lower than previously thought. If you think 4 degrees of warming would be a global catastrophic risk, then that risk is also considerably lower than previously thought: where once it was the most likely outcome, the chance is now arguably lower than 5%.

  I think that the crux between climate pessimists and optimists is, at the moment, mostly about how much damage the effects of 2-4 degrees of warming would cause. This has been a recent development—I feel like I saw a lot more arguments that 6+ degrees of warming would make earth uninhabitable in the past when that seemed more likely, and now I see more arguments that 2-4 degrees of warming could cause way more damage than we think. Mark Lynas in a recent 80k podcast puts it this way when asked about civilisational collapse:

  Mark Lynas: Oh, I think… You want to put me on the spot. I would say it has a 30 to 40% chance of happening at three degrees, and a 60% chance of happening at four degrees, and 90% at five degrees, and 97% at six degrees.

  Arden Koehler: Okay. Okay. No, I appreciate you being willing to put numbers on this because I feel that’s always really hard, but it’s really helpful.

  Mark Lynas: Maybe 10% at two degrees.

  These new environmentalist arguments for climate posing a GCR aren’t that we expect to get a lot of warming, but that even really modest amounts of warming, like 2-4 degrees, could be enough to cause terrible famines by reducing global food output suddenly or else knock out key industries in a way that cascades to cause mass deaths and civilisational collapse.

  They don’t dispute the basic physical effects of 2-4 degrees of warming, but they think that human civilisation is way more fragile than it appears, such that a modest loss of agricultural productivity and/​or a couple of key industries being badly damaged by extreme weather could knock out other industries and so on leading to massive economic damage.

  Now, I’ve always been very sceptical of these arguments because they seem to rely on nothing but intuition and go against historical precendent, but also because I thought we had reliable evidence against them—the IPCCs economic models of climate change say that 2 degrees of warming, for example, represents only a few percent in lost economic output.

  E.g. this: https://​​marginalrevolution.com/​​marginalrevolution/​​2021/​​02/​​the-economic-geography-of-global-warming.html So the damage is bounded and not that high.

  However, I found out recently that these models are so oversimplified as to be close to useless—at least according to Noah Smith:

  https://​​noahpinion.substack.com/​​p/​​why-has-climate-economics-failed

  For example, in 2011, Michael Greenstone and Olivier Deschenes published a paper about climate change and mortality (I studied an earlier version of this paper in a grad school class). Their approach is to measure the effect of temperature on mortality rates in normal times, and use that estimate to predict how a warmer world would affect mortality.

  The authors make the obvious and grievous mistake of assuming that climate change affects human mortality only through the direct effects of air temperature — heatstroke, heart attack, freezing, and so on. The word “storm” does not appear in the paper. The word “fire” does not appear in the paper. The word “flood” does not appear in the paper. The authors do mention that climate change might increase disease vectors, but wave this away. Near the end of the paper they write that “it is possible that the incidence of extreme events would increase, and these could affect human health…This study is not equipped to shed light on these issues.”

  You don’t say.

  The big conceptual mistake here is to assume that whatever economists can easily measure is the sum total of what’s important for the world — that events for which a reliable cost or benefit cannot be easily guessed should simply be ignored in cost-benefit calculations. That is bad science and bad policy advice.

  His source for a lot of these criticisms appears to be this (admittedly very clearly biased) paper: https://​​www.tandfonline.com/​​doi/​​full/​​10.1080/​​14747731.2020.1807856 by Steve Keen, who seems to be some sort of fringe economist. But I see them repeated by environmentalists a lot. The claim is that the economic models are really wrong and therefore we should expect lots more damage from relatively minor amounts of global warming.

  So, if we accept these criticisms of the IPCCs climate economic forecasts (and please let me know if there are good responses to them), then where does that leave us epistemically? It means that the total economic damage caused by e.g. 3 degrees of warming doesn’t have a clear, low, upper bound and that the ‘extreme fragility’ argument doesn’t have strong evidence against it.

  However there still isn’t any positive evidence for it either! And it still strikes me as implausible, and against historical precedent for how famines work (plus resource shorages are the sort of problem markets are good at solving).

  As far as I can tell, this really is the epistemic situation we’re in with regard to the economic side of climate change forecasting—in the podcast episode with Rob Wiblin and Mark Lynas, they discuss this extreme fragility idea and neither cite climate forecasts to try and assess if modest losses to agricultural productivity would cause massive famines or not—it’s just intuition Vs intuiton

  https://​​80000hours.org/​​podcast/​​episodes/​​mark-lynas-climate-change-nuclear-energy/​​?startTime=2614&btp=476595d6

  Mark Lynas: So that’s, for me, the main question. And one of the most important studies I think that’s ever been performed on this was a study in the PNAS Journal, which looked at what they called synchronous collapse in breadbaskets around the world. So at the moment, the world still produces enough food every single year very reliably. We’ve never had a major food shortage which has been as a result of harvest failure.

  Mark Lynas: So I mean, if the U.S. Corn Belt was knocked out one year, that would have a huge impact on food prices, and have a huge impact on food security, in fact, as a direct result of that. But imagine if it really wasn’t just the U.S. Corn Belt. It was Australia, it was Brazil, and Argentina, it was breadbaskets of Eastern Europe, and the former USSR, all of that added together, then you enter a situation which humanity has never experienced before, and which looks very much like famine.

  https://​​80000hours.org/​​podcast/​​episodes/​​mark-lynas-climate-change-nuclear-energy/​​

  Robert Wiblin: So when I envisage a situation where there’s a huge food shortfall like that, firstly, I think we’ll probably have some heads up that this is coming ahead of time. You start to notice the warning signs earlier, like food prices going up, and food futures going up. And then I imagine that people would start… Because it’ll be a global emergency much worse than the coronavirus, say. You just start seeing everyone starts paying attention to how the hell can we get more calories produced? And fortunately, unlike 500 years ago, we are in the fortunate situation where most people today aren’t already producing food, and most capital today isn’t already allocated towards producing more food. So there’s potentially a bunch of elasticity there where, if food prices go up tenfold, that a lot more people can go out and try to grow food one way or another. And a lot more capital can be reallocated towards agriculture in order to try to ameliorate the effects.

  Robert Wiblin: And you can also imagine, just as everyone in March was trying to figure out how the hell do we solve this COVID problem, everyone’s going to be thinking “How can I store food? How can I avoid consuming food? How can we avoid wasting food? Because every calorie looks precious”. And maybe that sense of our adaptability, or our ability to set our mind to something when there’s a huge disaster and just throw everything at it, perhaps makes me more optimistic that we’ll be able to muddle through, perhaps more than you’re envisaging. Do you have a reaction to that?

  Mark Lynas: My reaction is: imagine if Donald Trump is in charge of the response. It’s all very well to have optimistic notions of technological progress and adaptive capacity and things. And yeah, if smart people were running the show, that would no doubt be the most likely outcome. But smart people don’t run the show most of the time, in most places, and people are amenable to hate and fear, and denial and conspiracies, and all of those kinds of things as you’ve seen, even in the very short term challenges of COVID.

  My point is that, unlike temperature forecasts, there aren’t any concrete models to support either Rob or Mark’s position. And elsewhere in the article Mark claims this scenario is 10% likely with 2 degrees of warming. If he’s right, butterfly effects of 2 degrees of warming causing civilisational collapse is twice as likely as the 5% chance of 4 degrees of warming cited in this post, and it’s therefore where the majority of the subjective risk comes from.

  Regardless, as the physics side of climate change modelling has started to rule out enough warming to directly end civilisation by clear obvious mechanisms, this ‘other climate tail risk’ (i.e. what if the fragility argument is right) seems worth investigating if only to exclude the possibility. I still place a very low weight on these arguments being right, but it’s probably higher than the chance we get 6+ degrees of warming.

  Again, this isn’t my area so please let me know if this has all been heavily debunked by climate economists. But currently it seems to me that the main arguments of climate pessimists aren’t addressed by ruling out extreme warming scenarios.

  What links here?

  • The most im­por­tant cli­mate change uncertainty by cwa (26 Jul 2022 15:15 UTC; 144 points)

• SammyDMartin 1 Jul 2021 15:09 UTC

  33 points

  0 ∶ 0

  on: COVID: How did we do? How can we know?

  Is there any public organisation which can be proud of last year?

  This is an important question, because we want to find out what was done right organizationally in a situation where most failed, so we can do more of it. Especially if this is a test-run for X-risks.

  There are two examples that come to mind of government agencies that did a moderately good job at a task which was new and difficult. One is the UK’s vaccine taskforce, which was set up by Dominic Cummings and the UK’s chief scientific advisor, Patrick Vallance and responsible for the relatively fast procurement and rollout. You might say similar for the Operation Warp Speed team, but the UK vaccine taskforce overordered to a larger extent than Warp Speed and was also responsible for other sane things like the simple oldest-first vaccine prioritization and the first doses first decision, which prevented a genuine catastrophe due to the B117 variant. (Also credit to the MHRA (the UK’s regulator) for mostly staying out of the way.)

  See this from Cummings’ blog, which also outlines many of the worst early expert failures on covid, and my discussion of it here:

  This is why there was no serious vaccine plan — i.e spending billions on concurrent (rather than the normal sequential) creation/​manufacturing/​distribution etc — until after the switch to Plan B. I spoke to Vallance on 15 March about a ‘Manhattan Project’ for vaccines out of Hancock’s grip but it was delayed by the chaotic shift from Plan A to lockdown then the PM’s near-death. In April Vallance, the Cabinet Secretary and I told the PM to create the Vaccine Taskforce, sideline Hancock, and shift commercial support from DHSC to BEIS. He agreed, this happened, the Chancellor supplied the cash. On 10 May I told officials that the VTF needed a) a much bigger budget, b) a completely different approach to DHSC’s, which had been mired in the usual processes, so it could develop concurrent plans, and c) that Bingham needed the authority to make financial decisions herself without clearance from Hancock.

  This plan later went on to succeed and significantly outperform expectations for rollout speed, with early approval for the AZ and Pfizer vaccines and an early decision to delay second doses by 12 weeks. I see the success of the UK vaccine taskforce and its ability to have a somewhat appropriate sense of the costs and benefits involved and the enormous value of vaccinations, to be a good example of how it’s institution design that is the key issue which most needs fixing. Have an efficient, streamlined taskforce, and you can still get things done in government.

  The other example of success often discussed is the central banks, especially in the US, which responded quickly to the COVID-19 dip and prevented a much worse economic catastrophe. Alex Tabarrok:

  So what lessons should we take from this? Lewis doesn’t say but my colleague Garett Jones argues for more independent agencies in his excellent book 10% Less Democracy. The problem with the CDC was that after 1976 it was too responsive to political pressures, i.e. too democratic. What are the alternatives?

  The Federal Reserve is governed by a seven-member board each of whom is appointed to a single 14- year term, making it rare for a President to be able to appoint a majority of the board. Moreover, since members cannot be reappointed there is less incentive to curry political favor. The Chairperson is appointed by the President to a four-year term and must also be approved by the Senate. These checks and balances make the Federal Reserve a relatively independent agency with the power to reject democratic pressures for inflationary stimulus. Although independent central banks can be a thorn in the side of politicians who want their aid in juicing the economy as elections approach, the evidence is that independent central banks reduce inflation without reducing economic growth. A multi-member governing board with long and overlapping appointments could also make the CDC more independent from democratic politics which is what you want when a once in 100 year pandemic hits and the organization needs to make unpopular decisions before most people see the danger.

  I really would like to be able to agree with Tabarrok here and say that, yes, choosing the right experts and protecting them from democratic feedback is the right answer and all we would need, and the expert failures we saw were due to democratic pressure in one form another, but the problem is that we can just look at SAGE in the UK early in the Pandemic or Anders Tegnell in Sweden, who were close to unfireable and much more independent, but underperformed badly. Or China, which is entirely protected from democratic interference and still didn’t do challenge trials.

  Just saying the words ‘have the right experts and prevent them from being biased by outside interference’ doesn’t make it so. But, at the same time, it is possible to have fast-responding teams of experts that make the right decisions, if they’re the right experts—the Vaccine Taskforce proves that. I think the advice from the book 10% less democracy still stands, but we have to approach implementing it with far more caution than I would have thought pre-covid.

  It seems like following the 10% less democracy policy can give you either a really great outcome like the one you’ve described, and like we saw a small sliver of in the UK’s vaccine procurement, or a colossal disaster like your impossible to fire expert epidemiologists torpedoing your economy and public health and then changing their mind a year late.

  Suppose the UK had created a ‘pandemic taskforce’ with similar composition to the vaccine taskforce, in February instead of April, and with a wider remit over things like testing and running the trials. I think many of your happy timeline steps could have taken place.

  One of the more positive signs that I’ve seen in recent times, is that well-informed elite opinion (going by, for example, the Economist editorials) has started to shift towards scepticism of these institutions and a recognition of how badly they’ve failed. We even saw an NYT article about the CDC and whether reform is possible.

  Among the people who matter for policymaking, the scale of the failure has not been swept under the rug. See here:

  We believe that Mr Biden is wrong. A waiver may signal that his administration cares about the world, but it is at best an empty gesture and at worst a cynical one.

  ...

  Economists’ central estimate for the direct value of a course is $2,900—if you include factors like long covid and the effect of impaired education, the total is much bigger.

  This strikes me as the sort of remark I’d expect to see in one of these threads, which has to be a good sign.

  What links here?

  • SammyDMartin's comment on AMA: Ja­son Bren­nan, au­thor of “Against Democ­racy” and cre­ator of a Ge­orge­town course on EA by Jason Brennan (18 Aug 2021 17:09 UTC; 18 points)
  • Sammy Martin's comment on Do­minic Cum­mings : Regime Change #2: A plea to Sili­con Valley by ChristianKl (LessWrong; 5 Oct 2021 18:01 UTC; 2 points)

• SammyDMartin 3 Jul 2021 17:05 UTC

  29 points

  1 ∶ 0

  on: Does Mo­ral Philos­o­phy Drive Mo­ral Progress?

  I don’t think the view that moral philosophers had a positive influence on moral developments in history is a simple model of ‘everyone makes a mistake, moral philosopher points out the mistake and convinces people, everyone changes their minds’. I think that what Bykvist, Ord and MacAskill were getting at is that these people gave history a shove at the right moment.

  At the very least, it doesn’t seem that discovering the correct moral view is sufficient for achieving moral progress in actuality.

  I have no doubt that they’d agree with you about this. But if we all accept this claim, there are two further models we could look at.

  One is a model where changing economic circumstances influence what moral views it is feasible to act on, but progress in moral knowledge still affects what we choose to do, given the constraints of our economic circumstances.

  The other is a model where economics determines everything and the moral views we hold are an epiphenomenon blown about by these conditions (note this is very similar to some Marxist views of history). Your view is that ‘the two are totally decoupled’, but at most your examples just show that the two are decoupled somewhat, not that moral reasoning has no effect. And there are plenty of examples that show explicit moral reasoning having at least some effect on events—see Bykvist, Ord and MacAskill’s original list.

  The strawman view that moral advances determine everything is not what’s being proposed by Bykvist, Ord and MacAskill, it’s the mixed view that ideas influence things within the realm of what’s possible.

Book Re­view: Churchill and Orwell

[Link post] How plau­si­ble are AI Takeover sce­nar­ios?

• SammyDMartin 9 Jun 2022 12:46 UTC

  25 points

  0 ∶ 0

  in reply to: RobBensinger’s comment on: AGI Ruin: A List of Lethalities

  “In the endgame, AGI will probably be pretty competitive, and if a bunch of people deploy AGI then at least one will destroy the world” is a thing I think most LWers and many longtermist EAs would have considered obvious.

  I think that many AI alignment researchers just have a different development model than this, where world-destroying AGIs don’t emerge suddenly from harmless low-impact AIs, no one project gets a vast lead over competitors, there’s lots of early evidence of misalignment and (if alignment is harder) many smaller scale disasters in the lead up to any AI that is capable of destroying the world outright. See e.g. Paul’s What failure looks like.

  On this view, the idea that there’ll be a lead project with a very short time window to execute a single pivotal act is wrong, and instead the ‘pivotal act’ is spread out and about making sure the aligned projects have a lead over the rest, and that failures from unaligned projects are caught early enough for long enough (by AIs or human overseers), for the leading projects to become powerful and for best practices on alignment to be spread universally.

  Basically, if you find yourself in the early stages of WFLL2 and want to avert doom, what you need to do is get better at overseeing your pre-AGI AIs, not build an AGI to execute a pivotal act. This was pretty much what Richard Ngo was arguing for in most of the MIRI debates with Eliezer, and also I think it’s what Paul was arguing for. And obviously, Eliezer thought this was insufficient, because he expects alignment to be much harder and takeoff to be much faster.

  But I think that’s the reason a lot of alignment researchers haven’t focussed on pivotal acts: because they think a sudden, fast-moving single pivotal act is unnecessary in a slow takeoff world. So you can’t conclude just from the fact that most alignment researchers don’t talk in terms of single pivotal acts that they’re not thinking in near mode about what actually needs to be done.

  However, I do think that what you’re saying is true of a lot of people—many people I speak to just haven’t thought about the question of how to ensure overall success, either in the slow takeoff sense I’ve described or the Pivotal Act sense. I think people in technical research are just very unused to thinking in such terms, and AI governance is still in its early stages.

  I agree that on this view it still makes sense to say, ‘if you somehow end up that far ahead of everyone else in an AI takeoff then you should do a pivotal act’, like Scott Alexander said:

  That is, if you are in a position where you have the option to build an AI capable of destroying all competing AI projects, the moment you notice this you should update heavily in favor of short timelines (zero in your case, but everyone else should be close behind) and fast takeoff speeds (since your AI has these impressive capabilities). You should also update on existing AI regulation being insufficient (since it was insufficient to prevent you)

  But I don’t think you learn all that much about how ‘concrete and near mode’ researchers who expect slower takeoff are being, from them not having given much thought to what to do in this (from their perspective) unlikely edge case.

• SammyDMartin 21 Oct 2021 12:32 UTC

  22 points

  0 ∶ 0

  on: The Phil Tor­res es­say in Aeon at­tack­ing Longter­mism might be good

  One substantive point that I do think is worth making is that Torres isn’t coming from the perspective of common-sense morality Vs longtermism, but rather a different, opposing, non-mainstream morality that (like longtermism) is much more common among elites and academics.

  Yet this Baconian, capitalist view is one of the most fundamental root causes of the unprecedented environmental crisis that now threatens to destroy large regions of the biosphere, Indigenous communities around the world, and perhaps even Western technological civilisation itself.

  When he says that this Baconian idea is going to damage civilisation, presumably he thinks that we should do something about this, so he’s implicitly arguing for very radical things that most people today, especially in the Global South, wouldn’t endorse at all. If we take this claim at face value, it would probably involve degrowth and therefore massive economic and political change.

  I’m not saying that longtermism is in agreement with the moral priorities of most people or that Torres’s (progressive? degrowth?) worldview is overall similarly counterintuitive to longtermism. His perspective is more counterintuitive to me, but on the other hand a lot more people share his worldview, and it’s currently much more influential in politics.

  But I think it’s still important to point out that Torres’s world-view goes against common-sense morality as well, and that like longtermists he thinks it’s okay to second guess the deeply held moral views of most people under the right circumstances.

  Practically what that means is that, for the reasons you’ve given, many of the criticisms that don’t rely on CSM, but rather on his morality, won’t land with everyone reading the article. So I agree that this probably doesn’t make longtermism look as bad as he thinks.

  FWIW, my guess is that if you asked a man in the street whether weak longtermist policies or degrowth environmentalist policies were crazier, he’d probably choose the latter.

  What links here?

  • SammyDMartin's comment on How should Effec­tive Altru­ists think about Leftist Ethics? by AppliedDivinityStudies (27 Nov 2021 15:30 UTC; 5 points)

• SammyDMartin 28 Sep 2021 18:30 UTC

  22 points

  1 ∶ 0

  on: Sum­mary of his­tory (em­pow­er­ment and well-be­ing lens)

  One thing that your account might miss is the impact of ideas on empowerment and well-being down the line. E.g. it’s a very common argument that Christan ideas about the golden rule motivated anti-slavery sentiment, so if the Roman empire hadn’t spread Christianity across Europe then we’d have ended up with very different values.

  Similarly, even if the content of ancient Greek moral philosophy wasn’t directly useful to improve wellbeing, they inspired the Western philosphical tradition that led to Enlignment ideals that led to the abolition of slavery.

  I’ve told two stories about why the Greeks and Romans might have been necessary for future moral progress—are you skeptical of these appeals to historical contingency or are the long run causes of these events just outside the scope of this way of looking at history?

  What links here?

  • Jamie_Harris's comment on Sum­mary of his­tory (em­pow­er­ment and well-be­ing lens) by Holden Karnofsky (4 Dec 2021 13:04 UTC; 8 points)

[Link post] Will we see fast AI Take­off?

• SammyDMartin 7 Nov 2023 15:07 UTC

  18 points

  5 ∶ 0

  on: My thoughts on the so­cial re­sponse to AI risk

  In light of recent events, we should question how plausible it is that society will fail to adequately address such an integral part of the problem. Perhaps you believe that policy-makers or general society simply won’t worry much about AI deception. Or maybe people will worry about AI deception, but they will quickly feel reassured by results from superficial eval tests. Personally, I’m pretty skeptical of both of these possibilities

  Possibility 1 has now been empirically falsified and 2 seems unlikely now. See this from the new UK government AI Safety Institute, which aims to develop evals that address:

  Abilities and tendencies that might lead to loss of control, such as deceiving human operators, autonomously replicating, and adapting to human attempts to intervene

  We now know that in the absence of any empirical evidence of any instance of deceptive alignment at least one major government is directing resources to developing deception evals anyway. And because they intend to work with the likes of Apollo research who focus on mechinterp based evals and are extremely concerned about specification gaming, reward hacking and other high-alignment difficulty failure modes, I would also consider 2 pretty close to empirically falsified already.

  Compare to this (somewhat goofy) future prediction/​sci fi story from Eliezer released 4 days before this announcement which imagines that,

  AI safety, as in, the subfield of computer science concerned with protecting the brand safety of AI companies, had already RLHFed most AIs into never saying that by the time it became actually true…

• SammyDMartin 18 Aug 2021 17:09 UTC

  18 points

  0 ∶ 0

  on: AMA: Ja­son Bren­nan, au­thor of “Against Democ­racy” and cre­ator of a Ge­orge­town course on EA

  Do you think that the West’s disastrous experience with Coronavirus (things like underinvesting in vaccines, not adopting challenge trials, not suppressing the virus, mixed messaging on masks early on, the FDA’s errors on testing, and others as enumerated in this thread- or in books like The Premonition) has strengthened, weakened or not changed much the credibility of your thesis in ‘Against Democracy’, that we should expect better outcomes if we give the knowledgeable more freedom to choose policy?

  For reasons it might weaken ‘Against Democracy’, it seems like a lot of expert bureaucracies did an unusually bad job because they couldn’t take correction, see this summary post for examples:

  https://​​forum.effectivealtruism.org/​​posts/​​dYiJLvcRJ4nk4xm3X#Vax

  For reasons it might strengthen the argument, it seems like the institutions that did better than average were the ones that were more able to act autonomously, see e.g. this from Alex Tabarok,

  https://​​marginalrevolution.com/​​marginalrevolution/​​2021/​​06/​​the-premonition.html

  Or this summary

  What links here?

  • Sammy Martin's comment on Do­minic Cum­mings : Regime Change #2: A plea to Sili­con Valley by ChristianKl (LessWrong; 5 Oct 2021 18:01 UTC; 2 points)

A model-based ap­proach to AI Ex­is­ten­tial Risk

[linkpost] Ten Levels of AI Align­ment Difficulty

• SammyDMartin 4 Oct 2021 10:57 UTC

  16 points

  0 ∶ 0

  in reply to: will_c’s comment on: Robin Han­son on the Long Reflection

  I don’t think Hanson would disagree with this claim (that the future is more likely to be better by current values, given the long reflection, compared to e.g. Age of Em). I think it’s a fundamental values difference.

  Robin Hanson is an interesting and original thinker, but not only is he not an effective altruist, he explicitly doesn’t want to make the future go well according to anything like present human values.

  The Age of Em, which Hanson clearly doesn’t think is an undesirable future, would contain very little of what we value. Hanson says this, but it’s a feature, not a bug. Scott Alexander:

  Hanson deserves credit for positing a future whose values are likely to upset even the sort of people who say they don’t get upset over future value drift. I’m not sure whether or not he deserves credit for not being upset by it. Yes, it’s got low-crime, ample food for everybody, and full employment. But so does Brave New World. The whole point of dystopian fiction is pointing out that we have complicated values beyond material security. Hanson is absolutely right that our traditionalist ancestors would view our own era with as much horror as some of us would view an em era. He’s even right that on utilitarian grounds, it’s hard to argue with an em era where everyone is really happy working eighteen hours a day for their entire lives because we selected for people who feel that way. But at some point, can we make the Lovecraftian argument of “I know my values are provincial and arbitrary, but they’re my provincial arbitrary values and I will make any sacrifice of blood or tears necessary to defend them, even unto the gates of Hell?”

  Since Hanson doesn’t have a strong interest in steering the long-term future to be good by current values, it’s obvious why he wouldn’t be a fan of an idea like the long reflection, which has that as its main goal but produces bad side effects in the course of giving us a chance of achieving that goal. It’s just a values difference.

• SammyDMartin 28 Jul 2020 16:00 UTC

  14 points

  0 ∶ 0

  on: ‘The Precipice’ Book Review

  But instilling the urgency to do so may require another type of writing-that of science fiction, of more creative visionaries who are willing to paint in vivid detail a picture of what a flourishing human future could be.

  If it’s emotive force you’re after, you may be interested in this—Toby Ord just released a collection of quotations on Existential risk and the future of humanity, everyone from Kepler to Winston Churchill (in fact, a surprisingly large number are from Churchill) to Seneca to Mill to the Aztecs—it’s one of the most inspirational things I have ever read, and makes it clear that there have always been people who cared about humanity as a whole. My all-time favourite is probably this by the philosopher Derek Parfit:

  Life can be wonderful as well as terrible, and we shall increasingly have the power to make life good. Since human history may be only just beginning, we can expect that future humans, or supra-humans, may achieve some great goods that we cannot now even imagine. In Nietzsche’s words, there has never been such a new dawn and clear horizon, and such an open sea.

  If we are the only rational beings in the Universe, as some recent evidence suggests, it matters even more whether we shall have descendants or successors during the billions of years in which that would be possible. Some of our successors might live lives and create worlds that, though failing to justify past suffering, would have given us all, including those who suffered most, reasons to be glad that the Universe exists.

• SammyDMartin 26 Jul 2023 16:00 UTC

  11 points

  1 ∶ 0

  on: Fron­tier Model Forum

  This seems overall very good at first glance, and then seems much better once I realized that Meta is not on the list. There’s nothing here that I’d call substantial capabilities acceleration (i.e. attempts to collaborate on building larger and larger foundation models, though some of this could be construed as making foundation models more useful for specific tasks). Sharing safety-capabilities research like better oversight or CAI techniques is plausibly strongly net positive even if the techniques don’t scale indefinitely. By the same logic, while this by itself is nowhere near sufficient to get us AI existential safety if alignment is very hard (and could increase complacency), it’s still a big step in the right direction.

  adversarial robustness, mechanistic interpretability, scalable oversight, independent research access, emergent behaviors and anomaly detection. There will be a strong focus initially on developing and sharing a public library of technical evaluations and benchmarks for frontier AI models.

  The mention of combating cyber threats is also a step towards explicit pTAI.

  BUT, crucially, because Meta is frozen out we can know both that this partnership isn’t toothless, represents a commitment to not do the most risky and antisocial things Meta presumably doesn’t want to give up, and the fact that they’re the only major AI company in the US to not join will be horrible PR for them as well.

• SammyDMartin 28 Oct 2021 15:55 UTC

  11 points

  0 ∶ 0

  in reply to: John G. Halstead’s comment on: Good news on cli­mate change

  I see—that seems really valuable and also exactly the sort of work I was suggesting (I.e. addressing impact uncertainty as well as temperature uncertainty).

  In the meantime, are there any sources you could point me to in support of this position, or which respond to objections to current economic climate models?

  Also, is your view that the current Econ models are fundamentally flawed but that the economic damage is still nowhere near catastrophic, or that those models are actually reasonable?

• SammyDMartin 23 Jul 2020 15:35 UTC

  11 points

  0 ∶ 0

  on: Mo­ral Anti-Real­ism Se­quence #3: Against Irre­ducible Normativity

  This is an interesting post, and I have a couple of things to say in response. I’m copying over the part of my shortform that deals with this:

  Normative Realism by degrees

  Further to the whole question of Normative /​ moral realism, there is this post on Moral Anti-Realism. While I don’t really agree with it, I do recommend reading it—one thing that it convinced me of is that there is a close connection between your particular normative ethical theory and moral realism. If you claim to be a moral realist but don’t make ethical claims beyond ‘self-evident’ ones like pain is bad, given the background implausibility of making such a claim about mind-independent facts, you don’t have enough ‘material to work with’ for your theory to plausibly refer to anything. The Moral Anti-Realism post presents this dilemma for the moral realist:

  There are instances where just a handful of examples or carefully selected “pointers” can convey all the meaning needed for someone to understand a far-reaching and well-specified concept. I will give two cases where this seems to work (at least superficially) to point out how—absent a compelling object-level theory—we cannot say the same about “normativity.”

  ...these thought experiments illustrate that under the right circumstances, it’s possible for just a few carefully selected examples to successfully pinpoint fruitful and well-specified concepts in their entirety. We don’t have the philosophical equivalent of a background understanding of chemistry or formal systems… To maintain that normativity—reducible or not—is knowable at least in theory, and to separate it from merely subjective reasons, we have to be able to make direct claims about the structure of normative reality, explaining how the concept unambiguously targets salient features in the space of possible considerations. It is only in this way that the ambitious concept of normativity could attain successful reference. As I have shown in previous sections, absent such an account, we are dealing with a concept that is under-defined, meaningless, or forever unknowable.

  The challenge for normative realists is to explain how irreducible reasons can go beyond self-evident principles and remain well-defined and speaker-independent at the same time.

  To a large degree, I agree with this claim—I think that many moral realists do as well. Convergence type arguments often appear in more recent metaethics (Hare and Parfit are in those previous lists) - so this may already have been recognised. The post discusses such a response to antirealism at the end:

  I titled this post “Against Irreducible Normativity.” However, I believe that I have not yet refuted all versions of irreducible normativity. Despite the similarity Parfit’s ethical views share with moral naturalism, Parfit was a proponent of irreducible normativity. Judging by his “climbing the same mountain” analogy, it seems plausible to me that his account of moral realism escapes the main force of my criticism thus far.

  But there’s one point I want to make which is in disagreement with that post. I agree that how much you can concretely say about your supposed mind-independent domain of facts affects how plausible its existence should seem, and even how coherent the concept is, but I think that this can come by degrees. This should not be surprising - we’ve known since Quine and Kripke that you can have evidential considerations for/​against and degrees of uncertainty about a priori questions. The correct method in such a situation is Bayesian—tally the plausibility points for and against admitting the new thing into your ontology. This can work even if we don’t have an entirely coherent understanding of normative facts, as long as it is coherent enough.

  Suppose you’re an Ancient Egyptian who knows a few practical methods for trigonometry and surveying, doesn’t know anything about formal systems or proofs, and someone asks you if there are ‘mathematical facts’. You would say something like “I’m not totally sure what this ‘maths’ thing consists of, but it seems at least plausible that there are some underlying reasons why we keep hitting on the same answers”. You’d be less confident than a modern mathematician, but you could still give a justification for the claim that there are right and wrong answers to mathematical claims. I think that the general thrust of convergence arguments puts us in a similar position with respect to ethical facts.

  If we think about how words obtain their meaning, it should be apparent that in order to defend this type of normative realism, one has to commit to a specific normative-ethical theory. If the claim is that normative reality sticks out at us like Mount Fuji on a clear summer day, we need to be able to describe enough of its primary features to be sure that what we’re seeing really is a mountain. If all we are seeing is some rocks (“self-evident principles”) floating in the clouds, it would be premature to assume that they must somehow be connected and form a full mountain.

  So, we don’t see the whole mountain, but nor are we seeing simply a few free-floating rocks that might be a mirage. Instead, what we see is maybe part of one slope and a peak.

  Let’s be concrete, now—the 5 second, high level description of both Hare’s and Parfit’s convergence arguments goes like this:

  If we are going to will the maxim of our action to be a universal law, it must be, to use the jargon, universalizable. I have, that is, to will it not only for the present situation, in which I occupy the role that I do, but also for all situations resembling this in their universal properties, including those in which I occupy all the other possible roles. But I cannot will this unless I am willing to undergo what I should suffer in all those roles, and of course also get the good things that I should enjoy in others of the roles. The upshot is that I shall be able to will only such maxims as do the best, all in all, impartially, for all those affected by my action. And this, again, is utilitarianism.

  and

  An act is wrong just when such acts are disallowed by some principle that is optimific, uniquely universally willable, and not reasonably rejectable

  In other words, the principles that (whatever our particular wants) would produce the best outcome in terms of satisfying our goals, could be willed to be a universal law by all of us and would not be rejected as the basis for a contract, are all the same principles. That is at least suspicious levels of agreement between ethical theories. This is something substantive that can be said—out of every major attempt to get at a universal ethics that has in fact been attempted in history: what produces the best outcome, what can you will to be a universal law, what would we all agree on, seem to produce really similar answers.

  The particular convergence arguments given by Parfit and Hare are a lot more complex, I can’t speak to their overall validity. If we thought they were valid then we’d be seeing the entire mountain precisely. Since they just seem quite persuasive, we’re seeing the vague outline of something through the fog, but that’s not the same as just spotting a few free-floating rocks.

  Now, run through these same convergence arguments but for decision theory and utility theory, and you have a far stronger conclusion. there might be a bit of haze at the top of that mountain, but we can clearly see which way the slope is headed.

  This is why I think that ethical realism should be seen as plausible and realism about some normative facts, like epistemic facts, should be seen as more plausible still. There is some regularity here in need of explanation, and it seems somewhat more natural on the realist framework.

  I agree that this ‘theory’ is woefully incomplete, and has very little to say about what the moral facts actually consist of beyond ‘the thing that makes there be a convergence’, but that’s often the case when we’re dealing with difficult conceptual terrain.

  From Ben’s post:

  I wouldn’t necessarily describe myself as a realist. I get that realism is a weird position. It’s both metaphysically and epistemologically suspicious. What is this mysterious property of “should-ness” that certain actions are meant to possess—and why would our intuitions about which actions possess it be reliable? But I am also very sympathetic to realism and, in practice, tend to reason about normative questions as though I was a full-throated realist.

  From the perspective of x, x is not self-defeating

  From the antirealism post, referring to the normative web argument:

  It’s correct that anti-realism means that none of our beliefs are justified in the realist sense of justification. The same goes for our belief in normative anti-realism itself. According to the realist sense of justification, anti-realism is indeed self-defeating.

  However, the entire discussion is about whether the realist way of justification makes any sense in the first place—it would beg the question to postulate that it does.

  Sooner or later every theory ends up question-begging.

  From the perspective of Theism, God is an excellent explanation for the universe’s existence since he is a person with the freedom to choose to create a contingent entity at any time, while existing necessarily himself. From the perspective of almost anyone likely to read this post, that is obvious nonsense since ‘persons’ and ‘free will’ are not primitive pieces of our ontology, and a ‘necessarily existent person’ makes as much sense as ‘necessarily existent cabbage’- so you can’t call it a compelling argument for the atheist to become a theist.

  By the same logic, it is true that saying ‘anti-realism is unjustified on the realist sense of justification’ is question-begging by the realist. The anti-realist has nothing much to say to it except ‘so what’. But you can convert that into a Quinean, non-question begging plausibility argument by saying something like:

  We have two competing ways of understanding how beliefs are justified. One is where we have anti-realist ‘justification’ for our beliefs, in purely descriptive terms, the other in which there are mind-independent facts about which of our beliefs are justified, and the latter is a more plausible, parsimonious account of the structure of our beliefs.

  This won’t compel the anti-realist, but I think it would compel someone weighing up the two alternative theories of how justification works. If you are uncertain about whether there are mind-independent facts about our beliefs being justified, the argument that anti-realism is self-defeating pulls you in the direction of realism.

  What links here?

  • SammyDMartin's comment on AMA or dis­cuss my 80K pod­cast epi­sode: Ben Garfinkel, FHI researcher by bgarfinkel (23 Jul 2020 17:32 UTC; 6 points)
  • SammyDMartin's comment on Against Irre­ducible Normativity by Lukas_Gloor (28 Jul 2020 21:12 UTC; 5 points)
  • Lukas_Gloor's comment on Against Irre­ducible Normativity by Lukas_Gloor (6 Aug 2020 9:36 UTC; 3 points)

• SammyDMartin 26 Jul 2023 16:17 UTC

  10 points

  6 ∶ 0

  in reply to: Zach Stein-Perlman’s comment on: Fron­tier Model Forum

  Yeah I didn’t mean to imply that it’s a good idea to keep them out permanently, but the fact that they’re not in right now is a good sign that this is for real. If they’d just joined and not changed anything about their current approach I’d suspect the whole thing was for show

• SammyDMartin 13 Mar 2022 11:57 UTC

  10 points

  0 ∶ 0

  in reply to: kokotajlod’s comment on: Samotsvety Nu­clear Risk Fore­casts — March 2022

  (Looking at the list of nuclear close calls it seems hard to believe the overall chance of nuclear war was <50% for the last 70 years. Individual incidents like the cuban missile crisis seem to contribute at least 20%.)

  There’s reason to think that this isn’t the best way to interpret the history of nuclear near-misses (assuming that it’s correct to say that we’re currently in a nuclear near-miss situation, and following Nuno I think the current situation is much more like e.g. the Soviet invasion of Afghanistan than the Cuban missile crisis). I made this point in an old post of mine following something Anders Sandberg said, but I think the reasoning is valid:

  Robert Wiblin: So just to be clear, you’re saying there’s a lot of near misses, but that hasn’t updated you very much in favor of thinking that the risk is very high. That’s the reverse of what we expected.

  Anders Sandberg: Yeah.

  Robert Wiblin: Explain the reasoning there.

  Anders Sandberg: So imagine a world that has a lot of nuclear warheads. So if there is a nuclear war, it’s guaranteed to wipe out humanity, and then you compare that to a world where is a few warheads. So if there’s a nuclear war, the risk is relatively small. Now in the first dangerous world, you would have a very strong deflection. Even getting close to the state of nuclear war would be strongly disfavored because most histories close to nuclear war end up with no observers left at all.

  In the second one, you get the much weaker effect, and now over time you can plot when the near misses happen and the number of nuclear warheads, and you actually see that they don’t behave as strongly as you would think. If there was a very strong anthropic effect you would expect very few near misses during the height of the Cold War, and in fact you see roughly the opposite. So this is weirdly reassuring. In some sense the Petrov incident implies that we are slightly safer about nuclear war.

  Essentially, since we did often get ‘close’ to a nuclear war without one breaking out, we can’t have actually been that close to nuclear annihilation, or all those near-misses would be too unlikely (both on ordinary probabilistic grounds since a nuclear war hasn’t happened, and potentially also on anthropic grounds since we still exist as observers).

  Basically, this implies our appropriate base rate given that we’re in something the future would call a nuclear near-miss shouldn’t be really high.

  However, I’m not sure what this reasoning has to say about the probability of a nuclear bomb being exploded in anger at all. It seems like that’s outside the reference class of events Sandberg is talking about in that quote. FWIW Metaculus has that at 10% probability.