Muireall

• Muireall 7 Jun 2022 12:42 UTC

  2 points

  0 ∶ 0

  on: Open Thread: Novem­ber 2021

  Hi, everyone, I’m Muireall. I recently put down some thoughts on weighing the longterm future (https://​​muireall.space/​​repugnant/​​). I suspect something like this has been brought up before, but I haven’t been keeping up with writing on the topic for years. It occurred to me that this forum might be able to help with references or relevant keywords that come to mind. I’d appreciate any thoughts you have.

  The idea is that, broadly, if you accept the repugnant conclusion with a “high” threshold (some people consensually alive today don’t meet the “barely worth living” line), I think your expected utility for the longterm future has to take a big hit from negative scenarios. From that perspective, not only is it likely that future civilization will—as do, apparently, we—mistake negative for positive welfare, but also welfare should be put on hold (since apparently near-threshold lives can be productive) as they too invest in favor of the distant intergalactic future (until existential catastrophe comes for them).

  In other words, I worry (1) expected-total-utility motivations for longtermism underrate very bad outcomes, and (2) these motivations can put you in the position of continually making Pascalian bets long enough to all but guarantee gambler’s ruin before realizing your astronomical potential value.

• Muireall 7 Jun 2022 19:14 UTC

  2 points

  0 ∶ 0

  in reply to: Muireall’s comment on: Open Thread: Novem­ber 2021

  I’ll answer my own question a bit:

  • Scattered critiques of longtermism exist, but are generally informal, tentative, and limited in scope. This recent comment and its replies were the best directory I could find.

  • A longtermist critique of “The expected value of extinction risk reduction is positive”, in particular, seems to be the best expression of my worry (1). My points about near-threshold lives and procrastination are another plausible story by which extinction risk reduction could be negative in expectation.

  • There’s writing about Pascalian reasoning (a couple that came up repeatedly were A Paradox for Tiny Probabilities and Enormous Values, In defence of fanaticism).

  • I vaguely recall a named paradox, maybe involving “procrastination” or “patience”, about how an immortal investor never cashes in—and possibly that this was a standard answer to Pascal’s wager/​mugging together with some larger (but still tiny) probability of, say, getting hit by a meteor while you’re making the bet. Maybe I just imagined it.

• Muireall 9 Jun 2022 18:43 UTC

  1 point

  0 ∶ 0

  in reply to: Muireall’s comment on: Open Thread: Novem­ber 2021

  I added a more mathematical note at the end of my post showing what I mean by (2). I think in general it’s more coherent to treat trajectory problems with dynamic programming methods rather than try to integrate expected value over time.

  

• Muireall 16 Jun 2022 23:47 UTC

  3 points

  0 ∶ 0

  on: AGI Ruin: A List of Lethalities

  The concrete example I usually use here is nanotech, because there’s been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point.

  It sounds like this is well-traveled ground here, but I’d appreciate a pointer to this analysis.

• Muireall 17 Jun 2022 13:29 UTC

  3 points

  0 ∶ 0

  in reply to: RobBensinger’s comment on: AGI Ruin: A List of Lethalities

  Interesting, thanks. I read Nanosystems as establishing a high upper bound. I don’t see any of its specific proposals as plausibly workable enough to use as a lower bound in the sense that, say, a ribosome is a lower bound, but perhaps that’s not what Eliezer means.

• Muireall 23 Jun 2022 2:52 UTC

  26 points

  0 ∶ 0

  on: My thoughts on nan­otech­nol­ogy strat­egy re­search as an EA cause area

  I found this post helpful, since lately I’ve been trying to understand the role of molecular nanotechnology in EA and x-risk discussions. I appreciate your laying out your thinking, but I think full-time effort here is premature.

  Overall, then, adding the above probabilities implies that my guess is that there’s a 4-5% chance that advanced nanotechnology arrives by 2040. Again, this number is very made up and not stable.

  This sounds astonishingly high to me (as does 1-2% without TAI). My read is that no research program active today leads to advanced nanotechnology by 2040. Absent an Apollo program, you’d need several serial breakthroughs from a small number of researchers. Echoing Peter McCluskey’s comment, there’s no profit motive or arms race to spur such an investment. I’d give even a megaproject slim odds—all these synthesis methods, novel molecules, assemblies, information and power management—in the span of three graduate student generations? Simulations are too computationally expensive and not accurate enough to parallelize much of this path. I’d put the chance below 1e-4, and that feels very conservative.

  Here’s a quick attempt to brainstorm considerations that seem to be feeding into my views here: “Drexler has sketched a reasonable-looking pathway and endpoint”, “no-one has shown X isn’t feasible even though presumably some people tried”

  Scientists convince themselves that Drexler’s sketch is infeasible more often than one might think. But to someone at that point there’s little reason to pursue the subject further, let alone publish on it. It’s of little intrinsic scientific interest to argue an at-best marginal, at-worst pseudoscientific question. It has nothing to offer their own research program or their career. Smalley’s participation in the debate certainly didn’t redound to his reputation.

  So there’s not much publication-quality work contesting Nanosystems or establishing tighter upper bounds on maximum capabilities. But that’s at least in part because such work is self-disincentivizing. Presumably some arguments people find sufficient for themselves wouldn’t go through in generality or can’t be formalized enough to satisfy a demand for a physical impossibility proof, but I wouldn’t put much weight on the apparent lack of rebuttals.

• Muireall 24 Jun 2022 14:12 UTC

  18 points

  0 ∶ 0

  in reply to: Ben Snodin’s comment on: My thoughts on nan­otech­nol­ogy strat­egy re­search as an EA cause area

  [2023-01-19 update: there’s now an expanded version of this comment here.]

  Note: I’ve edited this comment after dashing it off this morning, mainly for clarity.

  Sure, that all makes sense. I’ll think about spending some more time on this. In the meantime I’ll just give my quick reactions:

  • On reluctance to be extremely confident—I start to worry when considerations like this dictate that one give a series of increasingly specific/​conjunctive scenarios roughly the same probability. I don’t expect a forum comment or blog post to get someone to such high confidence, but I don’t think it’s beyond reach.

  • We also have different expectations for AI, which may in the end make the difference.

  • I don’t expect machine learning to help much, since the kinds of structures in question are very far out of domain, and physical simulation has some intrinsic hardness problems.

  • I don’t think it’s correct to say that we haven’t tried yet.

  • Some of the threads I would pull on if I wanted to talk about feasibility, after a relatively recent re-skim:

    • We’ve done many simulations and measurements of nanoscale mechanical systems since 1992. How does Nanosystems hold up against those?

    • For example, some of the best-case bearings (e.g. multi-walled carbon nanotubes) seem to have friction worse than Drexler’s numbers by orders of magnitude. Why is that?

    • Edges also seem to be really important in nanoscale friction, but this is a hard thing to quantify ab initio.

    • I think there’s an argument using the Akhiezer limit on $Qf$ products that puts tighter upper bounds on dissipation for stiff components, at least at “moderate” operating speeds. This is still a pretty high bound if it can be reached, but dissipation (and cooling) are generally weak points in Nanosystems.

    • I don’t recall discussion of torsional rigidity of components. I think you can get a couple orders of magnitude over flagellar motors with CNTs, but you run into trouble beyond that.

    • Nanosystems mainly considers mechanical properties of isolated components and their interfaces. If you look at collective motion of the whole, everything looks much worse. For example, stiff 6-axis positional control doesn’t help much if the workpiece has levered fluctuations relative to the assembler arm.

    • Similarly, in collective motion, non-bonded interfaces should be large contributors to phonon radiation and dissipation.

    • Due to surface effects, just about anything at the nanoscale can be piezoelectric/​flexoelectric with a strength comparable to industrial workhorse bulk piezoelectrics. This can dramatically alter mechanical properties relative to the continuum approximation. (Sometimes in a favorable direction! But it’s not clear how accurate simulations are, and it’s hard to set up experiments.)

    • Current ab initio simulation methods are accurate only to within a few percent on “easy” properties like electric dipole moments (last I checked). Time-domain simulations are difficult to extend beyond picoseconds. What tolerances do you need to make reliable mechanisms?

  • In general I wouldn’t be surprised if a couple orders of magnitude in productivity over biological systems were physically feasible for typically biological products (that’s closer to my 1% by 2040 scenario). Broad-spectrum utility is much harder, as is each further step in energy efficiency or speed.

• Muireall 23 Sep 2022 22:44 UTC

  3 points

  0 ∶ 0

  on: The dis­count rate is not zero

  Both examples show how we can act to reduce the catastrophe rate over time, but there are also 3 key risk factors applying upward pressure on the catastrophe rate:

  • The lingering nature of present threats

  • Our ongoing ability to generate new threats

  • Continuously lowering barriers to entry/​access

  In the case of AI, its usually viewed that AI will be aligned or misaligned, meaning this risk is either be solved or not. It’s also possible that AI may be aligned initially, and become misaligned later^[11]. The need for ongoing protection from bad AI would therefore be ongoing. In this scenario we’d need systems in place to stop AI being misappropriated or manipulated, similar to how we guard nuclear weapons from dangerous actors. This is what I term “lingering risk”.

  I just want to flag one aspect of this I haven’t seen mentioned, which is that much of this lingering risk naturally grows with population, since you have more potential actors. If you have acceptable risk per century with 10 BSL-4 labs, the risk with 100 labs might be too much. If you have acceptable risk with one pair of nuclear rivals in cold war, a 20-way cold war could require much heavier policing to meet the same level of risk. I expanded on this in a note here.

Ap­pear­ance of impropriety

• Muireall 18 Jan 2023 20:43 UTC

  18 points

  5 ∶ 0

  on: Do­ing EA Better

  More specifically, EA shows a pattern of prioritising non-peer-reviewed publications – often shallow-dive blogposts^[36] – by prominent EAs with little to no relevant expertise.

  This is my first time seeing the “climate change and longtermism” report at that last link. Before having read it, I imagined the point of having a non-expert “value-aligned” longtermist applying their framework to climate change would be things like

  • a focus on the long-run effects of climate change

  • a focus on catastrophic scenarios that may be very unlikely but difficult to model or quantify

  Instead, the report spends a lot of time on

  • recapitulation of consensus modeling (to be clear, this is a good thing that’s surprisingly hard to come by), which mainly goes out to 2100

  • plausible reasons models may be biased towards negative outcomes, particularly in the most likely scenarios

  The two are interwoven, which weakens the report even as a critical literature review. When it comes to particular avenues for catastrophe, the analysis is often perfunctory and dismissive. It comes off less as a longtermist perspective on climate change than as having an insider evaluate the literature because only “we” can be trusted to reason well.

  I don’t know how canonical that report has become. The reception in the thread where it was posted looks pretty critical, and I don’t mean to pile on. I’m commenting because this post links the report in a way that looks like a backhanded swipe, so once I read it myself I felt it was worth sketching out my reaction a bit further.

• Muireall 21 Jan 2023 18:59 UTC

  4 points

  3 ∶ 1

  in reply to: Bob Jacobs’s comment on: CEA state­ment on Nick Bostrom’s email

  Differential response within the survey is again as bad.

  The response rate for the survey as a whole was about 20% (265 of 1345), and below 8% (102) for every individual question on which data was published across three papers (on international differences, the Flynn effect, and controversial issues).

  Respondents attributed the heritability of U.S. black-white differences in IQ 47% on average to genetic factors. On similar questions about cross-national differences, respondents on average attributed 20% of cognitive differences to genes. On the U.S. question, there were 86 responses, and on the others, there were between 46 and 64 responses.

  Steve Sailer’s blog was rated highest for accuracy in reporting on intelligence research—by far, not even in the ballpark of sources that got more ratings (those sources being exactly every mainstream English-language publication that was asked about). It was rated by 26 respondents.

  The underlying data isn’t available, but this is all consistent with the (known) existence of a contingent of ISIR conference attendees who are likely to follow Sailer’s blog and share strong, idiosyncratic views on specifically U.S. racial differences in intelligence. The survey is not a credible indicator of expert consensus.

  (More cynically, this contingent has a history of going to lengths to make their work appear more mainstream than it is. Overrepresenting them was a predictable outcome of distributing this survey. Heiner Rindermann, the first author on these papers, can hardly have failed to consider that. Of course, what you make of that may hinge on how legitimate you think their work is to begin with. Presumably they would argue that the mainstream goes to lengths to make their work seem fringe.)

• Muireall 19 Feb 2023 4:11 UTC

  6 points

  11 ∶ 0

  in reply to: EliezerYudkowsky’s comment on: Peo­ple Will Some­times Just Lie About You

  named the “Gell-Mann Amnesia Effect” by Richard Feynman

  Aside, but that was Michael Crichton.

• Muireall 19 Feb 2023 15:47 UTC

  30 points

  18 ∶ 8

  on: Peo­ple Will Some­times Just Lie About You

  I also don’t want to turn this into “and now Aella hijacks this issue to talk about personal social drama where she picks through the details of every rumor in order to convince you she’s not a terrible person”. But still, it’s hard to convey exactly how this works without examples, and specific instances of this happening to me are what have so strongly updated my views, so I’m going to pick just a few.

  I appreciate your taking care here, and concreteness is valuable in conversations like this. Even so, I’m not sure what I can take away with respect to EA from this middle half of your post. I don’t know you apart from your two prior comments on this forum. It’s hard to distinguish your account here from what I’d expect to read from a verbally skilled actor who’s fluent in the strategic use of bright lines to avoid getting pinned down. Narrative may be the best remaining means to warn against such an actor, and by construction that narrative can just as well be framed as mere insinuation or hostile framing. So it’s hard to treat statements like yours as evidence without a foundation of trust in involved individuals or even just familiarity with their community. Separately, the specifics of your analysis of statements about you seem to me at best tenuously related to coverage of EA, even in your transition back to discussing it.

  I’m left feeling that this wasn’t an appropriate forum for that section of your post. (I don’t object to your defending yourself—I doubt I’d have commented if you’d left this section at the link to your original response to your friend’s statements. Similarly, I’m happy to encourage critical reading.) I seem to be in the minority—you do say you’re “mini-famous”, so maybe more in-the-loop readers have more to go on—but I do wonder if others have thoughts along or outright against these lines.

Con­sid­er­a­tions on trans­for­ma­tive AI and ex­plo­sive growth from a semi­con­duc­tor-in­dus­try per­spec­tive

• Muireall 1 Jun 2023 1:54 UTC

  3 points

  0 ∶ 0

  on: A com­pute-based frame­work for think­ing about the fu­ture of AI

  In fact, algorithmic progress has been found to be similarly as important as compute for explaining progress across a variety of different domains, such as Mixed-Integer Linear Programming, SAT solvers, and chess engines—an interesting coincidence that can help shed light on the source of algorithmic progress (Koch et al. 2022, Grace 2013). From a theoretical perspective, there appear to be at least three main explanations of where algorithmic progress ultimately comes from:

  1. Theoretical insights, which can be quickly adopted to improve performance.

  2. Insights whose adoption is enabled by scale, which only occurs after there’s sufficient hardware progress. This could be because some algorithms don’t work well on slower hardware, and only start working well once they’re scaled up to a sufficient level, after which they can be widely adopted.

  3. Experimentation in new algorithms. For example, it could be that efficiently testing out all the reasonable choices for new potential algorithms requires a lot of compute.

  I always feel kind of uneasy about how the term “algorithmic progress” is used. If you find an algorithm with better asymptotics, then apparent progress depends explicitly on the problem size. MILP seems like a nice benchmark because it’s NP-hard in general, but then again most(?) improvements are exploiting structure of special classes of problems. Is that general progress?

  One important factor affecting our ability to measure algorithmic progress is the degree to which algorithmic progress on one task generalizes to other tasks. So far, much of our data on algorithmic progress in machine learning has been on ImageNet. However, there seem to be two ways of making algorithms more efficient on ImageNet. The first way is to invent more efficient learning algorithms that apply to general tasks. The second method is to develop task-specific methods that only narrowly produce progress on ImageNet.

  We care more about the rate of general algorithmic progress, which in theory will be overestimated by measuring the rate of algorithmic progress on any specific narrow task. This consideration highlights one reason to think that estimates overstate algorithmic progress in a general sense.

  I definitely agree with the last sentence, but I’m still not sure how to think about this. I have the impression that, typically, some generalizable method makes a problem feasible, at which point focused attention on applying related methods to that problem drives solving it towards being economical. I suppose for this framework we’d still care more about the generalizable methods, because those trigger the starting gun for each automatable task?

• Muireall 2 Jun 2023 21:14 UTC

  2 points

  0 ∶ 0

  on: Con­sid­er­a­tions on trans­for­ma­tive AI and ex­plo­sive growth from a semi­con­duc­tor-in­dus­try per­spec­tive

  I’ve also now put code for Appendix C.3 on GitHub.

• Muireall 2 Jun 2023 21:44 UTC

  1 point

  0 ∶ 0

  in reply to: Erin’s comment on: Erin’s Shortform

  I think FQxI usually gets around 200 submissions for its essay contests where the entire pot is less than the first prize here. I wouldn’t be surprised if Open Phil got over 100 submissions.

• Muireall 7 Jun 2023 1:32 UTC

  6 points

  2 ∶ 0

  on: Trans­for­ma­tive AGI by 2043 is <1% likely

  I’d have to think more carefully about the probabilities you came up with and the model for the headline number, but everything else you discuss is pretty consistent with my view. (I also did a PhD in post-silicon computing technology, but unlike Ted I went right into industry R&D afterwards, so I imagine I have a less synoptic view of things like supply chains. I’m a bit more optimistic, apparently—you assign <1% probability to novel computing technologies running global-scale AI by 2043, but I put down a full percent!)

  The table “Examples transistor improvements from history (not cherry-picked)” is interesting. I agree that the examples aren’t cherry picked, since I had nearly the same list (I decided to leave out lithography and included STI and the CFET on imec’s roadmap), but you could choose different prototype dates depending on what you’re interested in.

  I think you’ve chosen a fairly relaxed definition for “prototype”, which is good for making the point that it’s almost certain that the transistors of 2043 will use a technology we already have a good handle on, as far as theoretical performance is concerned.

  Another idea would be to follow something like this IRDS table that splits out “early invention” and “focused research”. They use what looks like a stricter interpretation of invention—they don’t explain further or give references, but I suspect they just have in mind more similarity to the eventual implementation in production. (There are still questions about what counts, e.g., 1987 for tri-gate or 1998 for FinFET?) That gives about 10–12 years from focused research to volume production.

  So even if some unforseeable breakthrough is more performant or easily scalable than what we’re currently thinking about, it still looks pretty tough to get it out by 2043.

• Muireall 7 Jun 2023 1:38 UTC

  3 points

  0 ∶ 0

  in reply to: Muireall’s comment on: Trans­for­ma­tive AGI by 2043 is <1% likely

  (Here’s my submission—I make some similar points but don’t do as much to back them up. The direction is more like “someone should try taking this sort of thing into account”—so I’m glad you did!)

• Muireall 12 Jun 2023 13:57 UTC

  16 points

  3 ∶ 0

  on: Cri­tiques of promi­nent AI safety labs: Conjecture

  Is there a canonical discussion of what you call “race dynamics” somewhere? I can see how proliferating firms and decentralized communities would “mak[e] potential moratoriums on capabilities research much harder (if not impossible) to enforce”, but it’s less clear to me what that means for how quickly capabilities advance. Is there evidence that, say, the existence of Anthropic has led to increased funding for OpenAI?

  In particular, one could make the opposite argument—competition, at least intra-nationally, slows the feedback cycle for advancing capabilities. For example, a lot of progress in information technology seems to have been driven by concentration of R&D into Bell Labs. If the Bell monopoly had been broken up sooner, would that have accelerated progress? If some publicly-funded entity had provided email and internet search services, would Google have reached the same scale?

  Meanwhile, training leading-edge models is capital intensive, and competing firms dilute available funding across many projects. Alternative commercial and open-source models drive potential margins down. Diminished prospects for monopoly limit the size and term of bets that investors are willing to make.

  I don’t know which way the evidence actually falls, but there seems to be a background assumption that competition, race dynamics, and acceleration of progress on capabilities always go hand in hand. I’d be very interested to read more detailed justifications for that assumption.