I think this would be most meaningful if opting out resulted in another person being sent codes instead. (Or a larger pool of people was invited to opt in and then a random subset of those who did were given codes.)
[I don’t really like the ritual at the moment but I like hill-climbing towards finding better versions of it]
Thanks a lot for sharing this denise. Here are some thoughts on your points.
On your point about moral realism, I’m not sure how that can be doing much work in an argument against longtermism specifically, as opposed to all other possible moral views. Moral anti-realism implies that longtermism isn’t true, but then it also implies that near-termism isn’t true. The thought seems to be that there could only be an argument that would give you reason to change your mind if moral realism were true, but if that were true, there would be no point in discussing arguments for and against longtermism because they wouldn’t have justificatory force.
Your argument suggests that you find a person-affecting form of utilitarianism most plausible. But to me we should not reach conclusions about ethics on the basis of what you find intuitively appealing without considering the main arguments for and against these positions. Person-affecting views have lots of very counter-intuitive implications and are actually quite hard to define.
I don’t think it is true that the case for longtermism rests on the total view. As discussed in the Greaves and MacAskill paper, many theories imply longtermism.
Your view that humanity is not super-awesome seems to me compatible with longtermism. The ‘not super-awesome’ critique attacks a premise of our strand of longtermism which is especially focused on ensuring human survival. But other forms of longtermism do not rely on these premises. For example, if you don’t think that humanity is super awesome, then focusing on values change looks a good bet, as does reducing s-risks.
I’m not sure your point that ‘the future will not be better than today’ hits the mark. More precisely, I think you want to say that ‘today the world is net bad and the future will be as well’. It could be true that the future is not better than today but that the future is extremely good. In that case, reducing extinction risks would still have astronomical expected value.
Independently of point 5, again I don’t think one needs to hold that the future is good for longtermism to be true. Suffering-focused people are longtermists but don’t think that the future is good. Totalists could also think that the future is not good in expectation. But still even if the future is bad in expectation, if the variance of possible realisable states of the future is high, that makes affecting the trajectory of the future extremely important.
On existential security, this is a good and underdiscussed point. I hadn’t thought about this much until recently, but after looking into it I became quite convinced that a period of existential security is very likely provided that we survive catastrophic risks and avoid value lock-in. My thoughts are not original and owe a lot to discussions with Carl Shulman, Max Daniel, Toby Ord, Anders Sandberg and others. One point is that biorisk and AI risk are transition risks not state risks. The coordination problems involved in solving them are so hard that once they are solved, they stay solved. To ensure AI safety, one has to ensure stability in coordination dynamics for millions of subjective years of strategic interplay between advanced AI systems. Once we can do that, then we have effectively solved all coordination problems. Solving biorisk is also very hard if you think there will be strongly democratised biotech. If you solve that and build digital minds to explore the galaxy, then you basically eliminate biorisk. If we go to the stars, then we at least avoid earth-bound GCRs. In short, we would have huge technological power and have solved the hardest coordination problems. If you buy limits to growth arguments, you might also think that tech progress will slow down, and so catastrophic risks will fall, as they are driven by tech progress. All of this suggests that conditional on survival of the time of perils, the probability that the future is extremely long is >>10%. So, the probability is not Pascalian.
You seem to suggest that we cannot influence the long-term future with the exception of what you call ‘lock-in events’ like extinction and permanent collapse, which are attractor states that could lock in a state of affairs in the next 100 years. I suppose another one would be AI-enabled permanent lock-in of bad values. But these are the main things that longtermists are working on, so I don’t see how this could be a criticism of longtermism.
I’m don’t think that the inference from ‘I don’t know how to influence the future’ to ‘donate to AMF’ follows. If you buy these cluelessness arguments (I don’t personally), then it seems like two obvious things to do would be to give later to prepare for when a potential lock in event rears its head. So you could invest in stocks, or you could grow the movement so that it is ready to deal with a lock in event. If you are very uncertain about how to affect the longterm future, but accept that the future is potentially extremely valuable, then this is the strongest argument ever for ‘more research needed’. If there is a big asteroid heading our way but we currently feel very unsure about how to affect that, but there are some smart people who think they can stop the asteroid, the correct answer seems to me to be “let’s put tonnes of resources into figuring out how to stop this asteroid” not “let’s donate to AMF”.
To the extent average utilitarianism is motivated by avoiding the Repugnant Conclusion, I suspect that most average utilitarians would be as disturbed by aggregating over time as they are by aggregating within a generation, since we can establish a Repugnant Conclusion over times pretty straightforwardly. That said, to the extent intuitions differ when we aggregate over times, I can see that this could pose a challenge to average utilitarians.
I can’t recall any work on this argument off the top of my head, but I did recently come across a hint of a related argument directed against distributive egalitarianism. From https://globalprioritiesinstitute.org/economic-inequality-and-the-long-term-future-andreas-t-schmidt-university-of-groningen-and-daan-juijn-ce-delft/ : “An additional question is whether distributive egalitarianism should extend to inequalities across generations.” Which links to a footnote: “One of us elsewhere argues that distributive egalitarianism is implausible, because its extension to intergenerational distributions is necessary yet implausible [redacted].” Not sure why the citation is redacted, but I think “one of us” refers to Andreas Schmidt. Of course, extending the analysis to future generations threatens average utilitarianism and distributive egalitarianism in different ways. But the fact that both are threatened by this type of argument suggests to me that a lot of moral theories ought to be stress-tested against “what about across generations?” arguments. I agree that there’s an interesting set of questions here.
I’d fairly strongly disagree with that take. I think it’s an extremely reasonable assumption that a somewhat cartoony red button someone put at the top of a website deliberately does not do harm to press. Someone deliberately chose to put it there, and most features on websites are optimised for user interaction. This only looks unreasonable within the strong frame of having cultural context about Petrov Day
Yes, I agree
I interpreted you as proposing doing an unilateralist action to demonstrate to others the harm of unilateralist actions. Apologies if I misread.
I have been wondering the same. However, given that OpenAI’s “AI and Compute” inclusion criteria are also a bit vague, I’m having a hard time which of our data points would fulfill their criteria.
In general, I would describe our dataset matching the same criteria because:
“relatively well known” equals our “lots of citations”.
“used a lot of compute for their time” equals our dataset if we exclude outliers from efficient ML models.
There’s a recent trend in efficient ML models that achieve similar performance by using less compute for inference and training (those models are then used for e.g., deployment on embedded systems or smartphones).
“gave enough information to estimate the compute”: We also rely on estimates from us or the community based on the information available in the paper. For a source of the estimate see the note on the cell in our dataset.
We’re working on gathering more compute data by directly asking researchers (next target n=100) .
I’d be interested in discussing more precise inclusion criteria. As I say in the post:
Also, it is unclear on which models we should base this trend. The piece AI and Compute also quickly discusses this in the appendix. Given the recent trend of efficient ML models due to emerging fields such as Machine Learning on the Edge, I think it might be worthwhile discussing how to integrate and interpret such models in analyses like this — ignoring them cannot be the answer.
I interpreted your comment as saying that I was “lambasting the foibles of being a well intentioned unilateralist”, and that I should not be doing so. If that was not the intent I’m glad.
The lesson people I would want people to learn is “I might not have considered all the reasons people might do stuff”. See comment below.
This is closer, I think the framing I might have had in mind is closer to:
people underestimate the probability of tail risks.
I think one of the reasons why is that they don’t appreciate the size of the space of unknown unknowns (which in this case includes people pushing the button for reasons like this).
causing them to see something from the unknown unknown space is therefore useful.
I think last year’s phishing incident was actually a reasonable example of this. I don’t think many people would have put sufficiently high probability on it happening, even given the button getting pressed.
Regardless of overarching opinions you may or may not have about the unilateralist’s curse, I think Petrov Day is a uniquely bad time to lambast the foibles of being a well-intentioned unilateralist.
I worry that people are updating in exactly the wrong way from the events, possibly to fit preconceived ideas of what’s correct.
I think either you misinterpreted my comment or I misinterpreted yours.
This wasn’t intended as a “you should have felt sorry for me if I’d done a unilateralist thing without thinking”.
I’m genuinely confused how you could have gotten that interpretation from my comment.
It was intended as a way of giving more information about the probability of unilateralist action than people would otherwise have had, which seems well within the spirit of the day.
So to be clearer, Petrov here should naively be read as a well-intentioned unilateralist. He happened to be right, and reasonable people can disagree about whether he was wrong but lucky or right all along. Regardless, I think it’s not very much in the spirit of the day to talk about or act out all the harms of being a well-intentioned unilateralist, though if you wish to do so, more power to you.
I also think it’s noteworthy that in the situation being celebrated the ability to resist social pressure was pointing in the opposite direction to the way it goes here, which seems like a problem with the current structure.
I agree, and have complained about this before. I’m also complaining about it now, in case that was not previously clear.
Thanks, agreed!
Thanks for the comment! The tl;dr is that getting nutrients distributed is a closely related issue of “viscosity,” where sufficiently thickly packed cells will have effectively ~0 fluid flow, and that the theoretical upper limit for cells (~infinite viscosity at ~0.65) is close to and slightly lower than sphere packing limits.
Below is some notes from being nerd-sniped by this comment, no guarantees that it’s at all good or useful. (Note that I certainly didn’t know things to that level of granularity before your comment, so it was helpful for you to do this spot check!)
-
I agree that sphere packing is a simplifying assumption, as cells as both deformable and not spheres. However, it’s not a crazy one to start with, and I think the practical bounds are much lower.
The problem of getting nutrients distributed is a different issue, of course
Actually, a closely related issue is viscosity, or how difficult it is for fluids to flow through something. The theoretical upper bound is when viscosity approaches ∞, where you can’t get any nutrients at all.
My impression is that the question of the theoretical upper bound of packing with soft objects but other constraints like shear in a viscous solution has been one that interested CHO scientists and other people for over a century, arguably starting with literally Einstein.
Humbird uses φmax=0.65 as the theoretical upper limit for soft spheres as footnote 35 on page 14, citing a PhD thesis. The thesis lays out the problem pretty cleanly (emphasis mine):
There must come a time, as more and more particles are added, which gives continuous three-dimensional contact throughout the suspension, thus making flow impossible, i.e. the viscosity tends to infinity. This will be also correlated with the existence of the yield stress discussed later.
The volume fraction at which this happens is called the maximum packing fraction # and its value will depend on the arrangement of the particles. Maximum packing fractions thus range from approximately 0.5 to 0.75. The maximum packing fraction, as well as being controlled by the type of packing, is very sensitive to particle-size distribution and particle shape.
The thesis then gives the relevant numbers in pg 81-82:
As in the case of suspensions, we define a maximum packing fraction # (which is usually 0.64 or even 0.74 for solid spheres in a face-centered-cubic crystal), depending on cell elasticity, i.e. their compactness [Quemada 1998]. Due to the presence of soft spherical cells, it is expected that the value of # will be in this range. # is determined using the reduced viscosity plot as a function of ( at a shear rate of 10 2 s −1 , = 0.0013 Pa.s the solvent viscosity). In our case, this data is found to match the well-known equation proposed by Krieger and Dougherty [Krieger 1959], this providing the value # = 0.65 (Fig. 4.5).
The surrounding thesis has a lot of science that I don’t feel confident in evaluating, and math that I think (perhaps arrogantly) that I am capable of evaluating, but (unfortunately) like many other problems I’ve encountered in this project, I chose not to.
Obviously I think there’s a reasonably high chance that numbers from a random PhD thesis might be wrong, though I’m somewhat comforted by other lines of attack (including sphere packing) that bounds it at similar numbers . That said, my best guess is that small disagreements here will not change anybody’s bottom-line cause prioritization of cultured meat, though perhaps it might be sometimes relevant for cultured meat implementation.
Perhaps the idea is that it should be a symbolic reminder that trusted community members could do bad things, rather than evidence for that proposition?
Yeah, that title/framing seems fine to me
But if you actually should press the button, and do so because you correctly understand why you should, then people shouldn’t learn the lesson “people will do wild crazy stuff out of misunderstandings or malice”, because that won’t be what happened.
Yeah I guess you could read what I’m saying as that I actually think I should have pressed it for these reasons, but my moral conviction is not strong enough to have borne the social cost of doing so.
One read of that is that the community is strong enough in its social pressure to quiet bad actors like me from doing stupid harmful stuff we think is right.
Another is that social pressure is often enough to stop people from doing the right thing, and that we should be extra grateful to Petrov, and others in similar situations, because of this.
Either reading seems reasonable to discuss today.
Possible features:
some way to be unilateralist in a defensive way
Sometimes it says the other site has launched and is wrong
As an update, I’ve decided not to push the button this year.
However I tentatively believe that this event in its current form is net negative for community health. I also see that a lot of the issues were predictable from last year’s comments but were not addressed.
For that reason, if next year’s event hasn’t been changed enough to be positive for community health (ie clear purpose, shared understanding around social pressure, opt-in if it could have any impact on people’s friendships) I am pre-committing to pressing the button. If I’m not invited I’ll encourage someone else to press it.
If you make it opt-in, please do feel free to invite me—really! There are a lot of versions of this that I think I’d enjoy and I promise I’m not an entirely uncooperative person