I am a generalist quantitative researcher. I am open to volunteering and paid work. I welcome suggestions for posts. You can give me feedback here (anonymously or not).
Vasco Grilošø
Karma: 10,740
Thanks for sharing, Michael. If I was as concerned about AI risk as @EliezerYudkowsky, I would use practically all the additional earnings (e.g. above Nateās 235 k$/āyear; in reality I would keep much less) to support efforts to decrease it. I would believe spending more money on personal consumption or investments would just increase AI risk relative to supporting the most cost-effective efforts to decrease it.
Hi Saulius.
I guess thereās a 45% chance that blocking a farm delays production by just one month, as the market might adjust quickly to meet demand elsewhere. I guess thereās a 50% chance of a one-year delay. And I guess thereās a 5% chance that stopping a farm leads to 10 fewer years of chicken farmingāon the assumption that, in some cases, limiting supply can reduce long-term demand rather than simply shifting production. These assumptions yield an expected delay of one year, though this is highly uncertain.
I think this significantly underestimates the reduction in farm-years per broiler farm blocked. Gemini says random broiler farms in Poland have an expected lifespan of 31.2 years. In addition, I think one can reasonably estimate that shifting the supply curve of chicken meat leftwards by 1 kg decreases its demand/āsupply by 0.24 kg. So I calculate a reduction of 7.49 farm-years per broiler farm blocked (= 31.2*0.24), 7.49 times your estimate.
Wow Iām mind blown that Yudowsky pays himself that much. If only because it leaves him open to criticisms likt these. I still donāt think the financial incentives are as strong as for people starting an accellerationist company, but its a fair point.
I think the strength of the incentives to behave in a given way is more proportional to the resulting expected increase in welfare than to the expected increase in net earnings. Individual human welfare is often assumed to be proportional to the logarithm of personal consumption. So a given increase in earnings increases welfare less for people earning more. In addition, a 1 % chance of earning 100 times more (for example, due to oneās company being successful) increases welfare less than a 100 % chance of earning 100 % more. More importantly, there are major non-financial benefits for Yudowsky, who is somewhat seen as a prophet in some circles.
Hi Nick.
Although their arguments are reasonable, my big problem with this is that these guys are so motivated that I find it hard to read what they write in good faith.
People who are very invested in arguing for slowing down AI development, or decreasing catastrophic risk from AI, like many in the effective altruism community, will also be happier if they succeed in getting more resources to pursue their goals. However, I believe it is better to assess arguments on their own merits. I agree with the title of the article that it is difficult to do this. I am not aware of any empirical quantitative estimate of the risk of human extinction resulting from transformative AI.
I would consider driving people to delusion and suicide, killing people for self-preservation and even Hitler the man himself to be at least a somewhat āalienā style of evil.
I agree those actions are alien in the sense of deviating a lot from what random people do. However, I think this is practically negligible evidence about the risk of human extinction.
UnĀfalsifiĀable stoĀries of doom
Thanks for this work. I find it valuable.
If AIs are conscious, then they likely deserve moral consideration
AIs could have negligible welfare (in expectation) even if they are conscious. They may not be sentient even if they are conscious, or have negligible welfare even if they are sentient. I would say the (expected) total welfare of a group (individual welfare times population) matters much more for its moral consideration than the probability of consciousness of its individuals. Do you have any plans to compare the individual (expected hedonistic) welfare of AIs, animals, and humans? You do not mention this in the section āWhatās nextā.
The choice of a prior is often somewhat arbitrary and intended to reflect a state of ignorance about the details of the system. The final (posterior) probability the model generates can vary significantly depending on what we choose for the prior. Therefore, unless we are confident in our choices of priors, we shouldnāt be confident in the final probabilities.
Do you have any ideas for how to decide on the priors for the probability of sentience? I agree decisions about priors are often very arbitrary, and I worry they will have significantly different implications.
[...] We report what each perspective concludes, then combine these conclusions based on how credible experts find each perspective.
[...]
Which theory of consciousness is right matters a lot. Because different stances give strikingly different judgments about the probability of LLM consciousness, significant changes in the weights given to stances will yield significant differences in the results of the Digital Consciousness Model. [...]
I like that your report the results for each perspective. People usually give weights that are at least 0.1/āānumber of modelsā, which are not much smaller than the uniform weight of 1/āānumber of modelsā, but this could easily lead to huge mistakes. As a silly example, if I asked random people with age 7 about whether the gravitational force between 2 objects is proportional to ādistanceā^-2 (correct answer), ādistanceā^-20, or ādistanceā^-200, I imagine I would get a significant fraction picking the exponents of ā20 and ā200. Assuming 60 % picked ā2, 20 % picked ā20, and 20 % picked ā200, a respondant may naively conclude the mean exponent of ā45.2 (= 0.6*(-2) + 0.2*(-20) + 0.2*(-200)) is reasonable. Alternatively, a respondant may naively conclude an exponent of ā9.19 (= 0.933*(-2) + 0.0333*(-20) + 0.0333*(-200)) is reasonable giving a weight of 3.33 % (= 0.1/ā3) to each of the 2 wrong exponents, equal to 10 % of the uniform weight, and the remaining weight of 93.3 % (= 1 ā 2*0.0333) to the correct exponent. Yet, there is lots of empirical evidence against the exponents of ā45.2 and ā9.19 which the respondants are not aware of. The right conclusion would be that the respondants have no idea about the right exponent, or how to weight the various models because they would not be able to adequately justify their picks. This is also why I am sceptical that the absolute value of the welfare per unit time of animals is bound to be relatively close to that of humans, as one may naively infer from the welfare ranges Rethink Priorities (RP) initially presented, or the ones in Bob Fischerās book about comparing welfare across species, where there seems to be only 1 line about the weights. āWe assigned 30 percent credence to the neurophysiological model, 10 percent to the equality model, and 60 percent to the simple additive modelā.
Mistakes like the one illustrated above happen when the weights of models are guessed independently of their output. People are often sensitive to astronomical outputs, but not to the astronomically low weights they imply. How do you ensure the weights of the models to estimate the probability of consciousness are reasonable, and sensitive to their outputs? I would model the weights of the models as very wide distributions to represent very high model uncertainty.
Thanks, Michael.
Thanks for the update. Do you plan to publish any cost-effectiveness analyses of grants you have made?
Thanks for the post, Carl.
As funding expands in focused EA priority issues, eventually diminishing returns there will equalize with returns for broader political spending, and activity in the latter area could increase enormously: since broad political impact per dollar is flatter over a large range political spending should either be a very small or very large portion of EA activity
Great point.
Agreed, David.
instead believe only humans and animals experiencing well-being is good
Nitpick. I would say humans, animals, microorganisms, and digital beings.
Thanks for sharing, Brendan. You may want to try RoastMyPost, and then share any feedback you may have with @Ozzie Gooen.
Thanks for the post, Joey.
Doing neglectedness right
āConsidering the main two areas I am considering, food systems climate is more neglected than clean energy climate.ā
I think this sort of comparison makes a lot of sense. It is trying to look at the real oppperuntiy cost of what else would be supported by people (or yourself) considering the area. [...]
I think you are suggesting people say i) āX is more neglected than Yā if ii) āX is more cost-effective than Y at the marginā. I believe it would be better for people to simply say ii) as applied to the relevant context. For example, that funding X with 10 k$ would save more lives than funding Y by the same amount. As you pointed out, i) could be interpreted in many different ways, and therefore can lead to misunderstandings.
I think a decent proxy for neglect is: what is the group right on the edge?
This is very unclear to me. For individual welfare per fully-healthy-animal-year proportional to āindividual number of neuronsā^āexponentā, and a range of 0.5 to 1.5 for āexponentā, which I believe covers reasonable best guesses, I estimate that the Shrimp Welfare Projectās (SWPās) Humane Slaughter Initiative (HSI) has increased the welfare of shrimps via increasing the adoption of electrical stunning 0.00167 (= 2.06*10^-5/ā0.0123) to 1.67 k (= 20.6/ā0.0123) times as cost-effectively as GiveWellās top charities increase the welfare of humans. So I can easily see HSI increasing the welfare of shrimps much more or less cost-effectively than GiveWellās top charities increase the welfare of humans.
Thanks for the post, Tristan. I am pessimistic about finding interventions that robustly increase welfare (in expectation) accounting for soil animals and microorganisms. I do not think electrically stunnning qualifies, although it decreases intense pain experienced by the target beneficiaries, and the ratio between effects on target beneficiaries and other organisms is much smaller than for the vast majority of interventions.
TL;DR: Almost all suffering in the world today is experienced by wild animals
This is unclear to me. I estimated the above is not the case for individual welfare per fully-healthy-animal-year proportional to āindividual number of neuronsā^āexponentā, and āexponentā = 1.5, which is the upper bound of the range of 0.5 to 1.5 that I guess covers reasonable best guesses. For that exponent, I calculate the absolute value of the total welfare of wild birds, mammals, and finfishes is 4.12 % of the total welfare of humans, and that the absolute value of the total welfare of soil ants, termites, springtails, mites, and nematodes is 1.89 % of the total welfare of humans.
The bet is neutral for both parties if the Metaculusā question resolves ambiguously. In this case, no transfer of money would happen. A higher probability of the question resolving ambiguously decreases the expected value of the bet for both parties, but this could be mitigated by increasing the potential benefits.
My bet does not depend on Metaculusā definition of āweak AGIā. I rely on Metaculusā definition of SAI given in a question about the time from āweak AGIā until SAI. However, the bet I suggested is just about the date of SAI.
I share some thoughts below. I still have very little confidence in my modelling of the individual welfare per fully-healthy-organism-year. However, I think this strengthens my redommendation of decreasing the uncertainty about how the individual welfare per unit time of different organisms and digital systems compares with that of humans.
Do you have an underlying causal model for why BMR^Exp1 and IW/āFHAY could be generally correlated beyond being correlated through the indirect connection BMR^Exp2 ā Neuron Count and NC^Exp2 ā Welfare ?
Producing welfare requires energy, and the output of a CobbāDouglas production function, which is typically used in economics, is proportional to āinput 1ā^āexponent 1ā*āinput 2ā^āexponent 2ā*...*āinput Nā^āexponent Nā. A greater energy consumption also means more room to process information, and I think this is necessary to produce welfare.
Nitpick. The exponents in the last sentence above should be different.
Do you have theories of consciousness that could give a model for how organisms without neurons could have hedonic experiences?
Here are Geminiās 10 most credible theories of consciousness that do not require biological neurons, and 10 most credible that predict bacteria have a probability of consciousness above exactly 0.
Naively I would assume that the correlation between BMR^Exp1 and IW/āFHAY is completely explained through the connection through the neuron count variable and extending the model to organisms without neurons would be fallacious.
You may well be right. Yet, I believe my recommendation stands even if one is certain that all organism without biological neurons have an expected welfare per unit time of exactly 0.
Hi Jan.
Are you open to bets about this? I would be happy to bet 10 k$ that Anthropic would not pay e.g. 3 billion $ for Yudkowsky and Soares to endorse their last model as good. We could ask the marketing team at Anthropic or marketing experts elsewhere. I am not officially proposing a bet just yet. We would have to agree on a concrete operationalisation.