After AGI is developed, we get an amount of utility equal to the logarithm of our remaining capital.
This doesnât seem to me like an appropriate assumption to make if analyzing this from an altruistic perspective.
If friendly AGI is developed, and assuming it can handle all future x-risks for us, then donât we just get utility equal to our cosmic endowment? We get a future of astronomical value. The amount of capital we have leftover affects how quickly we can begin colonization by a small amount, but isnât that a pretty trivial effect compared to the probability of actually getting friendly AGI?
It seems to me then that we should roughly be following Bostomâs maxipok rule to âMaximise the probability of an âOK outcomeâ, where an OK outcome is any outcome that avoids existential catastrophe.â In toy model terms, this would be maximizing the probability of friendly AGI without regard for how much capital is leftover after AGI.
Am I correct that thatâs not what your model is doing? If so, why do you think doing what your model is doing (with the 5th assumption quoted above) is more appropriate?
(With your assumption, it seems the model will say that we should spend less on AGI than we would with the assumption Iâm saying is more appropriate to make (maximize probability of friendly AGI), since your model will accept a marginally higher probability of x-risk in exchange for a sufficiently higher amount of capital remaining after friendly AGI.)
I donât have any well-formed opinions about what the post-AGI world will look like, so I donât think itâs obvious that logarithmic utility of capital is more appropriate than simply trying to maximize the probability of a good outcome. The way you describe it is how my model worked originally, but I changed it because I believe the new model gives a stronger result even if the model is not necessarily more accurate. I wrote in a paragraph buried in Appendix B:
In an earlier draft of this essay, my model did not assign value to any capital left over after AGI emerges. It simply tried to minimize the probability of extinction. This older model came to the same basic conclusionânamely, shorter timelines mean we should spend faster. (The difference was that it spent a much larger percentage of the budget each decade, and under some conditions it would spend 100% of the budget at a certain point.[5]) But I was concerned that the older model trivialized the question by assuming we could not spend our money on anything but AI safety researchâobviously if thatâs the only thing we can spend money on, then we should spend lots of money on it. The new model allows for spending money on other things but still reaches the same qualitative conclusion, which is a stronger result.
It seems to me that your concern âthat the older model trivialized the question by assuming we could not spend our money on anything but AI safety researchâ could be addressed by dividing existing longtermist or EA capital up into one portion to be spent on AI safety and one portion to be spent on other causes. Each capital stock can then be spent at independent rates according to the value of availabkr giving opportunities in their respective cause areas.
Your model already makes the assumption:
Prior to the emergence of AGI, we donât want to spend money on anything other than AI safety research.
And then:
The new model allows for spending money on other things [but only after AGI]
It just seems like a weird constraint to say that with one stock of capital you only want to spend it on one cause (AI safety) before some event but will spend it on any cause after the event.
Iâm not sure that I can articulate a specific reason this doesnât make sense right now, but intuitively I think your older model is more reasonable.
The reason I made the model only have one thing to spend on pre-AGI is not because itâs realistic (which it isnât), but because it makes the model more tractable. I was primarily interested in answering a simple question: do AI timelines affect giving now vs. later?
Model assumption:
This doesnât seem to me like an appropriate assumption to make if analyzing this from an altruistic perspective.
If friendly AGI is developed, and assuming it can handle all future x-risks for us, then donât we just get utility equal to our cosmic endowment? We get a future of astronomical value. The amount of capital we have leftover affects how quickly we can begin colonization by a small amount, but isnât that a pretty trivial effect compared to the probability of actually getting friendly AGI?
It seems to me then that we should roughly be following Bostomâs maxipok rule to âMaximise the probability of an âOK outcomeâ, where an OK outcome is any outcome that avoids existential catastrophe.â In toy model terms, this would be maximizing the probability of friendly AGI without regard for how much capital is leftover after AGI.
Am I correct that thatâs not what your model is doing? If so, why do you think doing what your model is doing (with the 5th assumption quoted above) is more appropriate?
(With your assumption, it seems the model will say that we should spend less on AGI than we would with the assumption Iâm saying is more appropriate to make (maximize probability of friendly AGI), since your model will accept a marginally higher probability of x-risk in exchange for a sufficiently higher amount of capital remaining after friendly AGI.)
I donât have any well-formed opinions about what the post-AGI world will look like, so I donât think itâs obvious that logarithmic utility of capital is more appropriate than simply trying to maximize the probability of a good outcome. The way you describe it is how my model worked originally, but I changed it because I believe the new model gives a stronger result even if the model is not necessarily more accurate. I wrote in a paragraph buried in Appendix B:
Thanks, I only read through Appendix A.
It seems to me that your concern âthat the older model trivialized the question by assuming we could not spend our money on anything but AI safety researchâ could be addressed by dividing existing longtermist or EA capital up into one portion to be spent on AI safety and one portion to be spent on other causes. Each capital stock can then be spent at independent rates according to the value of availabkr giving opportunities in their respective cause areas.
Your model already makes the assumption:
And then:
It just seems like a weird constraint to say that with one stock of capital you only want to spend it on one cause (AI safety) before some event but will spend it on any cause after the event.
Iâm not sure that I can articulate a specific reason this doesnât make sense right now, but intuitively I think your older model is more reasonable.
The reason I made the model only have one thing to spend on pre-AGI is not because itâs realistic (which it isnât), but because it makes the model more tractable. I was primarily interested in answering a simple question: do AI timelines affect giving now vs. later?