This certainly gets quite a bit of attention in internal conversations at the Future of Humanity Institute. Bostrom discussed it when first(?) writing about existential risk in 2001, under the name shrieks. Note I wouldn’t recommend reading that paper except for historical interest—his more modern exposition in Existential Risk Prevention as Global Priority is cleaner and excellent. I think your quality risk coincides with Bostrom’s notion of flawed realisation, although you might also mean to include subsequent ruination. Could you clarify?
Anyhow I’ll give my view briefly:
Much of the focus on risk from AI is about flawed realisations (from locking in the wrong values) more than never getting big.
Aside from concrete upcoming cases to lock-in values, it’s unclear whether we can affect the long-term trajectory. However we might be able to, so this gives only a modest reason to discount working to mitigate the risks of flawed realisations.
There are lots of plausible ways to indirectly help reduce future risk (both extinction risk and other kinds), by putting us in a better position to face future challenges. The further off challenges are, the more this looks like the right strategy. For extinction risks, some of them are close enough that the best portfolio looks like it includes quite a bit of directly addressing the risks. For risks of flawed realisation apart from AI, my guess is that the portfolio should be skewed heavily towards this capacity-building.
Many of the things we think of to do to improve long-term capacity to deal with challenges look less neglected right now than the direct risks. But not all (e.g. I think nurturing the growth of a thoughtful EA movement may be helpful here), and we should definitely be open to finding good opportunities in this space.
I would like to see more work investigating the questions in this area.
When I get multiple downvotes I like to use this to learn not to do things which people find unhelpful. Often I can go back and re-read my comment and work out what people didn’t like. Here I’m not so sure—something about tone? The fact that I gave my own framing of the issue more than building on the framing in the OP? Mixing two unrelated points (history of discussion and my views now) in one comment?
I’d appreciate pointers from anyone who downvoted, or who didn’t but felt a temptation to. I don’t want to discuss whether my post ‘deserved’ downvotes, I just want to understand what about it would drive them.
This certainly gets quite a bit of attention in internal conversations at the Future of Humanity Institute. Bostrom discussed it when first(?) writing about existential risk in 2001, under the name shrieks. Note I wouldn’t recommend reading that paper except for historical interest—his more modern exposition in Existential Risk Prevention as Global Priority is cleaner and excellent. I think your quality risk coincides with Bostrom’s notion of flawed realisation, although you might also mean to include subsequent ruination. Could you clarify?
Anyhow I’ll give my view briefly:
Much of the focus on risk from AI is about flawed realisations (from locking in the wrong values) more than never getting big.
Aside from concrete upcoming cases to lock-in values, it’s unclear whether we can affect the long-term trajectory. However we might be able to, so this gives only a modest reason to discount working to mitigate the risks of flawed realisations.
There are lots of plausible ways to indirectly help reduce future risk (both extinction risk and other kinds), by putting us in a better position to face future challenges. The further off challenges are, the more this looks like the right strategy. For extinction risks, some of them are close enough that the best portfolio looks like it includes quite a bit of directly addressing the risks. For risks of flawed realisation apart from AI, my guess is that the portfolio should be skewed heavily towards this capacity-building.
Many of the things we think of to do to improve long-term capacity to deal with challenges look less neglected right now than the direct risks. But not all (e.g. I think nurturing the growth of a thoughtful EA movement may be helpful here), and we should definitely be open to finding good opportunities in this space.
I would like to see more work investigating the questions in this area.
When I get multiple downvotes I like to use this to learn not to do things which people find unhelpful. Often I can go back and re-read my comment and work out what people didn’t like. Here I’m not so sure—something about tone? The fact that I gave my own framing of the issue more than building on the framing in the OP? Mixing two unrelated points (history of discussion and my views now) in one comment?
I’d appreciate pointers from anyone who downvoted, or who didn’t but felt a temptation to. I don’t want to discuss whether my post ‘deserved’ downvotes, I just want to understand what about it would drive them.
The downvoting throughout this thread looks funny. Absent comments, I’d view it as a weak signal.
“Quality risk” is meant to include both of those ideas, just any situation where we get “very large” (~”technologically mature”) but not “very good.”