I suffer strongly from the following, and I suspect many EAs do too (all numbers are to approximations to illustrate my point):
I think that AGI is coming within the next 50 years, 90% probability, with medium confidence
I think that there is a ~10% chance that development of AGI leads to catastrophic outcomes for humanity, with very low confidence
I think there is a ~50% chance that development of AGI leads to massive amounts of flourishing for humanity, with very low confidence
Increasing my confidence in points 2 & 3 seems very difficult and time consuming, as the questions at hand are exceptionally complex, and even identifying personal cruxes will be a challenge
I feel a moral obligation to prevent catastrophes and enable flourishing, where I have the influence to do so
I want to take actions that accurately reflect my values
Given the probabilities above, not taking strong, if not radical, action to try to influence the outcomes feels like a failure to embody my values, and a moral failure.
I’m still figuring out what to do about this. When you’re highly uncertain it’s obviously fine to hedge against being wrong, but again, given the numbers it’s hard to justify hedging all the way down to inaction.
I am trying to learn more about AI safety, but I’m not spending very much time on it currently. I’m trying to talk to others about it, but I’m not evangelising it, nor necessarily speaking with a great sense of urgency. At the moment, it’s low down my de factor priority list, even though I think there’s a significant chance it changes everything I know and care about. Is part of this a lack of visceral connection to the risks and rewards? What can I do to feel like my values are in line with my actions?
Your “90% confidence interval” of… what, exactly? This looks like a confidence interval over the value of your own subjective probability estimate? And “90% as the mean” of… a bunch of different guesses you’ve taken at your “true” subjective probability? I can’t imagine why anyone would do that but I can’t think what else this could coherently mean…?
If I can be blunt, I suspect you might be repeating probabilistic terms without really tracking their technical meaning, as though you’re just inserting nontechnical hedges. Maybe it’s worth taking the time to reread the map/territory stuff and then run through some calibration practice problems while thinking closely about what you’re doing. Or maybe just use nontechnical hedges more, they work perfectly well for expressing things like this.
Thanks for the feedback—it has indeed been a long time since I did high school statistics!
I specified that the numbers I gave were “approximations to prove my point” is because I know that I do not have a technical statistical model in my head, and I didn’t want to pretend that was the case. Given this is a non-technical, shortform post, I thought it was clear what I meant—apologies if that wasn’t so.
Skill up and work on technical AI safety! Two good resources: 1, 2. Even if you don’t yet feel the moral urgency, skilling up in ML can put you in a better position to do technical research in the future.
Thanks for the suggestion! I have actually spent quite a lot of time thinking about this—I had my 80k call last April and this was their advice. I’ve hesitated against doing this for a number of reasons:
I’m worried that even if I do upskill in ML, I won’t be a good enough software engineer to land a research engineering position, so part of me wants to improve as a SWE first
At the moment I’m very busy and a marginal hour of my time is very valuable, upskilling in ML is likely 200-500 hours, at the moment I would struggle to commit to even 5 hours per week
I don’t know whether I would enjoy ML, whereas I know I somewhat enjoy at least some parts of the SWE work I currently do
Learning ML potentially narrows my career options vs learning broader skills, so it’s hard to hedge
My impression is that there are a lot of people trying to do this right now, and it’s not clear to me that doing so would be my comparative advantage. Perhaps carving out a different niche would be more valuable in the future.
There are probably good rebuttals to at least some of these points, and I think that is adding to my confusion. My intuition is to keep doing what I’m currently doing, rather than go try and learn ML, but maybe my intuition here is bad.
Edit: writing this comment made me realise that I ought to write a proper doc with the pros/cons of learning ML and get feedback on it if necessary. Thanks for helping pull this useful thought out of my brain :)
I suffer strongly from the following, and I suspect many EAs do too (all numbers are to approximations to illustrate my point):
I think that AGI is coming within the next 50 years, 90% probability, with medium confidence
I think that there is a ~10% chance that development of AGI leads to catastrophic outcomes for humanity, with very low confidence
I think there is a ~50% chance that development of AGI leads to massive amounts of flourishing for humanity, with very low confidence
Increasing my confidence in points 2 & 3 seems very difficult and time consuming, as the questions at hand are exceptionally complex, and even identifying personal cruxes will be a challenge
I feel a moral obligation to prevent catastrophes and enable flourishing, where I have the influence to do so
I want to take actions that accurately reflect my values
Given the probabilities above, not taking strong, if not radical, action to try to influence the outcomes feels like a failure to embody my values, and a moral failure.
I’m still figuring out what to do about this. When you’re highly uncertain it’s obviously fine to hedge against being wrong, but again, given the numbers it’s hard to justify hedging all the way down to inaction.
I am trying to learn more about AI safety, but I’m not spending very much time on it currently. I’m trying to talk to others about it, but I’m not evangelising it, nor necessarily speaking with a great sense of urgency. At the moment, it’s low down my de factor priority list, even though I think there’s a significant chance it changes everything I know and care about. Is part of this a lack of visceral connection to the risks and rewards? What can I do to feel like my values are in line with my actions?
...What on earth does “90% probability, with medium confidence” mean? Do you think it’s 90% likely or not?
It means something like “my 90% confidence interval is 80% − 95%, with 90% as the mean”.
Your “90% confidence interval” of… what, exactly? This looks like a confidence interval over the value of your own subjective probability estimate? And “90% as the mean” of… a bunch of different guesses you’ve taken at your “true” subjective probability? I can’t imagine why anyone would do that but I can’t think what else this could coherently mean…?
If I can be blunt, I suspect you might be repeating probabilistic terms without really tracking their technical meaning, as though you’re just inserting nontechnical hedges. Maybe it’s worth taking the time to reread the map/territory stuff and then run through some calibration practice problems while thinking closely about what you’re doing. Or maybe just use nontechnical hedges more, they work perfectly well for expressing things like this.
Thanks for the feedback—it has indeed been a long time since I did high school statistics!
I specified that the numbers I gave were “approximations to prove my point” is because I know that I do not have a technical statistical model in my head, and I didn’t want to pretend that was the case. Given this is a non-technical, shortform post, I thought it was clear what I meant—apologies if that wasn’t so.
Skill up and work on technical AI safety! Two good resources: 1, 2. Even if you don’t yet feel the moral urgency, skilling up in ML can put you in a better position to do technical research in the future.
Thanks for the suggestion! I have actually spent quite a lot of time thinking about this—I had my 80k call last April and this was their advice. I’ve hesitated against doing this for a number of reasons:
I’m worried that even if I do upskill in ML, I won’t be a good enough software engineer to land a research engineering position, so part of me wants to improve as a SWE first
At the moment I’m very busy and a marginal hour of my time is very valuable, upskilling in ML is likely 200-500 hours, at the moment I would struggle to commit to even 5 hours per week
I don’t know whether I would enjoy ML, whereas I know I somewhat enjoy at least some parts of the SWE work I currently do
Learning ML potentially narrows my career options vs learning broader skills, so it’s hard to hedge
My impression is that there are a lot of people trying to do this right now, and it’s not clear to me that doing so would be my comparative advantage. Perhaps carving out a different niche would be more valuable in the future.
There are probably good rebuttals to at least some of these points, and I think that is adding to my confusion. My intuition is to keep doing what I’m currently doing, rather than go try and learn ML, but maybe my intuition here is bad.
Edit: writing this comment made me realise that I ought to write a proper doc with the pros/cons of learning ML and get feedback on it if necessary. Thanks for helping pull this useful thought out of my brain :)
Thanks for sharing this. I thought point 7 was well-put, and something I relate to (and could apply to several other cause areas/problems)
Hey,
My favorite tool for resolving internal conflict like this (which also resonated with many people I spoke to) is “internal double crux”.
TL;DR: Write down what you different parts want to say (and respond) to each other (as opposed to writing a list of something-like-pros-and-cons)
If you want, we can talk and do it together, and see if it works for you
This is a good idea, thanks for the suggestion! I’ve never really tried any of the CFAR stuff but this seems like a good place to start.
I’ll give it a go over the weekend and if I’m struggling then I’ll let you know and we can do it together :)
:)