”Thank you for your detailed comment! I appreciate you taking the time to write this out.
> On fundamental assumptions about reality changing due to ASI
This is a fair point – I agree it makes sense that most people wouldn’t worry about these discoveries in their day to day lives. However, the part I’d be most worried about would be the downstream effects from certain discoveries. For example, if it turns out our model of consciousness is wrong, I could see this causing social disruption/fragmentation. I don’t think this would be due to the discovery itself, but rather the way it was publicised, the factions that formed around it, whether it gets politicised, etc. If the discoveries really are way more shattering than anything we (as humans) have adapted to so far, I could see this being a big issue. Obviously this is very hard to predict/reason about though!
>How do you select for “truthfulness” when the nature of truth itself is being revised monthly?
Yeah, in retrospect this does seem overly dramatic. I think the point I was trying to make was more that the way people perceive what is fundamentally true would be changing at unprecedented speeds (which I assume would be a possibility during an IE).
>On automating forecasting and trust
Personally I’d place a lot of credibility on the automated AI forecasters (along with deferring to people I trust’s views on its accuracy). But I think there’s still a high enough chance that large parts of the population wouldn’t place this amount of trust on it. E.g. if conspiracy theorists claim (and gain traction) that the AI being biased towards some particular actor/group – or just another “tool from the elites to manipulate us”. I think this could get polarising especially if the change is rapid, similar to what we saw with trusting COVID advice. I’m not super confident about how likely this would be though, I’d need to look more into it.
>They [non-enhanced people] would functionally become children (or, even newborns if the intelligence explosion gets really crazy) in a world run by incomprehensible adults.
I think your “happy coincidence” point is very good. It is definitely right that Young Earth creationists can be terribly wrong while still functioning pretty well in society. But I think more extreme versions of cognitive enhancement would probably break this coincidence. Current epistemic inequality is about what people believe, whereas I’d expect future enhancement would be about how people think. If enhanced humans are thinking in fundamentally different ways (e.g. maybe through neural interfaces, expanded working memory, direct AI integration), they might design systems that require enhanced cognition just to interact with. I don’t know what else could be said here other than trying to advocate for keeping society “understandable” to everyone.
On deference—yes, this is very true that we already defer constantly. Though I think an issue could be that current deference assumes stable reference points, whereas in an intelligence explosion, how do we know which people/communities to trust when they might not exist long enough to build track records? This wouldn’t be an issue if people trusted the AI forecasters, but I think it would be for those who didn’t.
A lot of these questions are very hard for me to think through given how complicated and messy large scale human interactions are (and a lot could be wrong given this). I really hope AI can help with all this!
Lastly, thank you for recommending Dan William’s Substack! I looked at the recent posts and they seem very interesting/relevant, so I will definitely read more and see if it updates my views on this topic :)”
My response to his response:
”Thank you for your detailed comment! I appreciate you taking the time to write this out.
> On fundamental assumptions about reality changing due to ASI
This is a fair point – I agree it makes sense that most people wouldn’t worry about these discoveries in their day to day lives. However, the part I’d be most worried about would be the downstream effects from certain discoveries. For example, if it turns out our model of consciousness is wrong, I could see this causing social disruption/fragmentation. I don’t think this would be due to the discovery itself, but rather the way it was publicised, the factions that formed around it, whether it gets politicised, etc. If the discoveries really are way more shattering than anything we (as humans) have adapted to so far, I could see this being a big issue. Obviously this is very hard to predict/reason about though!
>How do you select for “truthfulness” when the nature of truth itself is being revised monthly?
Yeah, in retrospect this does seem overly dramatic. I think the point I was trying to make was more that the way people perceive what is fundamentally true would be changing at unprecedented speeds (which I assume would be a possibility during an IE).
>On automating forecasting and trust
Personally I’d place a lot of credibility on the automated AI forecasters (along with deferring to people I trust’s views on its accuracy). But I think there’s still a high enough chance that large parts of the population wouldn’t place this amount of trust on it. E.g. if conspiracy theorists claim (and gain traction) that the AI being biased towards some particular actor/group – or just another “tool from the elites to manipulate us”. I think this could get polarising especially if the change is rapid, similar to what we saw with trusting COVID advice. I’m not super confident about how likely this would be though, I’d need to look more into it.
>They [non-enhanced people] would functionally become children (or, even newborns if the intelligence explosion gets really crazy) in a world run by incomprehensible adults.
I think your “happy coincidence” point is very good. It is definitely right that Young Earth creationists can be terribly wrong while still functioning pretty well in society. But I think more extreme versions of cognitive enhancement would probably break this coincidence. Current epistemic inequality is about what people believe, whereas I’d expect future enhancement would be about how people think. If enhanced humans are thinking in fundamentally different ways (e.g. maybe through neural interfaces, expanded working memory, direct AI integration), they might design systems that require enhanced cognition just to interact with. I don’t know what else could be said here other than trying to advocate for keeping society “understandable” to everyone.
On deference—yes, this is very true that we already defer constantly. Though I think an issue could be that current deference assumes stable reference points, whereas in an intelligence explosion, how do we know which people/communities to trust when they might not exist long enough to build track records? This wouldn’t be an issue if people trusted the AI forecasters, but I think it would be for those who didn’t.
A lot of these questions are very hard for me to think through given how complicated and messy large scale human interactions are (and a lot could be wrong given this). I really hope AI can help with all this!
Lastly, thank you for recommending Dan William’s Substack! I looked at the recent posts and they seem very interesting/relevant, so I will definitely read more and see if it updates my views on this topic :)”