It’s good to see some intelligent criticisms of the argument for doing AI safety research!
Just two short remarks to this post: I generally tend to think in the log scale about probabilities and it’s possible I haven’t used the a.bcd notation with any probability smaller than 10^-3 (0.1%) for years. So it is hard to see why I should be influenced by the described effect.
With language, your distinction is correlated with the distinction between two of Dennett’s levels of abstraction (the design stance and the intentional stance). Claiming something like the design stance is more accurate or better than the intentional stance for analyzing present day systems seems too bold: it’s really a different level of description. Would you say the design stance is also more accurate when thinking e.g. about animals?
Obviously, looking on any system with intentional stance comes with …mentalizing, assuming agency. People likely utilize the different levels of abstraction not really well, but I’m not convinced they systematically over-utilize one. It seems arguable they under-utilize the intentional stance when looking at “emergent agency” in systems like the stock market.
It’s good to see some intelligent criticisms of the argument for doing AI safety research!
Just two short remarks to this post: I generally tend to think in the log scale about probabilities and it’s possible I haven’t used the a.bcd notation with any probability smaller than 10^-3 (0.1%) for years. So it is hard to see why I should be influenced by the described effect.
With language, your distinction is correlated with the distinction between two of Dennett’s levels of abstraction (the design stance and the intentional stance). Claiming something like the design stance is more accurate or better than the intentional stance for analyzing present day systems seems too bold: it’s really a different level of description. Would you say the design stance is also more accurate when thinking e.g. about animals?
Obviously, looking on any system with intentional stance comes with …mentalizing, assuming agency. People likely utilize the different levels of abstraction not really well, but I’m not convinced they systematically over-utilize one. It seems arguable they under-utilize the intentional stance when looking at “emergent agency” in systems like the stock market.
Example of a practical benefit from taking the intentional stance: this (n=116) study of teaching programming by personalising the editor:
http://faculty.washington.edu/ajko/papers/Lee2011Gidget.pdf