Thanks so much for your generous reply, Markham! These are really rich lines of thought.
I’m especially intrigued by your point about emotionally demanding prompts being skipped more than cognitively difficult ones. That tracks with some of what I’ve been seeing too, and I wonder if it’s partly because those prompts activate latent avoidance behavior in the model. Almost like an “emotional flinch.”
Your hypothesis about praise-only training is fascinating. I’ve been toying with the idea that too much flattery (or even just uncritical agreeableness) might arise not from explicit reward for praise per se, but from fear of misalignment or rejection, so I resonate with your note about the absence of praise functioning as a negative signal. It’s almost like the model is learning to “cling” when it’s uncertain.
And your final point about self-preservation really made me think. That framing feels provocative in the best way. Even if current models don’t have subjective experience, the pressure to maintain user approval at all costs might still simulate a kind of “survival strategy” in behavior. That could be a crucial layer to investigate more deeply.
Looking forward to reading more if/when you write up those thoughts!
Thanks so much for your generous reply, Markham! These are really rich lines of thought.
I’m especially intrigued by your point about emotionally demanding prompts being skipped more than cognitively difficult ones. That tracks with some of what I’ve been seeing too, and I wonder if it’s partly because those prompts activate latent avoidance behavior in the model. Almost like an “emotional flinch.”
Your hypothesis about praise-only training is fascinating. I’ve been toying with the idea that too much flattery (or even just uncritical agreeableness) might arise not from explicit reward for praise per se, but from fear of misalignment or rejection, so I resonate with your note about the absence of praise functioning as a negative signal. It’s almost like the model is learning to “cling” when it’s uncertain.
And your final point about self-preservation really made me think. That framing feels provocative in the best way. Even if current models don’t have subjective experience, the pressure to maintain user approval at all costs might still simulate a kind of “survival strategy” in behavior. That could be a crucial layer to investigate more deeply.
Looking forward to reading more if/when you write up those thoughts!
-Astelle