“a la Thinking Physics or other textbook exercises.”
Very much think this is the wrong move, for the reason you mention that it doesn’t even have a clear intended path to cut the enemy. I would advise that for projects where there’s an imaginable highly-detailed endstate you’re trying to get to (as opposed to chess, where there are a million different checkmate patterns with few shared features that can guide your immediate next moves), you should start by mapping out the endstate. From there, you can backchain until you see a node you could plausibly forward-chain to—aka “opportunistic search”.
I think the greatest bottleneck to producing more competent alignment researchers is basically self-confidence. People are too afraid of embarrassment, so they don’t trust in their own judgment, so they won’t try to follow it, so they won’t grow better judgments by successively making embarrassing mistakes and correcting themselves. It’s socially frowned upon to innocently take your own impressions seriously when there exists smarter people than you, and it reflects an oppressive “thou shalt fall in line” group mentality that I find really unkind.
Like a GAN that wants to produce art, but doesn’t trust its own discriminator, so it atrophies and the only source of feedback that remains for the generator is the extremely slow loop of outside low-bandwidth opinion. Or like the pianist who’s forgotten how to listen, and looks to their parent after every press of a key to infer whether it was beautifwl or not.
I think that researchers that intend to produce something should forget about probability. You’re not optimising for accurate forecasts, you are optimising for building new models that can be tested and iteratively modified/abandoned until you have something that seems robust to all the evidence it catches. It’s the difference between searching for sources of Bayesian evidence related to specific models you already know about, vs searching for information that maximises the expected Kullbeck-Leibler Divergence between all your prior and posterior intuitions in order to come up with new models no one’s thought of before.
That means that you have to just start out trying to make your own models at some point, and you have to learn to trust your impressions so you’re actively motivated to build them. Which also means you’ll probably suffer in terms of your forecasting ability for a while until you get good enough. But if you’re always greedily following the estimated-truth-gradient at every step, you have no momentum to escape being stuck in local optima.
I realise you were asking for concrete advice, but I usually don’t think people are bottlenecked by lack of ideas for concrete options. I think the larger problem is upstream, in their generator, and resolving it lets them learn to generate and evaluate-but-not-defer-to ideas on their own.[1]
Of course, my whole ramble lacks lots of nuance, disclaimers, and doesn’t apply to all that it looks like I’m saying it applies to. But I’m not expecting you to defer to me, I’m revealing patterns that I hope people will steal and apply for themselves. Whether lack of nuance makes me be literally wrong is irrelevant. I’m not optimising for being judged “right” or “wrong”—this isn’t a forecasting contest—I’m just trying to be helpfwl by revealing tools that may be used.
Very much think this is the wrong move, for the reason you mention that it doesn’t even have a clear intended path to cut the enemy. I would advise that for projects where there’s an imaginable highly-detailed endstate you’re trying to get to (as opposed to chess, where there are a million different checkmate patterns with few shared features that can guide your immediate next moves), you should start by mapping out the endstate. From there, you can backchain until you see a node you could plausibly forward-chain to—aka “opportunistic search”.
I think the greatest bottleneck to producing more competent alignment researchers is basically self-confidence. People are too afraid of embarrassment, so they don’t trust in their own judgment, so they won’t try to follow it, so they won’t grow better judgments by successively making embarrassing mistakes and correcting themselves. It’s socially frowned upon to innocently take your own impressions seriously when there exists smarter people than you, and it reflects an oppressive “thou shalt fall in line” group mentality that I find really unkind.
Like a GAN that wants to produce art, but doesn’t trust its own discriminator, so it atrophies and the only source of feedback that remains for the generator is the extremely slow loop of outside low-bandwidth opinion. Or like the pianist who’s forgotten how to listen, and looks to their parent after every press of a key to infer whether it was beautifwl or not.
I think that researchers that intend to produce something should forget about probability. You’re not optimising for accurate forecasts, you are optimising for building new models that can be tested and iteratively modified/abandoned until you have something that seems robust to all the evidence it catches. It’s the difference between searching for sources of Bayesian evidence related to specific models you already know about, vs searching for information that maximises the expected Kullbeck-Leibler Divergence between all your prior and posterior intuitions in order to come up with new models no one’s thought of before.
That means that you have to just start out trying to make your own models at some point, and you have to learn to trust your impressions so you’re actively motivated to build them. Which also means you’ll probably suffer in terms of your forecasting ability for a while until you get good enough. But if you’re always greedily following the estimated-truth-gradient at every step, you have no momentum to escape being stuck in local optima.
I realise you were asking for concrete advice, but I usually don’t think people are bottlenecked by lack of ideas for concrete options. I think the larger problem is upstream, in their generator, and resolving it lets them learn to generate and evaluate-but-not-defer-to ideas on their own.[1]
Of course, my whole ramble lacks lots of nuance, disclaimers, and doesn’t apply to all that it looks like I’m saying it applies to. But I’m not expecting you to defer to me, I’m revealing patterns that I hope people will steal and apply for themselves. Whether lack of nuance makes me be literally wrong is irrelevant. I’m not optimising for being judged “right” or “wrong”—this isn’t a forecasting contest—I’m just trying to be helpfwl by revealing tools that may be used.