I began my PhD with a focus on Bayesian deep learning with exactly the same reasoning as you. I also share your doubts about the relevance of BDL to long-term safety. I have two clusters of thoughts: some reasons why BDL might be worth pursuing regardless, and alternative approaches.
Considerations about BDL and important safety research:
Don’t overfit to recent trends. LLMs are very remarkable. Before them, DRL was very remarkable. I don’t know what will be remarkable next. My hunch is that we won’t get AGI by just doing more of what we are doing now. (People I respect disagree with that, and I am uncertain. Also, note I don’t say we could’t get AGI that way.)
Bayesian inference is powerful and general. The original motivation is still real. It is tempered by your (in my view, correct) observation that existing methods for approximate inference have big flaws. My view is that probability still describes the correct way to update given evidence and so it contains deep truths about reliable information processing. That means that understanding approximate Bayesian inference is still a useful guide for anyone trying to automatically process information correctly (and being aware of the necessary assumptions). And an awful lot of failure modes for AGI involve dangerous mistaken generalization. Also note that statements like “simple non-Bayesian techniques such as ensembles” are controversial, and there’s considerable debate about whether ensembles are working because they perform approximate integration. Andrew Gordon Wilson has written a lot about this, and I tentatively agree with much of it.
Your PhD is not your career. As Mark points out, a PhD is just the first step. You’ll learn how to do research. You really won’t start getting that good at it until a few years in, by which point you’ll write up the thesis and start working on something different. You’re not even supposed to just keep doing your thesis as you continue your research. The main thing is to have a great research role model, and I think Phillip is quite good (by reputation, I don’t know him personally).
BDL teaches valuable skills. Honestly, I just think statistics is super important for understanding modern deep learning, and it gives you a valuable lens to reason about why things are working. There are other specialisms that can develop valuable skills. But I’d be nervous about trading the opportunity to develop deep familiarity with the stats for practical experience on current SoTA systems (because stats will stay true and important, but SoTA won’t stay SoTA). (People I respect disagree with that, and I am uncertain.)
Big picture, I think intellectual diversity among AGI safety researchers is good, Bayesian inference is important and fundamental, and lots of people glom on to whatever the latest hot thing is (currently LLMs), leading to rapid saturation.
So what is interesting to work on? I’m currently thinking about two main things:
I don’t think that exact alignment is possible, in ways that are similar to how exact Bayesian inference is generally possible. So I’m working on trying to learn from the ways in which approximate inference is well/poorly defined to get insights for how alignment can be well/poorly defined and approximated. (Here I agree 100% with Mark that most of what is hard in AGI safety remains framing the problem correctly.)
I think a huge problem for AGI-esque systems is about to be hunting for dangerous failures. There’s a lot of BDL work on ‘actively’ finding informative data, but mostly for small-data in low-dimensions. I’m much more interested in huge data, high-dimensions, which creates whole new problems (e.g., you can’t just compute a score function for each possible datapoint). (Note that this is almost exactly the opposite to Mark’s point below! But I don’t exactly disagree with him, it’s just that lots of things are worth trying.)
There are other things that are important, and I agree that OOD detection is also important (and I’m working on a conceptual paper on this, rather than a detection method specifically). If you’d like to speak about any of this stuff I’m happy to talk. You can reach me at sebastian.farquhar@cs.ox.ac.uk
I can confirm that Philipp is a great supervisor! I also don’t plan on chasing the next best thing but want to understand ways to combine Bayesian ML with AI safety/alignment relevant things.
I began my PhD with a focus on Bayesian deep learning with exactly the same reasoning as you. I also share your doubts about the relevance of BDL to long-term safety. I have two clusters of thoughts: some reasons why BDL might be worth pursuing regardless, and alternative approaches.
Considerations about BDL and important safety research:
Don’t overfit to recent trends. LLMs are very remarkable. Before them, DRL was very remarkable. I don’t know what will be remarkable next. My hunch is that we won’t get AGI by just doing more of what we are doing now. (People I respect disagree with that, and I am uncertain. Also, note I don’t say we could’t get AGI that way.)
Bayesian inference is powerful and general. The original motivation is still real. It is tempered by your (in my view, correct) observation that existing methods for approximate inference have big flaws. My view is that probability still describes the correct way to update given evidence and so it contains deep truths about reliable information processing. That means that understanding approximate Bayesian inference is still a useful guide for anyone trying to automatically process information correctly (and being aware of the necessary assumptions). And an awful lot of failure modes for AGI involve dangerous mistaken generalization. Also note that statements like “simple non-Bayesian techniques such as ensembles” are controversial, and there’s considerable debate about whether ensembles are working because they perform approximate integration. Andrew Gordon Wilson has written a lot about this, and I tentatively agree with much of it.
Your PhD is not your career. As Mark points out, a PhD is just the first step. You’ll learn how to do research. You really won’t start getting that good at it until a few years in, by which point you’ll write up the thesis and start working on something different. You’re not even supposed to just keep doing your thesis as you continue your research. The main thing is to have a great research role model, and I think Phillip is quite good (by reputation, I don’t know him personally).
BDL teaches valuable skills. Honestly, I just think statistics is super important for understanding modern deep learning, and it gives you a valuable lens to reason about why things are working. There are other specialisms that can develop valuable skills. But I’d be nervous about trading the opportunity to develop deep familiarity with the stats for practical experience on current SoTA systems (because stats will stay true and important, but SoTA won’t stay SoTA). (People I respect disagree with that, and I am uncertain.)
Big picture, I think intellectual diversity among AGI safety researchers is good, Bayesian inference is important and fundamental, and lots of people glom on to whatever the latest hot thing is (currently LLMs), leading to rapid saturation.
So what is interesting to work on? I’m currently thinking about two main things:
I don’t think that exact alignment is possible, in ways that are similar to how exact Bayesian inference is generally possible. So I’m working on trying to learn from the ways in which approximate inference is well/poorly defined to get insights for how alignment can be well/poorly defined and approximated. (Here I agree 100% with Mark that most of what is hard in AGI safety remains framing the problem correctly.)
I think a huge problem for AGI-esque systems is about to be hunting for dangerous failures. There’s a lot of BDL work on ‘actively’ finding informative data, but mostly for small-data in low-dimensions. I’m much more interested in huge data, high-dimensions, which creates whole new problems (e.g., you can’t just compute a score function for each possible datapoint). (Note that this is almost exactly the opposite to Mark’s point below! But I don’t exactly disagree with him, it’s just that lots of things are worth trying.)
There are other things that are important, and I agree that OOD detection is also important (and I’m working on a conceptual paper on this, rather than a detection method specifically). If you’d like to speak about any of this stuff I’m happy to talk. You can reach me at sebastian.farquhar@cs.ox.ac.uk
Wow. That was really insightful.
I can confirm that Philipp is a great supervisor! I also don’t plan on chasing the next best thing but want to understand ways to combine Bayesian ML with AI safety/alignment relevant things.
I’ll write you a mail soon!