I disagree. Or at least I think the reasons in this post are not very good reasons for Bostrom to step down (it is plausible to me he could pursue more impactful plans somewhere else, potentially by starting a new research institution with less institutional baggage and less interference by the University of Oxford).
Bostrom is as far as I can tell the primary reason why FHI is a successful and truth-oriented research organization. Making a trustworthy research institution is exceptionally difficult, and its success is not primarily measured in the operational quality of its organization, but in the degree to which it produces important, trustworthy and insightful research. Bostrom has succeeded at this, and the group of people (especially the early FHI cast including Anders Sandberg, Eric Drexler, Andrew Snyder Beattie, Owain Evans, and Stuart Armstrong) he has assembled under the core FHI research team have made great contributions to many really important questions that I care about, and I cannot think of any other individual who would have been able to do the same (Sean gives a similar perspective in his comment).
I think Bostrom overstretched himself when he let FHI grow to dozens and hundreds of people, and this seems to me like it was a mistake, and leaned too hard on skills that are hard to combine with intellectual integrity and vision, and skills that Bostrom otherwise does not seem exceptionally strong at (like navigating university politics and facilitating operational scaling). I do think that aspect of FHI has recently run into a bunch of problems, but at what I think of FHI’s core responsibility he has done very exceptional work and I see no suitable replacement for Bostrom at this point.
When I think of the value that FHI has produced for the world it has been in the unabashed exploration of ideas, with great willingness to explore wherever they go, even if this might involve engaging with ideas that are scary, sound crazy and speculative, or are societally taboo. To me, the core value add of FHI has always seemed to me to be one of providing a beacon, and one of the world’s best places to work at, for people who want to take ideas seriously and think rigorously about the future. The cultural components to create this kind of environment are very rare, and do not exist almost anywhere else in the world (and for example, IMO very clearly do not exist at GCRI or CSER or CSET or OpenAI or Anthropic or Longview or CLTR).
I think these are the things to pay attention when evaluating the historical performance of Bostrom’s FHI. Not on the basis of the organizations ability to write PR-statements, or scale a large research lab while navigating politically difficult relationships with a 1000 year old university. Most well-run large research labs with squeaky-clean public image do not answer interesting questions that are crucial to humanity’s future, in a way that a a reader can trust is actually driven by the desire to get the right answer, as opposed to pushing some kind of political or intellectual agenda. Indeed, I personally trust people more who don’t spend a huge chunk of their energy trying to maintain a completely clean public image.
For basically any other organization that is not FHI that pursues similar questions, when I read their takes on the macrohistory, or the future of humanity, I usually primarily see attempts to spread some ideology, to gain resources for some interest group, or attempts to build some sphere of influence by telling the right kind of macrohistory. For core FHI work, like “Eternity in Six Hours” (one of the papers that’s been most influential on my world view) I see what seems to me genuine interest in figuring out the truth and to answer the big questions, instead of secretly trying to trick me into supporting them, or get me to buy into their ideology, or support their favorite political cause or social movement, or to suspiciously shy away from a conclusion whenever that conclusion would be too hard to defend publicly to people who only want to spend 5 minutes on this question.
It is possible, and would be very sad, that FHI sadly cannot continue being this beacon, both because it scaled too quickly and its cultural magic is therefore no longer there, and because it is too deeply entwined with the University of Oxford, which will smother both its operational capacity and intellectual exploration.
In that case, I think the right choice is not for Bostrom to leave FHI, but for FHI to shut down. FHI is responsible for many of the best intellectual contributions for exploring the future of humanity, and I think before we do something that would substantially sabotage that legacy, it would be better to close down in a structured manner. I think it would be bad to let it fall in the hands of someone interested in making FHI into just another talent funnel, or another machine for producing prestige for Effective Altruism or AI Alignment or the people running FHI, while using up the credibility and intellectual integrity of Bostrom and many other core researchers who have created one of the highest integrity research institutions in the world.
I don’t know enough details about the FHI situation to have a strong judgement on whether Bostrom should stay and try to right the ship, or shut FHI down, but I think asking Bostrom to step down because of controversies belies the value that FHI most provides for the world.
I do think it is plausible that we should consider FHI a lesson on why getting too involved with risk-averse institutions will ultimately bite us, and we should be more hesitant to embed ourselves deeply into institutions like Oxford university. If indeed FHI is irrevocably tied to Oxford, and Oxford is unlikely to give FHI the operational and intellectual independence it needs, then I strongly encourage Bostrom to close down FHI, and to start something new with more independence, less need for both difficult PR work, and less need for navigating messy university politics.
It’s also possible to me that Sean’s proposal of finding a co-director to run FHI with might be the best choice, and might allow Bostrom to focus more on producing great research, and run a somewhat larger organization at the same time, though of course and finding such a co-director is also very difficult (though does seem easier than finding someone to lead the institution fully on their own).
Given the problems that FHI has run into here, do you think it’s likely to continue to do good work? My guess is that being able to hire is crucial to that work. Given that, we have to ask ourselves: how long should we wait for Nick to find a resolution to these problems before we conclude that he’s unable to solve them? My guess is that two years into a hiring freeze is long enough.
I agree with you that we should update somewhat in the direction of cooperating with universities being difficult and costly. But we should also entertain the hypothesis that it’s good to collaborate with universities, but it’s not good to ask Nick to be the one who does it.
I think you’re probably right, but that “proof” is too strong? FHI’s research agenda seems more “out there”, and more potentially controversial, than GPI’s, in a way that could plausibly make collaboration with the current department leadership impossible for FHI even with excellent leadership, or at least impossible without making self-defeating concessions. (To be clear, I don’t think this is likely to be the case.)
Yeah I was definitely using the word “proof” colloquially and not literally. My understanding from inside info though is that FHI’s issues with Oxford have very little to do with their choice of research agenda. I think this is also clear from outside info (FHI had a similar research agenda for a long time and had university support).
While this is a data point that shows that in principle it’s currently possible to currently work with the University, GPI has quite a different strategy compared to FHI that aligns significantly more with traditional academia, so it doesn’t necessarily prove that it would be currently possible for FHI.
However, I think a stronger existence proof for it being possible to work with the University is that FHI managed to do that in some way reasonably for at least 10+ years. (They were established in 2005) - for comparison, GPI is only 5 years old.
The more important metric to me will be if it is possible to do highly (preferably positively) impactful work while collaborating with universities, which I’ve seen positive evidence from individual professors/labs but not for larger groups.
It seems that your comment is mainly about successes by Bostrom in the (medium to more distant) past, while the post is about experience in the more recent past and expectations for the future. I would say that the expectations for the future are what is relevant to evaluate whether it’s a good thing or not for Bostrom to step down as Director (?)
Just mentioning some examples:
Bostrom has succeeded at this, and the group of people (especially the early FHI cast including Anders Sandberg, Eric Drexler, Andrew Snyder Beattie, Owain Evans, and Stuart Armstrong) he has assembled under the core FHI research team have made great contributions to many really important questions that I care about, and I cannot think of any other individual who would have been able to do the same
All of the people mentioned joined a long time ago and all but Sandberg have left FHI. Is there anyone of a comparable quality that joined in the last 5 years?
For core FHI work, like “Eternity in Six Hours” (one of the papers that’s been most influential on my world view) I see what seems to me genuine interest in figuring out the truth and to answer the big questions, instead of secretly trying to trick me into supporting them, or get me to buy into their ideology, or support their favorite political cause or social movement, or to suspiciously shy away from a conclusion whenever that conclusion would be too hard to defend publicly to people who only want to spend 5 minutes on this question.
The paper you mentioned was written 10 years ago. Are there any comparable more recent examples?
I think it would be bad to let it fall in the hands of someone interested in making FHI into just another talent funnel, or another machine for producing prestige for Effective Altruism or AI Alignment or the people running FHI, while using up the credibility and intellectual integrity of Bostrom and many other core researchers who have created one of the highest integrity research institutions in the world.
It’s not clear to me why this is a point in favour of Bostrom rather than against: In the last five years (until the hiring freeze started) it seems that is roughly what FHI (minus the macro strategy group) started to become under his leadership.
Overall, it looks to me that even if one agrees with all your statements about past successes and value of Bostrom as a leader of FHI, it doesn’t really make a case for Bostrom staying on as FHI Director now . (Though I guess it still makes the case for shutting down FHI rather than having it continue under new leadership.)
All of the people mentioned joined a long time ago and all but Sandberg have left GPI. Is there anyone of a comparable quality that joined in the last 5 years?
Just two quick nitpicks: I think you mean “FHI” not “GPI”. And I think Drexler is still at FHI in addition to Sandberg. But you’re right that ASB, Owain, and Stuart Armstrong have left FHI.
Totally! There are lots of controversies that I think would be quite bad. If Bostrom had stolen money, or falsified data, or shared a bunch of AI capability research ideas irresponsibly, or had committed sexual assault, or had badly mismanaged some conflict of interest, or had displayed some other deceptive attitude.
I am not sure what you mean by “controversy bad enough”. A lot of people seem angry at Bostrom, but that surely isn’t a good measure of “how bad” something that someone did was.
On “controversy bad enough”, I asked that because your statement was “I think asking Bostrom to step down because of controversies belies the value that FHI most provides for the world”, and I didn’t know whether you were referring to these recent controversies specifically (the bad effects of the recent controversies isn’t bad enough to override the good that Bostrom is expected to provide in the future over the counterfactual), or about controversies generally (asking Bostrom to step down because of controversies in general misses the value of what Bostrom brings; we should not do this).
A lot of people seem angry at Bostrom, but that surely isn’t a good measure of “how bad” something that someone did was.
Yeah, I agree that in many cases it isn’t a good measure, but I think saying it “surely isn’t a good measure” probably goes too far. There probably are nonzero contexts where it is a good measure, even if this isn’t one of them. For example, if someone’s work and role was highly dependent on or affected by popular opinion or buy-in, then something can be bad just because of a lot of people being angry at it.
Ah, yeah, sorry, that was indeed unclear of me. I think saying “because of recent controversies” would have been more accurate. I wasn’t trying to make a statement about all types of controversies (people totally get angry and cause controversies for good reasons as well as bad reasons).
I think that he generally means something that can be described as a controversy but not much else in addition to that. A “simple” controversy you might say. For example, I assume that controversies which are also crimes, and controversies which are also signs of poor mental health which could impact the job, would not be included here.
Sounds like a reasonable assumption, but I just thought it’d would be worth clarifying because Habryka is someone who cares a lot about the actual literal meaning of words, so I should err on the side of taking what he says literally.
I disagree. Or at least I think the reasons in this post are not very good reasons for Bostrom to step down (it is plausible to me he could pursue more impactful plans somewhere else, potentially by starting a new research institution with less institutional baggage and less interference by the University of Oxford).
Bostrom is as far as I can tell the primary reason why FHI is a successful and truth-oriented research organization. Making a trustworthy research institution is exceptionally difficult, and its success is not primarily measured in the operational quality of its organization, but in the degree to which it produces important, trustworthy and insightful research. Bostrom has succeeded at this, and the group of people (especially the early FHI cast including Anders Sandberg, Eric Drexler, Andrew Snyder Beattie, Owain Evans, and Stuart Armstrong) he has assembled under the core FHI research team have made great contributions to many really important questions that I care about, and I cannot think of any other individual who would have been able to do the same (Sean gives a similar perspective in his comment).
I think Bostrom overstretched himself when he let FHI grow to dozens and hundreds of people, and this seems to me like it was a mistake, and leaned too hard on skills that are hard to combine with intellectual integrity and vision, and skills that Bostrom otherwise does not seem exceptionally strong at (like navigating university politics and facilitating operational scaling). I do think that aspect of FHI has recently run into a bunch of problems, but at what I think of FHI’s core responsibility he has done very exceptional work and I see no suitable replacement for Bostrom at this point.
When I think of the value that FHI has produced for the world it has been in the unabashed exploration of ideas, with great willingness to explore wherever they go, even if this might involve engaging with ideas that are scary, sound crazy and speculative, or are societally taboo. To me, the core value add of FHI has always seemed to me to be one of providing a beacon, and one of the world’s best places to work at, for people who want to take ideas seriously and think rigorously about the future. The cultural components to create this kind of environment are very rare, and do not exist almost anywhere else in the world (and for example, IMO very clearly do not exist at GCRI or CSER or CSET or OpenAI or Anthropic or Longview or CLTR).
I think these are the things to pay attention when evaluating the historical performance of Bostrom’s FHI. Not on the basis of the organizations ability to write PR-statements, or scale a large research lab while navigating politically difficult relationships with a 1000 year old university. Most well-run large research labs with squeaky-clean public image do not answer interesting questions that are crucial to humanity’s future, in a way that a a reader can trust is actually driven by the desire to get the right answer, as opposed to pushing some kind of political or intellectual agenda. Indeed, I personally trust people more who don’t spend a huge chunk of their energy trying to maintain a completely clean public image.
For basically any other organization that is not FHI that pursues similar questions, when I read their takes on the macrohistory, or the future of humanity, I usually primarily see attempts to spread some ideology, to gain resources for some interest group, or attempts to build some sphere of influence by telling the right kind of macrohistory. For core FHI work, like “Eternity in Six Hours” (one of the papers that’s been most influential on my world view) I see what seems to me genuine interest in figuring out the truth and to answer the big questions, instead of secretly trying to trick me into supporting them, or get me to buy into their ideology, or support their favorite political cause or social movement, or to suspiciously shy away from a conclusion whenever that conclusion would be too hard to defend publicly to people who only want to spend 5 minutes on this question.
It is possible, and would be very sad, that FHI sadly cannot continue being this beacon, both because it scaled too quickly and its cultural magic is therefore no longer there, and because it is too deeply entwined with the University of Oxford, which will smother both its operational capacity and intellectual exploration.
In that case, I think the right choice is not for Bostrom to leave FHI, but for FHI to shut down. FHI is responsible for many of the best intellectual contributions for exploring the future of humanity, and I think before we do something that would substantially sabotage that legacy, it would be better to close down in a structured manner. I think it would be bad to let it fall in the hands of someone interested in making FHI into just another talent funnel, or another machine for producing prestige for Effective Altruism or AI Alignment or the people running FHI, while using up the credibility and intellectual integrity of Bostrom and many other core researchers who have created one of the highest integrity research institutions in the world.
I don’t know enough details about the FHI situation to have a strong judgement on whether Bostrom should stay and try to right the ship, or shut FHI down, but I think asking Bostrom to step down because of controversies belies the value that FHI most provides for the world.
I do think it is plausible that we should consider FHI a lesson on why getting too involved with risk-averse institutions will ultimately bite us, and we should be more hesitant to embed ourselves deeply into institutions like Oxford university. If indeed FHI is irrevocably tied to Oxford, and Oxford is unlikely to give FHI the operational and intellectual independence it needs, then I strongly encourage Bostrom to close down FHI, and to start something new with more independence, less need for both difficult PR work, and less need for navigating messy university politics.
It’s also possible to me that Sean’s proposal of finding a co-director to run FHI with might be the best choice, and might allow Bostrom to focus more on producing great research, and run a somewhat larger organization at the same time, though of course and finding such a co-director is also very difficult (though does seem easier than finding someone to lead the institution fully on their own).
Given the problems that FHI has run into here, do you think it’s likely to continue to do good work? My guess is that being able to hire is crucial to that work. Given that, we have to ask ourselves: how long should we wait for Nick to find a resolution to these problems before we conclude that he’s unable to solve them? My guess is that two years into a hiring freeze is long enough.
I agree with you that we should update somewhat in the direction of cooperating with universities being difficult and costly. But we should also entertain the hypothesis that it’s good to collaborate with universities, but it’s not good to ask Nick to be the one who does it.
GPI is a pretty clear existence proof that while collaborating with universities is difficult and costly, it can be done.
I think you’re probably right, but that “proof” is too strong? FHI’s research agenda seems more “out there”, and more potentially controversial, than GPI’s, in a way that could plausibly make collaboration with the current department leadership impossible for FHI even with excellent leadership, or at least impossible without making self-defeating concessions. (To be clear, I don’t think this is likely to be the case.)
Yeah I was definitely using the word “proof” colloquially and not literally. My understanding from inside info though is that FHI’s issues with Oxford have very little to do with their choice of research agenda. I think this is also clear from outside info (FHI had a similar research agenda for a long time and had university support).
While this is a data point that shows that in principle it’s currently possible to currently work with the University, GPI has quite a different strategy compared to FHI that aligns significantly more with traditional academia, so it doesn’t necessarily prove that it would be currently possible for FHI.
However, I think a stronger existence proof for it being possible to work with the University is that FHI managed to do that in some way reasonably for at least 10+ years. (They were established in 2005) - for comparison, GPI is only 5 years old.
Thanks—that’s a good point.
The more important metric to me will be if it is possible to do highly (preferably positively) impactful work while collaborating with universities, which I’ve seen positive evidence from individual professors/labs but not for larger groups.
It seems that your comment is mainly about successes by Bostrom in the (medium to more distant) past, while the post is about experience in the more recent past and expectations for the future. I would say that the expectations for the future are what is relevant to evaluate whether it’s a good thing or not for Bostrom to step down as Director (?)
Just mentioning some examples:
All of the people mentioned joined a long time ago and all but Sandberg have left FHI. Is there anyone of a comparable quality that joined in the last 5 years?
The paper you mentioned was written 10 years ago. Are there any comparable more recent examples?
It’s not clear to me why this is a point in favour of Bostrom rather than against: In the last five years (until the hiring freeze started) it seems that is roughly what FHI (minus the macro strategy group) started to become under his leadership.
Overall, it looks to me that even if one agrees with all your statements about past successes and value of Bostrom as a leader of FHI, it doesn’t really make a case for Bostrom staying on as FHI Director now . (Though I guess it still makes the case for shutting down FHI rather than having it continue under new leadership.)
EDIT: Corrected a GPI/FHI typo.
Great points.
~
Just two quick nitpicks: I think you mean “FHI” not “GPI”. And I think Drexler is still at FHI in addition to Sandberg. But you’re right that ASB, Owain, and Stuart Armstrong have left FHI.
Thanks for pointing out the FHI/GPI mistake, I’ve corrected that.
I also thought Drexler was still at FHI, but I checked and this doesn’t seem to be the case: He’s not mentioned on the team page and his website at FHI has been taken down.
He’s at GovAI.
As an affiliate, though, not as an employee. (And they seem to have lots of affiliates, so not clear what this actually means.)
He listed GovAI on this (very good!) post too: https://www.lesswrong.com/posts/5hApNw5f7uG8RXxGS/the-open-agency-model
Yeah dunno exactly what the nature of his relationship/link
Is there any controversy that you’d consider bad enough for you to support Bostrom stepping down from FHI? Because this sounds like no?
Totally! There are lots of controversies that I think would be quite bad. If Bostrom had stolen money, or falsified data, or shared a bunch of AI capability research ideas irresponsibly, or had committed sexual assault, or had badly mismanaged some conflict of interest, or had displayed some other deceptive attitude.
I am not sure what you mean by “controversy bad enough”. A lot of people seem angry at Bostrom, but that surely isn’t a good measure of “how bad” something that someone did was.
Thanks for clarifying!
On “controversy bad enough”, I asked that because your statement was “I think asking Bostrom to step down because of controversies belies the value that FHI most provides for the world”, and I didn’t know whether you were referring to these recent controversies specifically (the bad effects of the recent controversies isn’t bad enough to override the good that Bostrom is expected to provide in the future over the counterfactual), or about controversies generally (asking Bostrom to step down because of controversies in general misses the value of what Bostrom brings; we should not do this).
Yeah, I agree that in many cases it isn’t a good measure, but I think saying it “surely isn’t a good measure” probably goes too far. There probably are nonzero contexts where it is a good measure, even if this isn’t one of them. For example, if someone’s work and role was highly dependent on or affected by popular opinion or buy-in, then something can be bad just because of a lot of people being angry at it.
Ah, yeah, sorry, that was indeed unclear of me. I think saying “because of recent controversies” would have been more accurate. I wasn’t trying to make a statement about all types of controversies (people totally get angry and cause controversies for good reasons as well as bad reasons).
I think that he generally means something that can be described as a controversy but not much else in addition to that. A “simple” controversy you might say. For example, I assume that controversies which are also crimes, and controversies which are also signs of poor mental health which could impact the job, would not be included here.
Just my guess
Sounds like a reasonable assumption, but I just thought it’d would be worth clarifying because Habryka is someone who cares a lot about the actual literal meaning of words, so I should err on the side of taking what he says literally.