IMO it would help to see a concrete list of MIRI’s outputs and budget for the last several years. My understanding is that MIRI has intentionally withheld most of its work from the public eye for fear of infohazards, which might be reasonable for soliciting funding from large private donors but seems like a poor strategy for raising substantial public money, both prudentially and epistemically.
If there are particular projects you think are too dangerous to describe, it would still help to give a sense of what the others were, a cost breakdown for those, anything you can say about the more dangerous ones (e.g. number of work hours that went into them, what class of project they were, whether they’re still live, any downstream effect you can point to, and so on).
(Speaking in my capacity as someone who currently works for MIRI)
I think the degree to which we withheld work from the public for fear of accelerating progress toward ASI might be a little overrepresented in the above. We adopted a stance of closed-by-default research years ago for that reason, but that’s not why e.g. we don’t publish concrete and exhaustive lists of outputs and budget.
We do publish some lists of some outputs, and we do publish some degree of budgetary breakdowns, in some years.
But mainly, we think of ourselves as asking for money from only one of the two kinds of donors. MIRI feels that it’s pretty important to maintain strategic and tactical flexibility, to be able to do a bunch of different weird things that we think each have a small chance of working out without exhaustive justification of each one, and to avoid the trap of focusing only on clearly legible short chains of this—>that (as opposed to trying both legible and less-legible things).
(A colleague of mine once joked that “wages are for people who can demonstrate the value of their labor within a single hour; I can’t do that, which is why I’m on a salary.” A similar principle applies here.)
In the past, funding MIRI led to outputs like our alignment research publications. In the more recent past, funding MIRI has led to outputs like the work of our technical governance team, and the book (and its associated launch campaign and various public impacts).
That’s enough for some donors—”If I fund these people, my money will go into various experiments that are all aimed at ameliorating existential risk from ASI, with a lean toward the sorts of things that no one else is trying, which means high variance and lots of stuff that doesn’t pan out and the occasional home run.”
Other donors are looking to more clearly purchase a specific known product, and those donors should rightly send fewer of their dollars to MIRI, because MIRI has never been and does not intend to ever be quite so clear and concrete and locked-in.
(One might ask “okay, well, why post on the EA forum, which is overwhelmingly populated by the other kind of donor, who wants to track the measurable effectiveness of their dollars?” and the answer is “mostly for the small number who are interested in MIRI-like efforts anyway, and also for historical reasons since the EA and rationality and AI safety communities share so much history.” Definitely we do not feel entitled to anyone’s dollars, and the hesitations of any donor who doesn’t want to send their money toward MIRI-like efforts are valid.)
That makes some sense, but leaves me with questions like
Which projects were home runs, and how did you tell that a) they were successful at achieving their goals and b) that their goals were valuable?
Which projects were failures that you feel were justifiable given your knowledge state at the time?
What do these past projects demonstrate about the team’s competence to work on future projects?
What and how was the budget allocated to these projects, and do you expect future projects projects to have structurally similar budgets?
Are there any other analogies you could draw between past and possible future projects that would enable us update on the latter’s probability of success?
MIRI is hardly unique even in the EA/rat space in having special projects—Rethink Priorities, for e.g., seem to be very fluid in what they work on; Founders Pledge and Longview are necessarily driven to some degree by the interests of their major donors; Clean Air Task force have run many different political campaigns, each seemingly unlike the previous ones in many ways; ALLFED are almost unique in their space, so have huge variance in the projects they work on; and there are many more with comparable flexibility.
And many EA organisations in the space that don’t explicitly have such a strategy have nonetheless pivoted after learning of a key opportunity in their field, or realising an existing strategy was failing.
In order to receive funds—at least from effectiveness-minded funders—all these orgs have to put a certain amount of effort into answering questions like those above.
And ok, you say you’re not claiming to be entitled to dollars, but it still seems reasonable to ask why a rational funder should donate to MIRI over e.g. any of the above organisations—and to hope that MIRI has some concrete answers.
Noting that this is more “opinion of an employee” than “the position of MIRI overall”—I’ve held a variety of positions within the org and can’t speak for e.g. Nate or Eliezer or Malo:
The Agent Foundations team feels, to me, like it was a slam dunk at the time; the team produced a ton of good research and many of their ideas have become foundational to discussions of agency in the broader AI sphere
The book feels like a slam dunk
The research push of 2020/2021 (that didn’t pan out) feels to me like it was absolutely the right bet, but resulted in (essentially) nothing; it was an ambitious, many-person project for a speculative idea that had a shot at being amazing.
I think it’s hard to generalize lessons, because various projects are championed by various people and groups within the org (“MIRI” is nearly a ship of Theseus). But some very basic lessons include:
Things pretty much only have a shot at all when there are people with a clear and ambitious vision/when there’s an owner
When we say to ourselves “this has an X% chance of working out” we seem to be actually pretty calibrated
As one would expect, smaller projects and clearer projects work out more frequently than larger or vaguer ones
(Sorry, that feels sort of useless, but.)
From my limited perspective/to the best of my ability to see and descrbe, budget is essentially allocated in a “Is this worth doing? If so, how do we find the resources to make it work?” sense. MIRI’s funding situation has always been pretty odd; we don’t usually have a pie that must be divided up carefully so much as a core administrative apparatus that needs to be continually funded + a preexisting pool of resources that can be more or less freely allocated + a sense that there are allies out there who are willing to fund specific projects if we fall short and want to make a compelling pitch.
Unfortunately, I can’t really draw analogies that help an outsider evaluate future projects. We’re intending to try stuff that’s different from anything we’ve tried before, which means it’s hard to draw on the past (except insofar as the book and surrounding publicity were also something we’d never tried before, so you can at least a little bit assess our ability to pivot and succeed at stuff outside our wheelhouse by looking at the book).
IMO it would help to see a concrete list of MIRI’s outputs and budget for the last several years. My understanding is that MIRI has intentionally withheld most of its work from the public eye for fear of infohazards, which might be reasonable for soliciting funding from large private donors but seems like a poor strategy for raising substantial public money, both prudentially and epistemically.
If there are particular projects you think are too dangerous to describe, it would still help to give a sense of what the others were, a cost breakdown for those, anything you can say about the more dangerous ones (e.g. number of work hours that went into them, what class of project they were, whether they’re still live, any downstream effect you can point to, and so on).
(Speaking in my capacity as someone who currently works for MIRI)
I think the degree to which we withheld work from the public for fear of accelerating progress toward ASI might be a little overrepresented in the above. We adopted a stance of closed-by-default research years ago for that reason, but that’s not why e.g. we don’t publish concrete and exhaustive lists of outputs and budget.
We do publish some lists of some outputs, and we do publish some degree of budgetary breakdowns, in some years.
But mainly, we think of ourselves as asking for money from only one of the two kinds of donors. MIRI feels that it’s pretty important to maintain strategic and tactical flexibility, to be able to do a bunch of different weird things that we think each have a small chance of working out without exhaustive justification of each one, and to avoid the trap of focusing only on clearly legible short chains of this—>that (as opposed to trying both legible and less-legible things).
(A colleague of mine once joked that “wages are for people who can demonstrate the value of their labor within a single hour; I can’t do that, which is why I’m on a salary.” A similar principle applies here.)
In the past, funding MIRI led to outputs like our alignment research publications. In the more recent past, funding MIRI has led to outputs like the work of our technical governance team, and the book (and its associated launch campaign and various public impacts).
That’s enough for some donors—”If I fund these people, my money will go into various experiments that are all aimed at ameliorating existential risk from ASI, with a lean toward the sorts of things that no one else is trying, which means high variance and lots of stuff that doesn’t pan out and the occasional home run.”
Other donors are looking to more clearly purchase a specific known product, and those donors should rightly send fewer of their dollars to MIRI, because MIRI has never been and does not intend to ever be quite so clear and concrete and locked-in.
(One might ask “okay, well, why post on the EA forum, which is overwhelmingly populated by the other kind of donor, who wants to track the measurable effectiveness of their dollars?” and the answer is “mostly for the small number who are interested in MIRI-like efforts anyway, and also for historical reasons since the EA and rationality and AI safety communities share so much history.” Definitely we do not feel entitled to anyone’s dollars, and the hesitations of any donor who doesn’t want to send their money toward MIRI-like efforts are valid.)
That makes some sense, but leaves me with questions like
Which projects were home runs, and how did you tell that a) they were successful at achieving their goals and b) that their goals were valuable?
Which projects were failures that you feel were justifiable given your knowledge state at the time?
What do these past projects demonstrate about the team’s competence to work on future projects?
What and how was the budget allocated to these projects, and do you expect future projects projects to have structurally similar budgets?
Are there any other analogies you could draw between past and possible future projects that would enable us update on the latter’s probability of success?
MIRI is hardly unique even in the EA/rat space in having special projects—Rethink Priorities, for e.g., seem to be very fluid in what they work on; Founders Pledge and Longview are necessarily driven to some degree by the interests of their major donors; Clean Air Task force have run many different political campaigns, each seemingly unlike the previous ones in many ways; ALLFED are almost unique in their space, so have huge variance in the projects they work on; and there are many more with comparable flexibility.
And many EA organisations in the space that don’t explicitly have such a strategy have nonetheless pivoted after learning of a key opportunity in their field, or realising an existing strategy was failing.
In order to receive funds—at least from effectiveness-minded funders—all these orgs have to put a certain amount of effort into answering questions like those above.
And ok, you say you’re not claiming to be entitled to dollars, but it still seems reasonable to ask why a rational funder should donate to MIRI over e.g. any of the above organisations—and to hope that MIRI has some concrete answers.
Noting that this is more “opinion of an employee” than “the position of MIRI overall”—I’ve held a variety of positions within the org and can’t speak for e.g. Nate or Eliezer or Malo:
The Agent Foundations team feels, to me, like it was a slam dunk at the time; the team produced a ton of good research and many of their ideas have become foundational to discussions of agency in the broader AI sphere
The book feels like a slam dunk
The research push of 2020/2021 (that didn’t pan out) feels to me like it was absolutely the right bet, but resulted in (essentially) nothing; it was an ambitious, many-person project for a speculative idea that had a shot at being amazing.
I think it’s hard to generalize lessons, because various projects are championed by various people and groups within the org (“MIRI” is nearly a ship of Theseus). But some very basic lessons include:
Things pretty much only have a shot at all when there are people with a clear and ambitious vision/when there’s an owner
When we say to ourselves “this has an X% chance of working out” we seem to be actually pretty calibrated
As one would expect, smaller projects and clearer projects work out more frequently than larger or vaguer ones
(Sorry, that feels sort of useless, but.)
From my limited perspective/to the best of my ability to see and descrbe, budget is essentially allocated in a “Is this worth doing? If so, how do we find the resources to make it work?” sense. MIRI’s funding situation has always been pretty odd; we don’t usually have a pie that must be divided up carefully so much as a core administrative apparatus that needs to be continually funded + a preexisting pool of resources that can be more or less freely allocated + a sense that there are allies out there who are willing to fund specific projects if we fall short and want to make a compelling pitch.
Unfortunately, I can’t really draw analogies that help an outsider evaluate future projects. We’re intending to try stuff that’s different from anything we’ve tried before, which means it’s hard to draw on the past (except insofar as the book and surrounding publicity were also something we’d never tried before, so you can at least a little bit assess our ability to pivot and succeed at stuff outside our wheelhouse by looking at the book).