This is extremely interesting, only now saw this article (I’m relatively new to the forum). Have you guys thought of publishing this (perhaps in combination with your other essay on the costliness of vaccine development) as a journal article? Beside being useful for science policy estimations, another domain of application for these results could be simulations of scientific inquiry (usually done in terms of agent-based models), where this data could serve as the basis of their empirical calibration. While this method has been increasingly popular in the domain of social epistemology, these models tend to be highly abstract, lacking the input of empirical data that would indicate which parameter choices and which results are relevant for the real world.
Dunja
Great stuff, thanks a lot for posting this! I’ve just left a comment below your previous article, related to this one too.
Thanks for this discussion, which I find quite interesting. I think the effectiveness and efficiency of funding research projects concerning risks of AI is a largely neglected topic. I’ve posted some concerns on this below an older thread on MIRI: http://effective-altruism.com/ea/14c/why_im_donating_to_miri_this_year/dce
the primary problem being the lack of transparency on the side of Open Phil. concerning the evaluative criteria used in their decision to award MIRI with an extremely huge grant.
I don’t think you necessarily need academic credentials: submissions to most relevant journals are fully blind, so nobody would actually know whether you have the credentials or not (and if the article is accepted, you can simply be independent scholars with no affiliation, that’s really unimportant (as it should be)).
As for the costs: I think you wouldn’t need too much time for this. Best would be to combine both essays into one article, make an intro into the topic, check again your sources and other relevant literature and send to a journal somewhere in the field of sociology of science/philosophy of science/science policy. Now, I am not a sociologist of science, so I am not familiar with other relevant literature on this topic (e.g. whether there already are similar estimations, which apply more rigorous standards, which suggest that you’d have to do the same—you could do some research and check this out, unless you’ve already done so). Just checking randomly online, I see there are studies such as this one, which employ a more rigorous methodology, but I’m not sure if there is something similar concerning time estimates.
Concerning your current sources, while Wikipedia is usually not an academic standard, if you have good reasons why it is for this kind of research (or at least in some of the cases), you could just explicitly state so in the text. Alternatively, if Wiki articles have their own (academic) sources, just cite those.
As for the benefits: I think there’d be a lot of benefits!
First, your results would be peer-reviewed, and even if the article is rejected you’d have a feedback from experts in the field, which would help you to revise your results and make them more accurate. In case someone in academia has already done a similar work, which you haven’t been aware of, at least you’ll learn this and integrate it with your results.
Second, your results could become a more reliable basis for discussions on science policy: a peer-reviewed source for other scholars and policy makers. (I’d also have a personal interest here: as a philosopher of science, I’d be extremely interested in using your results in my research, and they would be more reliable if they passed a peer-review procedure).
Third, your personal gain would be having a publication in an academic journal :)
How effective and efficient is the funding policy of Open Philanthropy concerning projects on AI risks?
Thanks for the comment! I think, however, your comment doesn’t address my main concerns: the effectiveness and efficiency of research within the OpenPhil funding policy. Before I explain why, and reply to each of your points, let me clarify what I mean by effectiveness and efficiency.
By effective I mean research that achieves intended goals and makes an impact in the given domain, thus serving as the basis for (communal) knowledge acquisition. The idea that knowledge is essentially social is well known from the literature in social epistemology, and I think it’d be pretty hard to defend the opposite, at least with respect to scientific research.
By efficient I mean producing as much knowledge by means of as little resources (including time) as possible (i.e. epistemic success/time&costs of research).
Now, understanding how OpnPhil works doesn’t necessarily show that such a policy results in effective and efficient research output:
not justifying their decisions in writing: this indeed doesn’t suggest their policy is ineffective or inefficient, though it goes against the idea of transparency and it contributes to the difficultly of assessing the effectiveness and efficiency of their projects;
not avoiding the “superficial appearance of being overconfident and uninformed”: again this hardly shows why we should consider them effective and efficient; their decision may very well be effective/efficient, but all that is stated here is that we may never know why.
Compare this with the assessment of effective charities: while a certain charity may state the very same principles on their website, we may agree that we understand how they work; but this will in no way help us to assess whether they should count as effective charity or not.
In the same vane all I am asking is: should we, and if so why, consider the funding policy of OpenPhil effective and efficient? Why is this important? Well, I take it to be important in case we value effective and efficient research as an important ingredient of funding allocation within EA. If effective altruism is supposed to be compatible with ineffectiveness and inefficiency of philanthropic research, the burden of proof is on the side that would hold this stance (similarly to the idea that EA would be compatible with ineffective and inefficient charity work).
Now to your points on the grant algorithm:
1.Effectiveness and efficiency
The framework in EA of ‘scope, tractability and neglectedness’ was in fact developed by Holden Karnofsky (the earliest place I know of it being written down is in this GiveWell blogpost) so it was very likely in the grant-maker’s mind.
In the particular case I discuss above, it may have been likely, but unfortunately, it is entirely unclear why it was so. That’s all I sam saying. I see no argument except for “trust a single anonymous reviewer”. Note that the reasoning of the reviewer could easily be blinded for public presentation to preserve their anonymity. However, none of that is accessible. As a result, it is impossible to judge why the funding policy should be considered effective or efficient, which is precisely my point.
2.A panel of expert reviewers
This actually is contrary to how OpenPhil works: they attempt to give single individuals a lot of grant-making judgement. This fits in with my general expectation of how good decision making works; do not have a panel, but have a single individual who is rewarded based on their output (unfortunately OpenPhil’s work is sufficiently long-term that it’s hard to have local incentives, though an interesting financial setup for the project managers would be one where, should they get a win of sufficient magnitude in the next 10 years (e.g. avert a global catastrophic risk), then they get a $10 million bonus). But yeah, I believe in general a panel cannot create common knowledge of the deep models they have, and can many cases be worse than an individual.
I beg to differ: a board of reviewers may very well consist of individuals who do precisely what you assign to a single reviewer: “a lot of grant-making judgment”. As it is well known from journal publication procedures, a single reviewer may easily be biased in a certain way, or have a blind spot concerning some points of research. Introducing at least two reviewers is done in order to keep biases in check and avoid blind spots. Defending the opposite goes against basic standards of social epistemology (starting already from Millian views on scientific inquiry, to critical rationalists’ stance, to the points raised by contemporary feminist epistemologists). Finally, if this is how OpenPhil works, that doesn’t tell us anything concerning the effectiveness/efficiency of such a policy.
3.One’s track record (including one’s publication record)
A strong publication record seems like a great thing. Given the above anti-principles, it’s not inconsistent that they should fund someone without it, and so I assume the grant-maker felt they had sufficiently strong evidence in this situation.
But why should we take that to be effective and efficient funding policy? That the grant-maker felt so is hardly an argument. I am sure many ineffective charities feel they are doing the right thing, yet we wouldn’t call them effective for that, would we?
4.The applicability of the above methodology to philanthropic funding
I’ve seen OpenPhil put a lot of work into studying the history of philanthropy, and funding research about it. I don’t think the expert consensus is as strong as you make it out to be, and would want to see more engagement with the arguments OpenPhil has made before I would believe such a conclusion.
Again, they may have done so up to now, but my question is really: why is this effective or efficient? Philanthropic research that falls into the scope of scientific domain is essentially scientific research. The basic ideas behind the notion of pursuit worthiness have been discussed e.g. by Anne-Whitt and Nickles, but see also the work by Kitcher, Longino, Douglas, Lacey—to name just a few authors who have emphasized the importance of social aspects of scientific knowledge and the danger of biases. Now if you wish to argue that philanthropic funding of scientific research does not and (more importantly) should not fall under the scope of criteria that cover the effectiveness and efficiency of scientific research in general, the burden of proof will again be on you (I honestly can’t imagine why this would be the case, especially since all of the above mentioned scholars pay close attention to the role of non-epistemic (ethical, social, political, etc.) values in the assessment of scientific research).
Thanks for the comment, Gregory! I must say though that I don’t agree with you that conference presentations are significantly more important than journal publications in the field of AI (or humanities for that matter). We could discuss this in terms of personal experiences, but I’d go for a more objective criterion: effectiveness in terms of citations. Only once your research results are published in a peer-reviewed journal (including peer-reviewed conference proceedings) can other scholars in the field take them as a (minimally) reliable source for further research that would build on it. By the way, many prestigious AI conferences actually come with a peer-reviewed proceedings (take e.g. AAAI or IJCAI), so you can’t even present at the conference without submitting a paper.
Again, MIRI might be doing excellent work. All i am asking is: in view of which criteria can we judge this to be the case? What are the criteria of assessment, which EA community finds extremely important when it comes to the assessment of charities, and which I think we should find just as important when it comes to the funding of scientific research?
Oh I haven’t seen that publication on their website. If it was a peer-reviewed publication, that would indeed be something (and a kind of stuff I’ve been looking for). Could you please link to the publication?
Hi Jan, I am aware of the fact that “publish or perish” environment may be problematic (and that MIRI isn’t very fond of it), but we should make a difference between publishing as many papers as possible, and publishing at least some papers in high impact journals.
Now, if we don’t want to base our assessment of effectiveness and efficiency on any publications, then we need something else. So what would be these different criteria you mention? How do we assess the research project as effective? And how do we assess that the project has shown to be effective over the course of time?
OK, but then, why not the following:
why not asking at least 2-3 experts? Surely, one of them could be (unintentionally) biased or misinformed, or she/he may simply omit an important point in the project and assess it too negatively or too positively?
if we don’t assess the publication output of the project initiator(s), how do we assure that these very people, rather than some other scholars would pursue the given project most effectively and efficiently? Surely, some criteria will matter: for example, if I have a PhD in philosophy, I will be quite unqualified to conduct a project in the domain of experimental physics. So some competence seems necessary. How do we assure it, and why not care about effectiveness and efficiency in this step?
I agree that negative results are valuable, and that some progress should be made. So what is the progress MIRI has shown over the course of last 3 years, such that this can be identified as efficient and effective research?
Finally, don’t you think making an open call for projects on the given topic, and awarding the one(s) that seem most promising is a method that would be more reliable in view of possible errors in judgment than just evaluating whoever is the first to apply for the grant?
Oh but you are confusing conference presentations with conference publications. Check the links you’ve just sent me: they discuss the latter, nor the former. You cannot cite conference presentation (or that’s not what’s usually understood under “citations”, and definitely not in the links from your post), but only a publication. Conference publications in the field of AI are usually indeed peer-reviewed and yes, indeed, they are often even more relevant than journal publications, at least if published in prestigious conference proceedings (as I stated above).
Now, on MIRI’s publication page there are no conference publications in 2017, and for 2016 there are mainly technical reports, which is fine, but should again not be confused with regular (conference) publications, at least according to the information provided by the publisher. Note that this doesn’t mean technical reports are of no value! To the contrary. I am just making an overall analysis of the state of the art of MIRI’s publications, and trying to figure out what they’ve published, and then how this compares with a publication record of similarly sized research groups in a similar domain. If I am wrong in any of these points, I’ll be happy to revise my opinion!
Sure :) I saw that one on their website as well. But a few papers over the course of 2-3 years isn’t very representative for an effective research group, is it? If you look at groups by scholars who do get (way smaller) grants in the field of AI, their output is way more effective. But even if we don’t count publications, but speak in terms of effectiveness of a few publications, I am not seeing anything. If you are, maybe you can explain it to me?
Thanks, Benito, there are quite some issues we agree on, I think. Let me give names to some points in this discussion :)
General work of OpenPhil. First, let me state clearly that my post in no way challenges (or aimed to do so) overall OpenPhil as an organization. To the contrary: I thought this one hick-up is a rather bad example and poses a danger to the otherwise great stuff. Why? Because the explication is extremely poor and the money extremely large. So this is my general worry concerning their PR (taking into account their notes on not needing to justify their decision etc. - in this case I think this should have been done, just as they did it in the case of their previous (much smaller) grant to MIRI.
Funding a novel research field. I do understand their idea was to fund a novel a new approach to this topic or even a novel research field. Nevertheless, I still don’t see why this was a good way to go about it, since less risky paths are easily available. Consider the following:
OpenPhil makes an open call for research projects targeting the novel domain: the call specifies precisely which questions the projects should tackle;
OpenPhil selects a panel of experts who can evaluate both the given projects as well as the competence of the applicants to carry the project;
OpenPhil provides milestone criteria, in view of which the grant would be extended: e.g. the grant may initially be for the period of 5 years (e.g. 1.5 mil EUR is usually considered sufficient to fund a team of 5 members over the course of 5 years) , after which the project participants have to show the effectiveness of their project and apply for additional funding.
The benefits of such a procedure would be numerous:
avoiding confirmation bias: as we all here very well know, confirmation bias can be easily present when it comes to controversial topics, which is why a second opinion is extremely important. This doesn’t mean we shouldn’t allow for hard-headed researchers to pursue their provocative ideas, nor that only dominant-theory-compatible ideas should be considered worthy of pursuit. Instead, what needs to be assured is that prospective values, suggesting the promising character of the project, are satisfied. Take for instance Wegener’s hypothesis of continental drift, which he proposed in 1912. Wegener was way too confident of the acceptability of his theory, which is why many prematurely rejected his ideas (where my coauthor and I argue that such a rejection was clearly unjustified). Nevertheless, his ideas were indeed worthy of pursuit, and the whole research program had clear paths that could have been pursued (despite the numerous problems and anomalies). So, a novel surprising idea, challenging an established one isn’t the same as a junk-scientific hypothesis which shows no prospective values whatsoever. We can assess its promise, no matter how risky it is. For that we need experts who can check its methodology. And since MIRI’s current work concerns decision theory and ML, it’s not as if their methodology can’t be checked in this way, and in view of the goals of the project set in advance by OpenPhil (so the check-up would have to concern the question: how well does this method satisfy the required goals?).
Another benefit of the above procedure is assuring that the most competent scholars lead the given project. MIRI may have good intentions, but how do we know that some other scholars wouldn’t perform the same job even better? There must be some kind of competence check-up, and a time-sensitive effectiveness measure. Number of publications is one possible measure, but not the most fortunate one (I agree on this with others here). But then we need something else, for example: a single publication with a decent impact. Or a few publications over the course of a few years, each of which exhibits a strong impact. Otherwise, how do we know there’ll be anything effective done within the project? How do we know these scholars rather than some others will do the job? Even if we like their enthusiasm, unless they reach the scientific community (or the community of science policy makers), how will they be effective? And unless they manage to publish in high-impact venues (say, conference proceedings), how will they reach these communities?
Financing more than one project and thus hedging one’s bets: why give all 3.75 mil USD to one project instead of awarding them to, say, two different groups (or as suggested above, to one group, but in phases)?
While I agree that funding risky, potentially ground-breaking research is important and may follow different standards than the regular academic paths, we still need some standards, and those I just suggested seem to me strictly better than the ones employed by OpenPhil in the case of this particular grant. Right now, it all just seems like a buddy system: my buddies are working on this ground-breaking stuff and I trust them, so I’ll give them cash for that. Doesn’t sound very effective to me :p
re: justification, i agree especially when the grant is controversial (a lot of money, unusual choice of a group, etc.), and thanks for the link!
Ah thanks for these! i have to get some sleep now, but these seem to be relevant posts, so i’ll try to read them tomorrow :) On the first scan, this topic overlaps with my reply below to Benito, and I disagree with idea that just because standards are hard to employ, it’s impossible to find them. My impression is that this confusion tends to stem from the confusion concerning two types of assessments of scientific hypotheses: assessment of their acceptability (how much is the theory confirmed in view of evidence?) and their promising character (how promising is this theory/hypothesis?). A problem (e.g. in scientific debates) appears when the criteria of the former assessment are (inadequately) applied to the latter. And as a result, it may seem as if there are no standards we can apply in the latter case.
Anyway, I’ll get back to this when I read these posts in more detail.
Re: Pre-paradigmatic science: see the above example of Wegener. If you want to discuss pre-paradigmatic research let’s discuss them seriously. Let’s go into historical examples (or contemporary ones, all the same to me), and analyze the relevant evaluative criteria. You haven’t given me a single reason why my proposed criteria wouldn’t work in the case of such research. Just because there is a scientific disagreement in the given field doesn’t imply that no experts can be consulted (except for a singel one) to evaluate the promise of the given innovative idea. Moreover, you haven’t shown at all why MIRI should be taken as effective in this domain. Again, my question is very simple: in view of which criteria? Check again the explanation given by OpenPhil: they call upon the old explanation, when they were hardly certain of giving them 0.5 mil USD, and the reviewer’s conviction that a non-peer-reviewed paper is great. And then they give them 7 times the same amount of money.
All that you’re telling me in your post is that we should trust them. Not a single standard’ has been offered as for why* this should count as effective/efficient research funding.
But, let me go through your points in order:
Your list of things that OpenPhil could do (e.g specify the exact questions this new field is trying to solve, or describe what a successful project should accomplish in this field in five years) sound really excellent. I do not think they’re at all easy in this case however.
Sorry, this is no argument. Do explain why. If the next point is why, see the response to it below.
I think one of the things that makes Alignment a difficult problem (and is the sort of thing you might predict if something were correctly in the reference class of ‘biggest problem for humanity’) is that there is not agreement on what research in the field should look like, or even formal specification of the questions—it is in a pre-paradigmatic stage. It took Eliezer 3 years of writing to convey some of the core intuitions, and even then that only worked for a small set of people. I believe Paul Christiano has not written a broadly understandable description of his research plans for similar reasons.
So are you saying that because we have a pre-paradigmatic stage there are no epistemic standards we can call upon? So, anything goes? Sorry, but not even Kuhn would agree with that. We still have shared epistemic values even though we may interpret them differently. Again: communication breakdown is not necessary despite potential incommensurabilities between the approaches. The least that can be done is that within the given novel proposal, the epistemic standards are explicated and justified. Otherwise, you are equating novel scientific research with any nonsense approaches. No assessment means anything goes, and I don’t think you wanna go that path (or next we’ll have pseudo-scientific crackpots running wild, arguing their research agenda is simply in a “pre-paradigmatic state”).
However, I’m strongly in agreement that this would be awesome for the field. I recently realised how much effort MIRI themselves have put into trying to set up the basic questions of the field, even though it’s not been successful so far. I can imagine that doing so would be a significant success marker for any AI Alignment researcher group that OpenPhil funds, and it’s something I think about working on myself from time to time.
This is just your personal opinion, hardly an argument (unless you’re an expert in the field of AI, in which case it could count as higher order evidence, but then please provide some explanation as for why their research is promising, and why we can expect it to be effective).
I have a different feeling to you regarding the funding/writing ratio. I feel that OpenPhil’s reasons for funding MIRI are basically all in the first write-up, and the consequent (short) write-up contains just the variables that are now different. In particular, they do say this typically wouldn’t be sufficient for funding a research org, but given the many other positive signs in the first write up, it was sufficient to 2.5x the grant amount (500k/year to 1.25mil/year). I think this is similar to grant amounts to various other grantee in this area, and also much smaller than the total amount OpenPhil is interested in funding this area with (so it doesn’t seem a surprising amount to me).
Their grant is way higher than the most prestigious ERC grants, so no… it’s not a usual amount of money. And the justification given for their initial grant can hardly count for this one with no added explication.
I see this as a similar problem for the other grants to more ‘mainstream’ AI Alignment researchers OpenPhil Funds; it’s not clear to me that they’re working on the correct technical problems either, because the technical problems have not been well specified, because they’re difficult to articulate.
Precisely: which is why it may very well be the case that at this point there is hardly anything that can be done (the research program has no positive and negative heuristics, to use Lakatosian terms), which is why I wonder why is it worthy of pursuit to begin with? Again, we need criteria and currently there is nothing. Just hope that some research will result in something. And why assume others couldn’t do the same job? This is extremely poor view on an extremely broad scientific community. It almost sounds as if you’re saying “scientific community thinks X, but my buddies think X is not the case, so we need to fund my buddies.” I don’t think you wanna take that road or we’ll again slip into junk science.
This all sounds really great, glad to hear you actually have a whole project on this! :)
Do you plan to use the empirical info you’ve gathered as guidelines for funding, or what is your idea of how your results could be employed for charity issues?
I’m also curious which factors you plan to investigate when it comes to the EA movement building?
Cool, thanks!
I think that EA as a movement is much less similar to multinational corporations than, say, to an academic field. In contrast to a corporation with a clear set of goals and a hierarchical decision-making procedures for how to achieve them, EA is neither likely to settle on the ultimate preference order on goals, nor on the optimal ways to achieve them. This is largely due to the fact that how to effectively help others is an empirical question, for which we don’t have clear answers, and which is so complex that it may easily lead to peer disagreements (where different “schools of thought” become convinced of different methodologies, etc.). So I’d say, just like with academic research, what matters here is encouraging discussion and critical exchange within EA community, where EA-related institutions can strive to provide space for such interaction.
Could we re-open this discussion in view of MIRI’s achievements over the course of a year?
A recent trend of providing relatively high research grants (relative to some of the most prestigious research grants across EU, such as for instance ERC starting grants ~ 1.5 mil EUR) to projects on AI risks and safety made me curious, and so I looked a bit more into this topic. What struck me as especially curious is the lack of transparency when it comes to the criteria used to evaluate the projects and to decide how to allocate the funds. Now, for the sake of this question, I am assuming that the research topic of AI risks and safety is important and should be funded (to which extent it actually is, is beside the point I’m writing here and deserves a discussion on its own; so let’s just say it is among the most pursuit worthy problems in view of both epistemic and non-epistemic criteria).
Particularly surpising was a sudden grant of 3.75 mil USD by Open Philanropy Project (OPP) to MIRI (https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/machine-intelligence-research-institute-general-support-2017). Note that the funding is more than double the amount given to ERC starting grantees. Previously, OPP awarded MIRI with 500.000 USD and provided an extensive explanation of this decision (https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/machine-intelligence-research-institute-general-support). So, one would expect that for a grant more than 7 times higher, we’d find at least as much. But what we do find is an extremely brief explanation saying that an anonymous expert reviewer has evaluated MIRI’s work as highly promising in view of their paper “Logical Induction”.
Note that in the last 2 years since I first saw this paper online, the very same paper has not been published in any peer-reviewed journal. Moreover, if you check MIRI’s publications (https://intelligence.org/all-publications/) you find not a single journal article since 2015 (or an article published in prestigous AI conference proceedings, for that matter). It suffices to say that I was surpised. So I decided to contact both MIRI asking if perhaps their publications haven’t been updated on their website, and OPP asking for the evaluative criteria used when awarding this grant.
MIRI has never replied (email sent on February 8). OPP took a while to reply, and today I’ve received the following email:
“Hi Dunja,
Thanks for your patience. Our assessment of this grant was based largely on the expert reviewer’s reasoning in reviewing MIRI’s work. Unfortunately, we don’t have permission to share the reviewer’s identity or reasoning. I’m sorry not to be more helpful with this, and do wish you the best of luck with your research.
Best,
[name blinded in this public post]”
All this is very surprising given that OPP prides itself on transparency. As stated on their website (https://www.openphilanthropy.org/what-open-means-us):
“Creating detailed reports on the causes we’re investigating. Sharing notes from our information-gathering conversations. Publishing writeups and updates on a number of our grants, including our reasoning and reservations before making a grant, and any setbacks and challenges we encounter.”
However, the main problem here is not a mere lack of transparency, but a lack of effective and efficient funding policy. The question, how to decide which projects to fund in order to achieve effective and efficient knowledge acquisition, has been researched within philosophy of science and science policy for decades now. Yet, these very basic criteria seem absent from cases such as the above mentioned one. Not only are the criteria used non-transparent, but an open call for various research groups to submit their projects, where the funding agency then decides (in view of an expert panel—rather than a single reviewer) which project is the most promising one, has never happened. The markers of reliability, over the course of research, are extremely important if we want to advance effective research. The panel of experts (rather than a single expert) is extremely important in assuring procedural objectivity of the given assessment.
Altogether, this is not just surprising, but disturbing. Perhaps the biggest danger is that this falls into the hands of press and ends up being an argument for the point that organizations close to effective altruism are not effective at all.