The Case for Impact Purchase | Part 1
In this post I will argue for impact purchase as a complement to (not a replacement for) other funding mechanisms, such as grants.
What is impact purchase?
Impact purchase is an alternative funding mechanism for altruistic work. I think the easiest way to explain what impact purchase is, is to compare it with grants.
For a more thorough explanation, and an actual example, read this excellent blogpost.
About this blogpost and the next one
This is part one of what I intend to be a two part blog post series. The second part will be significantly more work, so I’ll do that if this one gets enough interest.
Part 1 (this blogpost) will focus on the timing aspect of impact purchase, i.e. why it is sometimes better to evaluate and fund a project after it is finished.
Part 2 (next blogpost) will focus on the whys and hows of paying for value rather than cost. Most of my suggestions regarding implementation will be in this post.
Disclaimer
So far, my only investigation into this previous attempt to implement impact purchase, was to look around at their now inactive website. I don’t even know why the project ended. If someone could supply more information about this in the comments, that would be great. Otherwise, I will try to find out more for part 2.
Comment request
I am arguing a lot from my own experience and perspective. I hope that other people who want impact purchase to become a thing, will add their stories in the comments, or even just indicate that this is something you would apply for. I think this is relevant both to see how widespread the interest for this is, and what sort of projects it would support.
Reasons to evaluate a project after it is completed
I think the best part of impact purchase is that projects are evaluated after they are done.
You might think that this would be great for the funder (because it is much easier for them to know what they are paying for) but less great for potential recipients (because they now get to take all the risk). If that is how you think, then this should surprise you: I’m a potential recipient and I love the idea of having the funding application happen after the project. I don’t speak for everyone (and this is why we should still have grants) but I do know several other potential recipients who share my preferences.
More flexibility
For me, the time period between [I have a plan], and [I start work on this project], is typically somewhere in the range negative one month to a few weeks. By negative time, I mean that I start working before having a proper plan.
Having a plan is, in my experience, absolutely necessary to get a grant, but getting a grant takes months. There is typically a few months waiting time until the next suitable grant application opens. But even if I’m lucky with that, I’ll have to wait about two more months for grant evaluation. I’m not waiting around for that, and I would be very surprised if this problem was unique to me.
With impact purchase I can just follow the schedule that works best for me and my projects, and have it evaluated and funded some time in the future.
It is easier to evaluate a project after it happened
I have not done grant evaluation, but my impression is that it is super hard. Evaluating the impact of a project after it has happened is probably not easy either, but it has to be easier. This gain can be used to save time, do better evaluations, give more feedback to applicants, or some combination thereof.
If EA is vetting constrained, the gains from evaluating some projects after rather than before they happen could be hugely beneficial.
Why would you (or any EA funder) pay for something that has already happened?
However, you might think that paying after the project is over, is actually a bad deal for the funder, because you want counterfactual impact. You want your donation to create value that otherwise would not have happened. So why would you agree to pay for something that has already happened? Here are two ways of looking at it
Empower people based on their track record
The impact seller is someone who recently did something valuable with their time (because otherwise they would not have impact to sell), which is strong evidence that they are the kind of person who would do valuable things in the future. So let’s give them some money, no strings attached, and let them do whatever they want. Based on past evidence it will probably be good.
Under this motivation, the agreement is still that you pay people based on their past work (if you do something else it is not an impact purchase). But as a donor you are doing this with the expectation of more good work from them in the future.
Incentivise future altruistic work
The prospect of getting paid may motivate people to do good work. By paying some people for their past work, you build expectations and trust in the system, which will motivate more good work.
The difference in motivation matters for implementation
I think any donor needs to be clear about which of these reasons (or other) is motivating you, because your implementation would probably be somewhat different. If you want to empower great people, you may want to look more at who they are and what you expect them to do next. If you want to incentivise future work, it is important that your payouts are predictable.
My story (skip if you want)
Read this as a case story. Impact purchase should obviously not be implemented just for me. I would not write this post if I did not know others who would benefit too. But maybe using myself as an example will clarify some things? I don’t know.
I’m an AI Safety organiser. Last year I organised the Technical AI Safety Unconference (TAISU) and the Learning-by-doing AI Safety Research workshop (LAISR). Both of them were successful and I’d like to run more of these events.
However, this year I’m only planning to run TAISU and not LAISR and it has to do with money. I think that LAISR is potentially very valuable and I would love to keep developing this concept, but the target participants are mostly students, who are not able to pay much for the event. The only way I can get paid for LAISR is with external funding.
Last year I lived at EA hotel, and also used this place as an event venue, so money was not a problem. But I moved out now so I need to start caring about actually getting paid. I still have a lot of free time, so if impact purchase were a thing, I could experiment around with various projects, get paid for the ones that succeed, and learn from that.
I could try applying for a grant, but I’m not going to do that. I have had enough of getting turned down for grants. Maybe you think I should take the fact that I can’t get grants as a signal of something. But given that I mostly succeed, when I get to work on projects on my own terms, I take it as a signal that the grant application process is a bad match for my working style.
I imagine some reader of this post wanting to tell me that I too can learn the skill of applying for grants, and writing up a proper project plan in advance would be a valuable thing to do anyway. If that is you, please trust me when I say that this would not work for me. See the bit about flexibility earlier in this post. It would hurt both my wellbeing and productivity to try to conform to a grant format.
I need to know that a project will happen in order to have the motivation to engage in planning. One of the first things I do when organising an event is to write up the event invitation. I think about what information needs to be in the announcement (dates, venue, cost, etc) and work from there. Anything that does not need to go in the announcement, I’ll figure out later. This process works for me, because it works with the structure of my motivation, not against it. But starting with a grant application triggers no motivation for me.
The one thing I do like about grants is the invitation to ask for money. I have considered trying to get some sort of peer funding (Patreon or similar), but find it incredibly uncomfortable to ask for money unprompted (though I rather do that then grants at this point). I’m hoping that if impact purchase becomes a thing there will be some formal application system.
Some concerns and my responses
Grants provide good feedback, and help people not waste their time on bad projects.
I’ve heard this type of argument from many people when discussing grants. There seems to be this idea that applying for grants is a great source of feedback.
In my personal experience, the information value from applying for grants has been close to zero. Some reasons I personally have been turned down for grants:
I did not have a budget. Because of this my project did not get evaluated. (I could not provide a budget since I did not at that time know what my living costs would be, and therefore could not say what salary I needed.)
I applied for too little money. The lower bound was 10k and I only needed 5k. Because of this my project did not get evaluated.
My project was related to EA Hotel, and no grant making body ever makes up their mind about EA Hotel. Because of this my project did not get evaluated.
I’m not saying none of these problems were my fault. I am not saying I did not learn anything (I did learn how not to write grant applications). Also, I would not be surprised if other people have had very different experiences. I’m only saying that realistically speaking, in the real world, grants are not a reliable way to get project feedback. But even if it were, this would still not be a very strong argument, because it is possible to just ask for feedback on a project idea.
However, I admit that there is something very special and real about feedback that comes in the form “someone is willing to pay for this”. I want this type of feedback! This is actually one of the reasons I want impact purchase. But I’m fine with trying something out first and having it evaluated after the fact, because that would be less of a cost for me than conforming to whatever is needed to get a grant. This is not just speculation, to this point I have infinitely more success in completing projects than navigating grant applications, and I’d love to have some of my past projects evaluated from a funding perspective.
What about reputation risks and other unilateralist curse type of stuff?
Another thing I have come across is the idea that we need to make people apply for grants from central organisations, in order to police what project people are working on.
Given that people in the EA movment are already doing lots of stuff without central oversight, I’m not convinced that this is something we need to worry about. Or maybe the current situation is already a ticking bomb, and introducing impact purchase will make it worse? I don’t think so, but it is a valid concern.
Either way, I do think it is a good idea to avoid buying impacts from projects which have had a large downside risk. I think the evaluation for deciding to buy impact or not, should be at least partly based on: “Do we approve of this work?”, which include things like downside risk and negative externalities.
When not to go for impact purchase
These are the reasons I’m suggesting impact purchase as a complement, not as a replacement for other funding mechanisms. Removing grants would be bad.
When the money is needed up front
Not everyone can afford to run their project without payment in advance.
When the applicant can’t afford not getting paid
Even if a person has the cash at hand to do their project, it might still be a very bad idea for them to spend that money without any guarantee of reimbursement. Grants are safer and some people will need that safety.
Some initial thoughts about implementation
I will discuss implementation more in Part 2, when I have had more time to sort out my own thoughts regarding paying for value. However here’s one thing I think I can say already:
Consistency is important
Impact purchase will have a much stronger effect on people’s behaviours if potential recipients can either predict or have the possibility to learn what outcomes will get funded.
Imagine someone did a project which you like and you would like, which leads you to give them some money. Will this encourage this person (and maybe others) to do more of the same type of projects? Well maybe. Is a past impact purchase strong evidence that the same project will get rewarded in the future? If “yes” your money will much more strongly encourage similar things in the future.
“If I keep this up, I’ll keep getting funded” creates a form of financial stability that can potentially be very powerful. Impact purchases will never be a perfectly reliable source of income, but what sort of employment is these days? The more uncertain the payments from impact purchases are, the more people living off this will be incentivised to put their rewards into personal savings accounts, rather than use the money for their next cool thing.
- Should Grants Fund EA Projects Retrospectively? by (8 Sep 2021 2:58 UTC; 32 points)
- Making Impact Purchases Viable by (17 Apr 2020 23:01 UTC; 23 points)
- 's comment on Ben Snodin’s Quick takes by (5 May 2020 9:13 UTC; 19 points)
- 's comment on EA is risk-constrained by (24 Jun 2020 12:37 UTC; 10 points)
- 's comment on Making Impact Purchases Viable by (19 Apr 2020 14:00 UTC; 8 points)
- 's comment on MichaelA’s Quick takes by (15 Nov 2020 12:17 UTC; 6 points)
- 's comment on Impact Markets: The Annoying Details by (18 Aug 2022 15:34 UTC; 3 points)
- 's comment on Impact Markets: The Annoying Details by (20 Jul 2022 11:37 UTC; 3 points)
- 's comment on Dawn Drescher’s Quick takes by (6 Jun 2020 11:10 UTC; 3 points)
I’ve not thought about this idea much or read the linked articles on impact purchases, but a few quick thoughts:
I think prizes suffer from only allowing the most risk-tolerant to be incentivized by them since there is generally an aspect of competition in them and the winner often takes all or most of the prize funds.
Impact purchases seem like an improvement over this if you set it up like a grant that pays at the end rather than the beginning, so it’s tied to a single project/team and not a competition.
There might be hybrid model possible where a certain amount of funds are granted at the start of the project to cover costs and additional funding is awarded only as certain project milestones are hit, up to and including completion of the project. Some of this completion money is for awarding impact and not just funding the next phase of the project, as would be the case in a grant, with most of the impact award money held back until the end.
This lets me imagine funding something at like 20% the value of its impact up until it is created at which point I pay off the remaining 80% owed.
For me the most important consideration is flexibility, i.e. not having to wait for a grant comity to make up their mind before I can start. For this problem, the hybrid model is no better than a grant, unless it can speed up the application process by an order of magnitude.
Also, any major improvement in evaluation time also needs to be combined with running applications. Otherwise the applicant still have to wait a few months (on expectation) for the next grant round to come around. I guess there is a reason no major grant agency has running applications?
Let’s say you have running applications that evaluation time is proportional to the amount of money (I have no idea of that is true). Then funding something 20% up front would take a 1⁄5 of the time to evaluate, which is not bad. But I’m not sure how useful that would be for the applicant. I think most applications will fall into one of two cases:
1) The applicants is fine with taking the risk of not getting paid anything
2) The applicant needs to know that the majority of the budget will get covered
For example, if I run an event online or at CEEALAR (formally EA Hotel), I’m ok with taking the risk of not getting paid, I’ll just adjust my calculations for deciding if I want to run that event again. But if I run an event that has actual cost for me (other than my time), like travel and/or venue, then I need to know that those cost will be covered, 20% up front is probably not good enough.
But if an applicant is willing to put up with the hassle of applying for a grant (because they need the guaranteed money), then having some token amount depend on the outcome might be motivating. However, this also means that the grant maker need to evaluate the project twice, which takes even more time. But if I imagine myself as the recipient, I would very much welcome a post project evaluation from the grant maker, if this is something they want to do.
I think a improvement of this suggestion, is to cover any necessary cost in an initial grant (weather that be 0% or 90%). And offer an additional payment as a bonus if the project is successful. Where projects that request 0% in advance are “auto accepted” for the first half (which is £0). There might still be some point to pre-register projects with the grant makers, I think? Maybe they can say what metrics to track for the post evaluation? E.g. what questions they want in an event evaluation survey, and similar?
You can also borrow against the future prize or impact purchase, e.g. as Goldman Sachs allows you to do (in some limited cases). This moves the risk onto diversified private investors.
That’s pretty cool, but it seems mostly focused around supporting government agencies using this as an alternative funding mechanism to save money or defer costs or avoid paying for undelivered services, thus improving government spending efficiency. I wonder what it would take to develop that into something that would support a wider range of funding sources? Seems like something someone with some expertise and experience in finance could potentially pioneer as a neglected way to generally support EA.
I’m interested in seeing a second post on impact purchases and would personally consider selling impact in the future. I have a few general comments about this:
Impact purchases seem similar to value-based fees that are sometimes used in commercial consulting (instead of time- or project-based fees) and may be able to provide a complementary perspective. Although in business the ‘impact’ would usually be something easy to track (like additional revenue) and the return the consultant gets (like percentage of revenue up to a capped value) would be agreed on in advance. I wonder if a similar pre-arrangement for impact purchase could work for EA projects that have quantifiable impact outcomes, such as through a funder agreeing to pay some amount per intervention distributed, student educated, etc. Of course, the tracked outcome should reflect the funders true goals to prevent gaming the metric.
It seems like impact purchases would be particularly helpful for people coming into the EA community who don’t yet have good EA references/prestige/track-record but are confident they can complete an impactful project, or who want to work on unorthodox ideas that the community doesn’t have the expertise to evaluate. If they try something out and it works then they can get funds to continue and preliminary results for a grant, if not, it’s feedback to go more mainstream. For this dynamic to work people should probably be advised to plan relatively short projects (say a up too few months), otherwise they could spend a lot of time on something nobody values.
This could be a particularly interesting time to trial impact purchases used in conjunction with government UBI (if that ends up being fully brought in anywhere). UBI then removes the barrier of requiring a secure salary before taking on a project.
From my experience applying to a handful of early-career academic grants and a few EA grants, I agree that almost none provide any/useful feedback (beyond accepted or declined), either for the initial application or for progress or completion reports. However, worse than having no feedback is that I once heard from an European Research Council (ERC) grant reviewer that their review committees are required to provided feedback on rejected applications, but also instructed to make sure the feedback is vague and obfuscated so the applicant will have no grounds to ask for an appeal, which means the applicant gets feedback the reviewers know won’t be useful for improving their project… Why do they bother???
With regards to implementation. I think one point to consider is the demand from impacters relative to funds of purchasers. At least in academia, funding is constrained and grant success rates are often <20%, and so grantees know that it is unlikely they’ll get a grant to do their project (academic granters often say they turn away a lot of great projects they want to fund). If impact purchasers were similarly funding constrained relative to the number of good projects, I think the whole scheme would be less appealing as then even if I complete a great project, getting its impact bought would still involve a bit/lot of luck.
These posts about impact prizes and altruistic equity may also be of interest to consider.
Impact purchases + EA Hotel seems like a match made in heaven. EA Hotel even talks about taking a hits-based approach, so having a pool of funds to award both EA Hotel (or whatever it’s new name is) and the persons staying at the hotel who did the work that earned the funding sounds like a pretty interesting idea!
Update:
I have changed my mind quite a bit since writing this blogpost. The updates are coming from the discussions with you in the comments, so thanks for everyone discussing with me.
Everything in this comment are still work in progress. I’ll write something more formal and well though through later, when I have a more stable opinion. But my views have already change enough so that I wanted to add this update.
------------------------------------------------
What I actually want there to be is some sort of trust based funding. If I proven my self enough (e.g. by doing good work) then I get money, and no questions asked. The reason I want this is becasue of flexibility (see main post).
Giving away money = Giving away power
Impact perches has the neat structure that if I done X amount of good I get X amount worth of trust (i.e. money). This seems to be the exact right amount, because it is the most you can give away and still be protected from exploitation. If someone who are not aligned with the goal of the funder tires to use impact purchase as a money pump, they still have to do an amount of good equal to the payout they want.
But...
Khorton:
We actually know this form an other field. In most of academia, the law of the land is publish or perish. Someone living of impact purchases will face a similar situation, and it is not good, at least not in the long run.
Halffull
To the extent that this is true, impact purchase will not work.
In theory we could have impact investors, who funds a risky project and earn money by selling the impact of the few projects which impact reached the stars (literally and/or figuratively). But this requires an other layer which may or may not happen in reality (probably won’t happen). Also, from the perspective of the applicant, how is this any different from applying for a grant? So what have we gained?
If not impact purchase, then what?
I still would like to solve the problem of inflexibility that grants have. An actually I think the solutions already exist (to some extent).
1) Get a paid job, with high autonomy.
2) Start an organisation and fundraise. I did not think of this until now, but when orgs fundraise, they typically don’t present a plan for what they will do with the money. They mainly point towards what they have done so far, and ask for continued trust.
3) …? I’d be very interested in other suggestions. I would not be surprised if there are other obvious things I have missed.
There are also other solutions that don’t exist yet (or not very much) in EA, but could be implement by any institution or person with spare money:
a) “Trusted person”-job: A generic employment you offer to anyone who you like to keep up the good work, or something like that.
b) Support people on Ko-fi or Patreon, or similar, and generally encourage this behaviour from others too. (I know this is happening already, but not enough for people to make a living.)
Why not both? Grants available to people giving them a living wage, but then if their project turns out to be high value, award them a bonus—if it’s sufficiently high value, the bonus being sufficiently high to give them several months (or even years) runway.
That would also give you all the drawbacks of grants
See “Reasons to evaluate a project after it is completed” in the original post
If you want to give me a living wage without me first having to prove my self in some way, please give me money.
For most people, grants aren’t simply “available”. There has to be some evidence. This can be provided either by arguing your case (normal grant application) or by just doing the work. I think many people (including me) would prefer to just do the work, and let that speak for itself (for the reasons explained in the original post).
Some more additions:
I) I found out what happened to impactpurchase.org
Paul Christiano (from privet email, with the permission to quote):
II) Justin Shovelain told me (and gave me permission to share this information) that he would probably have focused more on Coronavirus stuff early on, if he though there where a way to get paid for this work.
This is another type of situation where grants are too slow.
I’m confused why there are several people apparently excited about this idea. It strikes me that impact finance so obviously outperforms impact purchases. So much so that I’m worried that I must be misunderstanding impact purchases.
Impact finance solves both of the stark problems of impact purchases:
(a) if the people doing the work only receive money after the event, how do they live?
(b) what incentive do purchasers of impact certificates have to purchase?
I perhaps need to clarify what I meant by Impact Finance. Given that this risks being confused with Impact Investment, I’ll rename it to Conditional Impact Finance.
Conditional Impact Finance occurs when a Project receives funding involving a Conditional Donor and an Impact Investor as follows:
(1) The Conditional Donor (who may be a pool of donors) agrees to fund the Project *on the condition* that they achieve a certain outcome
(2) The funds needed + a bonus go into escrow. If the outcome is achieved the funds are paid out, if not they are returned to the donor.
(3) In order that the Project receives the funds it needs to proceed with the work, it receives funding from an Impact Investor (or a pool of investors).
Further notes:
(1) Examples of conditions (fictional):
“The ABC innovative education Project will receive funding if a longitudinal RCT shows that employment rates among beneficiaries are at least 15% higher than control in 15 years’ time”
“The DEF malaria Project will receive funding if the malaria-linked mortality rate in Busia, Kenya is better than a control under a RCT conducted by LSHTM”
(2) A bonus is needed primarily to pay the interest on the debt (although perhaps a bonus for the staff might be a good idea too)
(3) If the project fails, the impact investor loses out. This means that all the incentives that currently apply to financial markets now apply to impact.
I consider this better for any reasonably sized project. If we are thinking of just one person doing some work on their own, I could imagine Impact Purchase maybe having some value.
I believe this is what is being done by alice.si
This is very similar to this notion of impact prizes. The main difference there seems to be that there is a specific allotted sum of money for a variety of possible possible projects, which share that allotted amount proportionally to their estimated impact.
I think that the downside of impact prizes compared to Conditional Impact Finance is mainly that it is much more volatile for investors—both because of dependencies between different projects and somewhat due to the continuum of possible values of estimated impact. Also, it is much harder on the donors. Well, there is also the problem that it may be clear that other competing projects are closing in on something much better (9x is enough to limit the prize to 10% of the original amount), and also competing interests between projects.
The major upside of impact prizes seems to be that the incentives of the project is better aligned with maximizing impact because they get a prize which scales sort of linearly with impact (unless they are enormously successful).
Impact purchases are one way of creating more impact finance. In particular, they can make it worthwhile for non-altruistic financiers to fund altruistic projects. This is particularly beneficial in cases where it’s hard for a single altruist to evaluate all the people who want funding.
With regard to (b), the incentives for impact purchasers are roughly similar to the incentives of someone who’s announced a prize. In both cases, the payer create incentives for others to do the work that will lead to payouts.
I’m confused by this response. I answered all of this in the blogpost. Did I fail to communicate? I am not saying that you have to agree, but if you read what I wrote and still don’t understand why *I* think some times paying after a project is a good idea, that is confusing to me, and I would like to understand better what part of the blogpost you found confusing.
I wrote up my thoughts in this post: Making Impact Purchases Viable. Briefly, I argue that:
Restrictions that limit the number of sellers are necessary to fix the imbalance between the size of the buyer’s market and the seller’s market.
Restrictions are also important for providing sufficient incentive for a reasonable proportion of the impact to be counterfactual.
Another issue I don’t have a solution to is the potential of impact purchases to lead to bitterness or demotivate people.
What’s the difference between an Impact Purchase and a prize? For example, what would be the difference between Impact Purchases for good quality blog posts and the EA Forum prize?
Prices and impact purchase are very similar. I would say that impact purchase would be an improvement though.
For an impact purchase the amount of money is decided based on how good impact of the project was. For a price, the price money usually set in advance, and there is often a winner takes it all dynamic.
Prices feels aversive to me becasue I win by being better than others, which means I’m disincentivised to help. This is fine whenever I don’t expect to win anyway, like with the forum prices. Very few blogpost gets awarded. Because the base rate is low I don’t feel like I’m loosing much by helping others. But if I’m trying to make a living out of selling impacts more regularly, I would not want this comparative aspect.
Although, it is not that simple, becasue there is inevitably competition in the market for selling impact. So in practice maybe it is not so different? I’m honestly a bit confused about if I think there is a real difference or not.
Maybe the difference is that a price often have a narrower focus, which pushes the competition to be between very similar work.
I would hate there to be an EA events price, because that would make me view fellow organisers as my competition, and those are exactly the people I should exchange experience and advise with. It would be much less bad to compete on “team organisers” against everyone else, about who can have the biggest overall impact.
I’m curious about how exactly this would work. My prior is that impact is clustered at the tails.
This means that there will frequently be small impact projects, and very occasionally be large impact projects—My guess is that if you want to be able to incentivize the frequent small
impact projects at all, you won’t be able to afford the large impact projects, because they are many magnitudes of impact larger. You could just purchase part of their impact, but in practice this means that there’s a cap on how much you can receive from impact purchase.
Maybe a cap is fine, and you know that all you’re ever get from an impact purchase is for instance $50,000, and the prestige comes with what % of impact they bought at that price.
Lest assume for now that impact is clustered as the tails.
(I don’t have a strong prior, but this at least don’t seem implausible to me)
Then how would you like to spend funding? Since there will limited amour of money, what is your motivation for giving the low impact projects anything at all?
Is it to support the people involved to do keep working, and eventually learn and/or get lucky enough to do something really important?
I’m not sure. The vibe I got from the original post was that it would be good to have small rewards for small impact projects?
I think the high impact projects are often very risky, and will most likely have low impact. Perhaps it makes sense to compensate people for taking the hit for society so that 1⁄1,000,000 of the people who start such projects can have high impact?
I’m unsure what size you have in mind when you say small.
I don’t think small monetary rewards (~£10) are very useful for anything (unless lots of people are giving small amounts, or if I do lot that add up to something that matters).
I also don’t think small impact projects should be encouraged. If we respect peoples time and effort, we should encourage them to drop small impact projects and move on to bigger and better things.
If you think that the projects with highest expected impact also typically have low success rate, then standard impact purchase is probably not a good idea. Under this hypothesis, what you want to do is to reward people for expected success rather than actual success.
I talk about success rather than impact, because for most project, you’ll never know the actual impact. By “success” I mean your best estimate of the projects impact, from what you can tell after the project is over. (I really meant success not impact from the start, probably should have clarified that some how?)
I’d say that for most events, success is fairly predictable, and more so with more experience as an organiser. If I keep doing events the randomness will even out. Would you say that events are low impact? Would you say events are worth funding?
Can you give an example of the type of high impact project you have in mind? How does your statement about risk change if we are talking about success instead?
I think most events will be comparatively low impact compared to the highest impact events. Let’s say you have 100,000 AI safety events. I think most of them will be comparatively low impact, but one in particular ends up creating the seed of a key idea in AI safety, another ends up introducing a key pair of researchers that go on to do great things together.
Now, if I want to pay those two highest impact events their relative money related to all the other events, I have a few options:
1. Pay all of the events based on their expected impact prior to the events, so the money evens out.
2. Pay a very small amount of money to the other events, so I can afford to pay the two events that had many orders of magnitude higher impact.
3. Only buy a small fraction of the impact of the very high impact events, so I have money left over to pay the small events and can reward them all on impact equally.
Whait what?
100 000 AI Safety Events?
Like 100 000 individual events?
There is a typo here right?
Nope, 1⁄50,000 seems like a realistic ratio for very high impact events to normal impact events.
It can’t take more that ~50 events for every AI Safety researcher to get to know each other.
And key ideas are not seeded at a single point in time, it is something that comes together from lots of reading and talking.
There is not *the one event* that made the different and all the others where practically useless. That’s not how research work. Sure there are randomness and some meetings are more important than others.
But if it took on average 50 000 events for one such a key introduction to happen, then we might as well give up on having events. Or find a better way to do it. Otherwise we are just wasting everyone’s time.
But all the other events were impactful, just not compared to those one or two events. The goal of having all the events is to hopefully be one of the 1⁄50,000 that has ridiculous outsized impact—It’s high expected value even if comparatively all the other events have low impact. And again, that’s comparatively. Compared to say, most other events, an event on AI safety is ridiculously high impact.
This is true, much of the networking impact of events is frontloaded.
I happen to think that relative utility is very clustered at the tails, whereas expected value is more spread out.. This comes from intuitions from the startup world.
However, it’s important to note that I also have developed a motivation system that allows me to not find this discouraging! Once I started thinking of opportunities for doing good in expected value terms, and concrete examples of my contributions in absolute rather than relative terms, neither of these facts was upsetting or discouraging.
Some relevant articles:
https://forum.effectivealtruism.org/posts/2cWEWqkECHnqzsjDH/doing-good-is-as-good-as-it-ever-was
https://www.independent.co.uk/news/business/analysis-and-features/nassim-taleb-the-black-swan-author-in-praise-of-the-risk-takers-8672186.html
https://foreverjobless.com/ev-millionaires-math/
https://www.facebook.com/yudkowsky/posts/10155299391129228
I’m ok with hit based impact. I just disagree about events.
I think you are correct about this for some work, but not for others. Things like operations and personal assistant are multipliers, which can consistently increase the productivity of those who are served.
Events that are focused on sharing information and networking fall in this category. People in a small field will get to know each other and each others work eventually, but if there are more events it will happen sooner, which I model as an incremental improvement.
But some other events feels much more hits based not that I think of it. Anything focused on getting people started (e.g. helping them choose the right career) or events focused on ideation.
But there are other types of event that are more hit based, and I notice that I’m less interested in doing them. This is interesting. Because these events also differ in other ways, there are alternative explanations. But seems worth looking at.
Thanks for providing the links, I should read them.
(Of course everything relating to X-risk is all or nothing in therms of impact, but we can’t measure and reward that until it does not matter anyway. Therefore in terms of AI Safety I would measure success in terms of research output, which can be shifted incrementally.)
If you’re trying to encourage or motivate people, my very rusty understanding of the psychology literature is that you should give people occasional rewards, rather than systematically rewarding people for what you want. Because the systematic rewards effectively undermine intrinsic motivation—you want people to be focusing on helping people, not meeting your criteria.
I sort of agree with this, but I want to add some things.
I agree that money is not the best motivator. If I was trying to solve [people are not motivated enough] I would probably suggest some community measure rather than a new funding structure.
Money is for buying people the time (i.e. not having do some day-job just to earn a living), or funding other things they need for whatever awesome project they are doing.
However money can defiantly influence motivation. 80k mentions list “Pay you feel is unfair.” as one of four “major negatives” which “tend to be linked to job dissatisfaction.”
https://80000hours.org/2014/09/update-dont-follow-your-passion/
Yes, that’s actually what I’m talking about.
Imagine I promise to give £10 to every small impact project, £100 to every medium impact, and £1000 to every large impact. You complete a project. It took you 400 hours of work and you’re very proud of it—you think it’s had a very significant impact. I pay you £10.
How do you think it would feel? How would it affect your future motivation? Are you sure it’s not better to a) get nothing and not have the system of judgement at all, or b) get a surprise thank you note from me with £10 inside that you weren’t expecting?
I think a lot of people spend a lot of time and effort on things that aren’t immediately useful, but I want them to keep their motivation because I believe that one day they may have a hit! If they keep getting £10 cheques for every 10 week cycle of work I’m afraid they’re going to be demotivated.
In this situation I would think you evaluated my project as “small impact” which is possibly useful information, depending on how reliable I think you evaluation is. If I trust your judgement, this would obviously be discouraging, since I though it was much more impressive. But in the end I rather be right then proud, so that I can make sure to do better things in the future.
How I react would also depend on if your £10 is all I get, or if I get £10 each from lots of people, becasue that could potentially add up, maybe?
What it mainly comes down to in the end is: Do I get paid enough to sustainably afford to do this work. Or do I need to focus my effort on getting a paid job instead.
If you are a funder, and you think what I’m doing is good, but not good enough to pay me a liveable wages, then I’d much prefer that you don’t try to encourage me, but instead just be upfront about this. Encouraging people to keep up an unsustainable work situation is exploitative and will backfire in the long run.
I definitely agree with that. But on the other hand, refusing to pay someone who’s good idea didn’t work out and ‘have impact’ for no fault of their own also seems exploitative!
I think people who are using this type of work as a living should get paid a salary with benefits and severance. A project to project lifestyle doesn’t seem conducive to focusing on impact.
Letting the person running the project take all the risk, might not be optimal, but I would also say it is not exploitative as long as they know this from the start.
I’m not yet sure if I think the amount of money should be 100% based on actual impact, or if we also want to reward people for project that had high expected impact but low actual impact. The main argument for focusing on actual impact is that it is less objective.
Um, I was going to argue with this. But actually I think you are right.
Something like: “We like what you have done so far, so we will hire you to keep doing good things based on your own best judgment.”
Agreed. In my brief experience with academic consulting one thing I’ve realised is that it is really quite reasonable for contracted consultants to charge a 50-100% premium (on top of their utilisation ratio—usually 50%, so another x2 markup) to account for their lack of benefits.
So if somebody is expecting to earn a ‘fair’ salary from impact purchases compared to employment (or from any other type of short-term contract work really) they should expect a funder to pay premium for this compared to employing them (or funding another organisation to do so) - this doesn’t seem like a good use of funds in the long-term if it is possible to employee that person.
Are there plans to publish Part II?
Sort of, and it might take some time. The short of it is that I’m less enthusiastic about impact purchase.
I want some sort of funding system that is flexible, and I think the best way to do this is to sponsor people and not project. If someone has though their past work shown competence and good judgement, I think they should be given a salary and freedom to do what they think is best.
I though the way to active this was impact purchase, but as someone pointed out in a comment, this makes for a very economical uncertain situation for the people living this way, which causes stress and short-sightedness which is not the best.
When I wrote this post, I assumed that I needed to have a plan to get a grant in the current system. But after talking to one of the fund manager of Long Term Future Fund, I found out that it is possible to get a grant by simply producing a track record and some vague plan to do more of the same. I’ve decided to try this out for my self. I’m waiting for an answer from Long Term Future Fund, and plan to write some update after I know how that goes.
If I get the grant this would prove that is is at least possible to get funding with out a clear plan. If I get rejected the concussions I take from that depends on what feedback I get with my rejection. Either way I decided to wait and see how the grant applications goes, before writing the follow up.
Regarding impactpurchase.org, there is some discussion in this comment thread.