Glancing over the projects, I am confused about how some of them fit into the theoretical case for impact certificates as I understand it. Of course, my understanding could be incomplete or even wrong! To pick two examples, chosen because they are more legible as examples rather than out of any view of their merits:
Someone would like funding for an MPhil or MPhil/PhD. I can see how we would have a significantly better view on how impactful this grant (if made) was ~5-10 years down the future. I don’t understand how the grantmakers are likely to have a better view of the grant’s impact in spring 2025 than they have in spring 2024.
Someone would like funding to distribute books. It’s unlikely we could ever reasonably and cost-effectively know how impactful this grant was. Even with ~twenty years’ hindsight, figuring out the positive impact of the recipients and how much (if any) of that impact to assign to receiving a book sounds like a questionable use of resources.
I appreciate the desire for shorter feedback loops to prove / improve the impact-certificate model. But for these kinds of projects, it seems like we aren’t really measuring impact—we are guessing how the grantmakers will evaluate these projects a year later on much the same information that is available now.
There are, of course, reasons one might want to do that. If I recall correctly, SFF does something vaguely similar to address grant opportunities that are too time-sensitive to wait for the next grantmaking cycle. But here, I’m having a hard time understanding the value proposition of devoting resources to predict how grantmakers will view the projected impact of these projects in ~12 months’ time on ~the same information.
hey jason, thanks for leaving your thoughts! i wrote a lot below — if you’d prefer to talk about this over a call, i’d be down! (i’d also be happy to take notes on the call and post them in this thread, for anyone who comes across it later. i’ll update this comment if/when that happens.)
***
it looks like you had some uncertainty about how we’re concretely planning to go about reviewing projects that will still be uncertain ~12 months from now. concretely, the way that the philanthropies will review projects a year from now is listed here:
Final oracular funders will operate on a model where they treat retrospective awards the same as prospective awards, multiplied by a probability of success. For example, suppose LTFF would give a $20,000 grant to a proposal for an AI safety conference, which they think has a 50% chance of going well. Instead, an investor buys the impact certificate for that proposal, waits until it goes well, and then sells it back to LTFF. They will pay $40,000 for the certificate, since it’s twice as valuable as it was back when it was just a proposal with a 50% success chance.
however, you also have a more substantive point:
I’m having a hard time understanding the value proposition of devoting resources to predict how grantmakers will view the projected impact of these projects in ~12 months’ time on ~the same information.
if that ends up being the case — that, in ~12 months, grantmakers are acting on ~the same information about the projects — i think that we will have ex-post provided little value. however, i think the key here is the extent to which new information is revealed in the next ~12 months. here’s how i’m thinking about it:
there will be less uncertainty about the impactfulness of most projects in ~12 months compared to now. that delta in uncertainty is really important, and the more of it there is, the better. (i’ll touch on the examples you listed in a moment.)
however, the delta in uncertainty changes from one project to another:
for some projects, you get a lot of new information from the point in time they first received investment to the point in time the grantmakers review them.
...but for others, you get little new information; in your words, “grantmakers will [be] view[ing] the projected impact of these projects in ~12 months’ time on ~the same information.”
impact markets work best when they list projects in the first category — ones that have a big delta in uncertainty from now to ~12 months from now; ones for which you get a lot of new information from the point in time they first received investment to the point in time the grantmakers review them. ideally, we want the projects on an impact market to maximize that delta in uncertainty.
my understanding of your claim is: you agree with the above three bullet points, and are concerned that a number of projects that we’re listing will have a really small delta in uncertainty from now to ~12 months from now.
i’m not too worried about this concern, for two overlapping reasons:
a lot happens in a year! for most of the projects, i’d be shocked if they were in a roughly similar position as they are immediately after getting funding — even if the position they’re in is “we straightforwardly did what we said we would do,” you’ve already reduced a lot of uncertainty. i agree that we aren’t truly measuring impact in some platonic sense of impact measurement, but if i were a grantmaker, i would much much much rather be in the position of evaluating a project in the “~12 months from now” category than the “now” category. for the two examples that you happened to list, i think it’s actually quite possible that there will be significant new information that comes out in the ~12 months after a project gets investment:[1]
Someone would like funding for an MPhil or MPhil/PhD.
~12 months from now, they didn’t get into any grad schools/they decided not to go to grad school/etc.
~12 months from now, they got into a great grad school, and their first few semesters they got great grades/published great research/etc.
Someone would like funding to distribute books.
~12 months from now, they took the money, then disappeared/totally failed/the project didn’t materialize/etc.
~12 months from now, they successfully distributed the books, in exactly the way they described.
if we’re running the impact market correctly, the above point (1) should be baked into investment decisions — investors want returns, which incentivizes them to pick the projects that will have the highest delta in uncertainty from now to ~12 months’ time. after all, if a philanthropy reviews a project in a years’ time and sees exactly the same information as an investor does today… then that investor won’t make any returns.
i think you picked up on one potential failure mode, and i’d be interested to see if we end up failing in this way. right now, i’m not too concerned that will happen, though.
also, thanks for your detailed comment — i really appreciate the feedback. if you think i’ve missed something or that i went wrong somewhere, i’d really love to hear your thoughts. again, feel free to leave a comment in response, or if you’d prefer to talk about this over a call, i’d be down! :)
Glancing over the projects, I am confused about how some of them fit into the theoretical case for impact certificates as I understand it. Of course, my understanding could be incomplete or even wrong! To pick two examples, chosen because they are more legible as examples rather than out of any view of their merits:
Someone would like funding for an MPhil or MPhil/PhD. I can see how we would have a significantly better view on how impactful this grant (if made) was ~5-10 years down the future. I don’t understand how the grantmakers are likely to have a better view of the grant’s impact in spring 2025 than they have in spring 2024.
Someone would like funding to distribute books. It’s unlikely we could ever reasonably and cost-effectively know how impactful this grant was. Even with ~twenty years’ hindsight, figuring out the positive impact of the recipients and how much (if any) of that impact to assign to receiving a book sounds like a questionable use of resources.
I appreciate the desire for shorter feedback loops to prove / improve the impact-certificate model. But for these kinds of projects, it seems like we aren’t really measuring impact—we are guessing how the grantmakers will evaluate these projects a year later on much the same information that is available now.
There are, of course, reasons one might want to do that. If I recall correctly, SFF does something vaguely similar to address grant opportunities that are too time-sensitive to wait for the next grantmaking cycle. But here, I’m having a hard time understanding the value proposition of devoting resources to predict how grantmakers will view the projected impact of these projects in ~12 months’ time on ~the same information.
hey jason, thanks for leaving your thoughts! i wrote a lot below — if you’d prefer to talk about this over a call, i’d be down! (i’d also be happy to take notes on the call and post them in this thread, for anyone who comes across it later. i’ll update this comment if/when that happens.)
***
it looks like you had some uncertainty about how we’re concretely planning to go about reviewing projects that will still be uncertain ~12 months from now. concretely, the way that the philanthropies will review projects a year from now is listed here:
however, you also have a more substantive point:
if that ends up being the case — that, in ~12 months, grantmakers are acting on ~the same information about the projects — i think that we will have ex-post provided little value. however, i think the key here is the extent to which new information is revealed in the next ~12 months. here’s how i’m thinking about it:
there will be less uncertainty about the impactfulness of most projects in ~12 months compared to now. that delta in uncertainty is really important, and the more of it there is, the better. (i’ll touch on the examples you listed in a moment.)
however, the delta in uncertainty changes from one project to another:
for some projects, you get a lot of new information from the point in time they first received investment to the point in time the grantmakers review them.
...but for others, you get little new information; in your words, “grantmakers will [be] view[ing] the projected impact of these projects in ~12 months’ time on ~the same information.”
impact markets work best when they list projects in the first category — ones that have a big delta in uncertainty from now to ~12 months from now; ones for which you get a lot of new information from the point in time they first received investment to the point in time the grantmakers review them. ideally, we want the projects on an impact market to maximize that delta in uncertainty.
my understanding of your claim is: you agree with the above three bullet points, and are concerned that a number of projects that we’re listing will have a really small delta in uncertainty from now to ~12 months from now.
i’m not too worried about this concern, for two overlapping reasons:
a lot happens in a year! for most of the projects, i’d be shocked if they were in a roughly similar position as they are immediately after getting funding — even if the position they’re in is “we straightforwardly did what we said we would do,” you’ve already reduced a lot of uncertainty. i agree that we aren’t truly measuring impact in some platonic sense of impact measurement, but if i were a grantmaker, i would much much much rather be in the position of evaluating a project in the “~12 months from now” category than the “now” category. for the two examples that you happened to list, i think it’s actually quite possible that there will be significant new information that comes out in the ~12 months after a project gets investment:[1]
Someone would like funding for an MPhil or MPhil/PhD.
~12 months from now, they didn’t get into any grad schools/they decided not to go to grad school/etc.
~12 months from now, they got into a great grad school, and their first few semesters they got great grades/published great research/etc.
Someone would like funding to distribute books.
~12 months from now, they took the money, then disappeared/totally failed/the project didn’t materialize/etc.
~12 months from now, they successfully distributed the books, in exactly the way they described.
if we’re running the impact market correctly, the above point (1) should be baked into investment decisions — investors want returns, which incentivizes them to pick the projects that will have the highest delta in uncertainty from now to ~12 months’ time. after all, if a philanthropy reviews a project in a years’ time and sees exactly the same information as an investor does today… then that investor won’t make any returns.
i think you picked up on one potential failure mode, and i’d be interested to see if we end up failing in this way. right now, i’m not too concerned that will happen, though.
also, thanks for your detailed comment — i really appreciate the feedback. if you think i’ve missed something or that i went wrong somewhere, i’d really love to hear your thoughts. again, feel free to leave a comment in response, or if you’d prefer to talk about this over a call, i’d be down! :)
obviously, all of the ones i’m listing are examples, and i’m making no forecast about the probabilities of any of them to actually happen