For a project to be considered for retroactive funding, participants must post a specific amount of money as collateral.
If a retroactive funder determines that the project was net-negative, they can burn the collateral to punish the people that participated in it. Otherwise, the project receives its collateral back.
This eliminates the “no downside” problem of retroactive funding and makes some net-negative projects unprofitable.
The amount of collateral can be chosen adaptively. Start with a small amount and increase it slowly until the number of net-negative projects is low enough. Note that setting the collateral too high can discourage net-positive but risky projects.
Related: requiring some kind of insurance that pays out when a certificate becomes net-negative.
Suppose we somehow have accurate positive and negative valuations of certificates. We can have insurers sell put options on certificates, and be required to maintain that their portfolio has positive overall impact. (So an insurer needs to buy certificates of positive impact to offset negative impact they’ve taken on.)
Ultimately what’s at stake for the insurer is probably some collateral they’ve put down, so it’s a similar proposal.
Crypto’s inability to take debts or enact substantial punishments beyond slashing stakes is a huge limitation and I would like it if we didn’t have to swallow that (ie, if we could just operate in the real world, with non-anonymous impact traders, who can be held accountable for more assets than they’d be willing to lock in a contract.)
Given enough of that, we would be able to implement this by just having an impact cert that’s implicated in a catastrophe turn into debt/punishment, and we’d be able to make that disincentive a lot more proportional to the scale of its potential negative externalities, and we would be able to allow the market to figure out how big that risk is for itself, which is pretty much the point of an impact market.
Though, on reflection, I’m not sure I would want to let the market to decide that. The problem with markets is that they give us a max function, they’re made of auctions, whoever pays most decides the price, and the views of everyone else are not taken into account at all. Markets, in a sense, subject us to the decisions of the people with the most extreme beliefs. Eventually the ones who are extreme and wrong go bankrupt and disappear, but I don’t find this very reassuring, with rare catastrophic risks, which no market participant can have prior experience of. It’s making me think of the unilateralist’s curse. So, yeah, maybe we shouldn’t use market processes to price risk of negative externalities.
Nice, that’s pretty interesting. (It’s hacky, but that seems okay.)
It’s easy to see how this works in cases where there’s a single known-in-advance funder that people are aiming to get retro funding from (evaluated in five years, say). Have you thought about whether it could work with a more free market, and not necessarily knowing all of the funders in advance?
This kind of thing could be made more sophisticated by making fines proportional to the harm done, requiring more collateral for riskier projects, or setting up a system to short sell different projects. But simpler seems better, at least initially.
Have you thought about whether it could work with a more free market, and not necessarily knowing all of the funders in advance?
Yeah, that’s a harder case. Some ideas:
People undertaking projects could still post collateral on their own (or pre-commit to accepting a fine under certain conditions). This kind of behavior could be rewarded by retro-funders giving these projects more consideration and the act of posting collateral does constitute a costly signal of quality. But that still requires some pre-commitments from retro funders or a general consensus from the community.
If contributors undertake multiple projects it should be possible to punish after-the-fact by docking some of their rewards from other projects. For example, if someone participates in 1 beneficial project and 1 harmful project, their retro funding rewards from the beneficial project can be reduced due to their participation on the harmful project. Unfortunately, this still requires some sort of pre-commitment from funders.
This kind of thing could be made more sophisticated by making fines proportional to the harm done
I was thinking of this. Small funders could then potentially buy insurance from large funders in order to allow them to fund projects that they deem net positive even though there’s a small risk of a fine that would be too costly for them.
I take it that Harsimony is proposing for the IC-seller to put up a flexible amount of collateral when they start their project, according to the possible harms.
There are two problems, though:
This requires centralised prospective estimation of harms for every project. (A big part of the point of impact certificates is to evaluate things retroactively, and to outsource prospective evaluations to the market, thereby incentivising accuracy in the latter.
This penalises IC-sellers based on how big their harms initially seem, rather than how big they eventually turn out to be.
It would be better if the IC-seller is required to buy insurance that will pay out the whole cost of the harm, as evaluated retrospectively. In order for the IC-seller to prove that they are willing to be accountable for all harms, they must buy insurance when they sell their IC. And to ensure that the insurer will pay out correctly, we must only allow insurers who use a standard, trusted board of longtermist evaluators to estimate the harms.
This means that a centralised system is only required to provide occasional retrospective evaluations of harm. The task of evaluating harms in prospect is delegated to insurers, similar to the role insurers play in the real world.
(This is my analysis, but the insurance idea was from Stefan.)
Although, the costs of insurance would need to be priced according to the ex ante costs, not the ex post costs.
For example: Bob embarks on a project with a 50% chance of success. If it succeeds, it saves one person’s life, and Bob sells the IC. If it fails, it kills two people.
Clearly, the insurance needs to be priced to take into account a 50% chance of two deaths. So we would have to require Bob to buy the insurance when he initially embarks on the project (which is a tough ask, given that few currently anticipate selling their impact). Or else we would need to rely on a (centralised) retrospective evaluation of ex ante harm, for every project (which seems laborious).
I love the insurance idea because compared to our previous ideas around shorting with hedge tokens that compound automatically to maintain a −1x leverage, collateral, etc. (see Toward Impact Markets), the insurance idea also has the potential of solving the incentive problems that we face around setting up our network of certificate auditors! (Strong upvotes to both of you!)
(The insurances would function a bit like the insurances in Robin Hanson’s idea for a tort law reform.)
or setting up a system to short sell different projects.
I don’t think that short selling would work. Suppose a net-negative project has a 10% chance to end up being beneficial, in which case its certificates will be worth $1M (and otherwise the certificates will end up being worth $0). Therefore, the certificates are worth today $100K in expectation. If someone shorts the certificates as if they are worth less than that, they will lose money in expectation.
I don’t think such a rule has a chance of surviving if impact markets take off?
Added complexity to the norms for trading needs to pay for itself to withstand friction or else decay to its most intuitive equilibrium.
Or the norm for punishing defectors needs to pay for itself in order to stay in equilibrium.
Or someone needs to pay the cost of punishing defectors out of pocket for altruistic reasons.
Once a collateral-charging market takes off, someone could just start up an exchange that doesn’t demand a collateral, and instead just charge a nominal fee that doesn’t disincentivise risky investments but would still make them money. Traders would defect to this market if it’s more profitable for them.
(To be clear, I think I’m very pro GoodX’s project here; I’m just skeptical of the collateral suggestion.)
Traders would adopt a competitor without negative externality mechanisms, but charities wouldn’t, there will be no end buyers there, I wouldn’t expect that kind of vicious amoral competitive pressure between platforms to play out.
But afaik the theory of change of this project doesn’t rely on altruistic “end buyers”, it relies on profit-motivated speculation? At least, the aim is to make it work even in the worst-case scenario where traders are purely motivated by profit, and still have the trades generate altruistic value. Correct me if I’m wrong,
My understanding is that without altruistic end-buyers, then the intrinsic value of impact certificates becomes zero and it’s entirely a confidence game.
There might be a market for that sort of ultimately valueless token now (or several months ago? I haven’t been following the NFT stuff), I’m not sure there will be for long.
I think there’s an argument for the thing you were saying, though… Something like… If one marketplace forbids most foundational AI public works, then another marketplace will pop up with a different negative externality estimation process, and it wont go away, and most charities and government funders still aren’t EA and don’t care about undiscounted expected utility, so there’s a very real risk that that marketplace would become the largest one.
I guess there might not be many people who are charitibly inclined, and who could understand, believe in, and adopt impact markets, but also don’t believe in tail risks. There are lots of people who do one of those things, but I’m not sure there are any who do all.
I proposed a simple solution to the problem:
For a project to be considered for retroactive funding, participants must post a specific amount of money as collateral.
If a retroactive funder determines that the project was net-negative, they can burn the collateral to punish the people that participated in it. Otherwise, the project receives its collateral back.
This eliminates the “no downside” problem of retroactive funding and makes some net-negative projects unprofitable.
The amount of collateral can be chosen adaptively. Start with a small amount and increase it slowly until the number of net-negative projects is low enough. Note that setting the collateral too high can discourage net-positive but risky projects.
Related: requiring some kind of insurance that pays out when a certificate becomes net-negative.
Suppose we somehow have accurate positive and negative valuations of certificates. We can have insurers sell put options on certificates, and be required to maintain that their portfolio has positive overall impact. (So an insurer needs to buy certificates of positive impact to offset negative impact they’ve taken on.)
Ultimately what’s at stake for the insurer is probably some collateral they’ve put down, so it’s a similar proposal.
Crypto’s inability to take debts or enact substantial punishments beyond slashing stakes is a huge limitation and I would like it if we didn’t have to swallow that (ie, if we could just operate in the real world, with non-anonymous impact traders, who can be held accountable for more assets than they’d be willing to lock in a contract.)
Given enough of that, we would be able to implement this by just having an impact cert that’s implicated in a catastrophe turn into debt/punishment, and we’d be able to make that disincentive a lot more proportional to the scale of its potential negative externalities, and we would be able to allow the market to figure out how big that risk is for itself, which is pretty much the point of an impact market.
Though, on reflection, I’m not sure I would want to let the market to decide that. The problem with markets is that they give us a max function, they’re made of auctions, whoever pays most decides the price, and the views of everyone else are not taken into account at all. Markets, in a sense, subject us to the decisions of the people with the most extreme beliefs. Eventually the ones who are extreme and wrong go bankrupt and disappear, but I don’t find this very reassuring, with rare catastrophic risks, which no market participant can have prior experience of. It’s making me think of the unilateralist’s curse.
So, yeah, maybe we shouldn’t use market processes to price risk of negative externalities.
Nice, that’s pretty interesting. (It’s hacky, but that seems okay.)
It’s easy to see how this works in cases where there’s a single known-in-advance funder that people are aiming to get retro funding from (evaluated in five years, say). Have you thought about whether it could work with a more free market, and not necessarily knowing all of the funders in advance?
This kind of thing could be made more sophisticated by making fines proportional to the harm done, requiring more collateral for riskier projects, or setting up a system to short sell different projects. But simpler seems better, at least initially.
Yeah, that’s a harder case. Some ideas:
People undertaking projects could still post collateral on their own (or pre-commit to accepting a fine under certain conditions). This kind of behavior could be rewarded by retro-funders giving these projects more consideration and the act of posting collateral does constitute a costly signal of quality. But that still requires some pre-commitments from retro funders or a general consensus from the community.
If contributors undertake multiple projects it should be possible to punish after-the-fact by docking some of their rewards from other projects. For example, if someone participates in 1 beneficial project and 1 harmful project, their retro funding rewards from the beneficial project can be reduced due to their participation on the harmful project. Unfortunately, this still requires some sort of pre-commitment from funders.
I was thinking of this. Small funders could then potentially buy insurance from large funders in order to allow them to fund projects that they deem net positive even though there’s a small risk of a fine that would be too costly for them.
I take it that Harsimony is proposing for the IC-seller to put up a flexible amount of collateral when they start their project, according to the possible harms.
There are two problems, though:
This requires centralised prospective estimation of harms for every project. (A big part of the point of impact certificates is to evaluate things retroactively, and to outsource prospective evaluations to the market, thereby incentivising accuracy in the latter.
This penalises IC-sellers based on how big their harms initially seem, rather than how big they eventually turn out to be.
It would be better if the IC-seller is required to buy insurance that will pay out the whole cost of the harm, as evaluated retrospectively. In order for the IC-seller to prove that they are willing to be accountable for all harms, they must buy insurance when they sell their IC. And to ensure that the insurer will pay out correctly, we must only allow insurers who use a standard, trusted board of longtermist evaluators to estimate the harms.
This means that a centralised system is only required to provide occasional retrospective evaluations of harm. The task of evaluating harms in prospect is delegated to insurers, similar to the role insurers play in the real world.
(This is my analysis, but the insurance idea was from Stefan.)
Although, the costs of insurance would need to be priced according to the ex ante costs, not the ex post costs.
For example: Bob embarks on a project with a 50% chance of success. If it succeeds, it saves one person’s life, and Bob sells the IC. If it fails, it kills two people.
Clearly, the insurance needs to be priced to take into account a 50% chance of two deaths. So we would have to require Bob to buy the insurance when he initially embarks on the project (which is a tough ask, given that few currently anticipate selling their impact). Or else we would need to rely on a (centralised) retrospective evaluation of ex ante harm, for every project (which seems laborious).
I love the insurance idea because compared to our previous ideas around shorting with hedge tokens that compound automatically to maintain a −1x leverage, collateral, etc. (see Toward Impact Markets), the insurance idea also has the potential of solving the incentive problems that we face around setting up our network of certificate auditors! (Strong upvotes to both of you!)
(The insurances would function a bit like the insurances in Robin Hanson’s idea for a tort law reform.)
I don’t think that short selling would work. Suppose a net-negative project has a 10% chance to end up being beneficial, in which case its certificates will be worth $1M (and otherwise the certificates will end up being worth $0). Therefore, the certificates are worth today $100K in expectation. If someone shorts the certificates as if they are worth less than that, they will lose money in expectation.
I don’t think such a rule has a chance of surviving if impact markets take off?
Added complexity to the norms for trading needs to pay for itself to withstand friction or else decay to its most intuitive equilibrium.
Or the norm for punishing defectors needs to pay for itself in order to stay in equilibrium.
Or someone needs to pay the cost of punishing defectors out of pocket for altruistic reasons.
Once a collateral-charging market takes off, someone could just start up an exchange that doesn’t demand a collateral, and instead just charge a nominal fee that doesn’t disincentivise risky investments but would still make them money. Traders would defect to this market if it’s more profitable for them.
(To be clear, I think I’m very pro GoodX’s project here; I’m just skeptical of the collateral suggestion.)
Traders would adopt a competitor without negative externality mechanisms, but charities wouldn’t, there will be no end buyers there, I wouldn’t expect that kind of vicious amoral competitive pressure between platforms to play out.
But afaik the theory of change of this project doesn’t rely on altruistic “end buyers”, it relies on profit-motivated speculation? At least, the aim is to make it work even in the worst-case scenario where traders are purely motivated by profit, and still have the trades generate altruistic value. Correct me if I’m wrong,
Update: If it wasn’t clear, I was wrong. :p
My understanding is that without altruistic end-buyers, then the intrinsic value of impact certificates becomes zero and it’s entirely a confidence game.
There might be a market for that sort of ultimately valueless token now (or several months ago? I haven’t been following the NFT stuff), I’m not sure there will be for long.
Aye, I updated. I was kinda dumb. The magical speculation model is probably not worth going for when end-buyers seem within reach.
I think there’s an argument for the thing you were saying, though… Something like… If one marketplace forbids most foundational AI public works, then another marketplace will pop up with a different negative externality estimation process, and it wont go away, and most charities and government funders still aren’t EA and don’t care about undiscounted expected utility, so there’s a very real risk that that marketplace would become the largest one.
I guess there might not be many people who are charitibly inclined, and who could understand, believe in, and adopt impact markets, but also don’t believe in tail risks. There are lots of people who do one of those things, but I’m not sure there are any who do all.