But abandoning the project of impact markets because of the downsides seems about as misguided to us as abandoning self-driving cars because of adversarial-example attacks on street signs.
I think the analogy would work better if self-driving cars did risky things that could cause a terrible accident, in order to prevent the battery from running out reach the destination sooner.
Attributed Impact may look complicated but we’ve just operationalized something that is intuitively obvious to most EAs – expectational consequentialism. (And moral trade and something broadly akin to UDT.)
I think the following concern (quoted from the OP) is still relevant here:
For that approach to succeed, retro funders must be familiar with it and be sufficiently willing and able to adhere to it. However, some potential retro funders are more likely to use a much simpler approach, such as “you should buy impact that you like”.
Other things being equal, simpler approaches are easier to communicate, more appealing to potential retro funders, more prone to become a meme and a norm, and more likely to be advocated for by teams who work on impact markets and want to get more traction.
You later wrote:
We may sometimes have to explain why it sets bad incentives to fund projects that were net-negative in ex ante expectation to start, but the more sophisticated the funder is, the less likely it is that we need to expound on this.
Does your current plan not involve explaining to all the retro funders that that they should consider the ex-ante EV as an upper bound?
We already can’t prevent anyone from becoming a retro funder. Anyone with money and a sizable Twitter following can reward people for any contributions that they so happen to want to reward them for – be it AI safety papers or how-tos for growing viruses.
I don’t see how this argument works. Given that a naive impact market incentivizes people to treat extremely harmful outcomes as if they were neutral (when deciding what projects to do/fund), why should your above argument cause an update towards the view that launching a certain impact market is net-positive? How does the potential harm that other people can cause via Twitter etc. make launching a certain impact market be a better idea than it would otherwise be?
We think that very few investors will put significant money into a project that is not clearly in line with what major retro funders already explicitly profess to want to retro-fund only because there may later be someone who does.
Why? Conditional on impact markets gaining a lot of traction and retro funders spending billions of dollars in impact markets 5 years from now, why wouldn’t it make sense to buy many certificates of risky projects that might end up being extremely beneficial (according to at least one relevant future retro funder)?
An important safety mechanism that we have already started implementing is to reward solutions to problems with impact markets. A general ban on using such rewards would remove this promising mechanism.
Do you intent to allow people to profit from outreach interventions that attract new retro funders? (i.e. by allowing people to sell certificates of such outreach interventions.)
“A naive implementation of this idea would incentivize people to launch a safe project and later expand it to include high-risk high-reward interventions” – That would have to be a very naive implementation because if the actual project is different from the project certified in the certificate, then the certificate does not describe it. It’s a certificate for a different project that failed to happen.
I disagree. I think this risk can easily materialize if the description of the certificate is not very specific (and in particular if it’s about starting an organization, without listing specific interventions.)
First of all, what we’ve summarized as “curation” so far could really be distinguished as follows:
Making access for issuers invite-only, maybe keeping the whole marketplace secret (in combination with #2) until we find someone who produces cool papers/articles and who we trust and then invite them.
Making access for investors/retro funders invite-only, maybe keeping the whole marketplace secret (in combination with #1) until we find an impact investor or a retro funder who we trust and then invite them.
Read every certificate either before or shortly after it is published. (In combination with exposé certificates in case we make a mistake.)
Let’s say #3 is a given. Do you think the marketplace would fulfill your safety requirements if only #1, only #2, or both were added to it?
Does your current plan not involve explaining to all the retro funders that that they should consider the ex-ante EV as an upper bound?
It involves explaining that. What we wrote was to argue that Attributed Impact is not as complicated as it may sound but rather quite intuitive.
How does the potential harm that other people can cause via Twitter etc. make launching a certain impact market be a better idea than it would otherwise be?
If you want to open a bazaar, one of your worries could be that people will use it to sell stolen goods. Currently these people sell the stolen goods online or on other bazaars, and the experience may be a bit clunky. By default these people will be happy to use your bazaar for their illegal trade because it makes life slightly easier for them. Slightly easier could mean that they get to sell a bit more quickly and create a bit more capacity for more stealing.
But if you enact some security measures to keep them out, you quickly reach the point where the bazaar is less attractive than the alternatives. At that point you already have no effect anymore on how much theft there is going on in the world in aggregate.
So the trick is to tune the security measures just right that they make the place less attractive than alternatives to the thieves and yet don’t impose prohibitively high costs on the legitimate sellers.
Do you intent to allow people to profit from outreach interventions that attract new retro funders? (i.e. by allowing people to sell certificates of such outreach interventions.)
My intent so far was to focus on text that is accessible online, e.g., articles, papers, some books. There may be other classes of things that are similarly strong candidates. Outreach seems like a bad fit to me. I’ve so far only considered it once when someone (maybe you) brought it up as something that’d be a bad fit for an impact market and I agreed.
I disagree. I think this risk can easily materialize if the description of the certificate is not very specific (and in particular if it’s about starting an organization, without listing specific interventions.)
Also very bad fit for an impact market as we envision it. To be a good fit, the object needs to have some cultural rights along the lines of ownership or copyright associated with it so market participants can agree on an owner. It needs to have a start and an end in time. It need so generate a verifiable artifact. Finally, it should not try super hard to try to fit something into that mold that doesn’t fit. There are a bunch of examples in my big post. So a paper, article, book, etc. (a particular version of it) is great. Something ongoing like starting an org is not a good fit. Something where you influence others and most of your impact is leveraging behavior change of others is really awkward because you can’t credibly assign an owner.
First of all, what we’ve summarized as “curation” so far could really be distinguished as follows:
Making access for issuers invite-only, maybe keeping the whole marketplace secret (in combination with #2) until we find someone who produces cool papers/articles and who we trust and then invite them.
Making access for investors/retro funders invite-only, maybe keeping the whole marketplace secret (in combination with #1) until we find an impact investor or a retro funder who we trust and then invite them.
Read every certificate either before or shortly after it is published. (In combination with exposé certificates in case we make a mistake.)
Let’s say #3 is a given. Do you think the marketplace would fulfill your safety requirements if only #1, only #2, or both were added to it?
An impact market with invite-only access for issuers and investors seems safer than otherwise. But will that be a temporary phase after which our civilization ends up with a decentralized impact market that nobody can control or shut down, and people are incentivized to recruit as many new retro funders as they can? In the Toward Impact Markets post (March 2022) you wrote:
We are fairly convinced that the blockchain-based solution is going to be the culmination of our efforts one day, but we’re ambivalent over which MVP will allow us to test the market more quickly and productively.
That came after the sentence “A web2 solution like that would have a few advantages too:”, after which you listed three advantages that have nothing to do with safety.
But if you enact some security measures to keep them out, you quickly reach the point where the bazaar is less attractive than the alternatives. At that point you already have no effect anymore on how much theft there is going on in the world in aggregate.
I don’t think the analogy works. Right now, there seems to be no large-scale retroactive funding mechanisms for anthropogenic x-risk interventions. Launching an impact market can change that. An issuer/investor/funder who will use your impact market would probably not use Twitter or anything else to deal with retroactive funding if you did not launch your impact market. The distribution mismatch problem applies to those people. (In your analogy there’s a dichotomy of good people vs. thieves, which has no clear counterpart in the domain of retroactive funding.) Also, if your success inspires others to launch/join competing impact markets, you can end up increasing the number of people who use the other markets.
I think the analogy would work better if self-driving cars did risky things that could cause a terrible accident, in order to
prevent the battery from running outreach the destination sooner.I think the following concern (quoted from the OP) is still relevant here:
You later wrote:
Does your current plan not involve explaining to all the retro funders that that they should consider the ex-ante EV as an upper bound?
I don’t see how this argument works. Given that a naive impact market incentivizes people to treat extremely harmful outcomes as if they were neutral (when deciding what projects to do/fund), why should your above argument cause an update towards the view that launching a certain impact market is net-positive? How does the potential harm that other people can cause via Twitter etc. make launching a certain impact market be a better idea than it would otherwise be?
Why? Conditional on impact markets gaining a lot of traction and retro funders spending billions of dollars in impact markets 5 years from now, why wouldn’t it make sense to buy many certificates of risky projects that might end up being extremely beneficial (according to at least one relevant future retro funder)?
Do you intent to allow people to profit from outreach interventions that attract new retro funders? (i.e. by allowing people to sell certificates of such outreach interventions.)
I disagree. I think this risk can easily materialize if the description of the certificate is not very specific (and in particular if it’s about starting an organization, without listing specific interventions.)
First of all, what we’ve summarized as “curation” so far could really be distinguished as follows:
Making access for issuers invite-only, maybe keeping the whole marketplace secret (in combination with #2) until we find someone who produces cool papers/articles and who we trust and then invite them.
Making access for investors/retro funders invite-only, maybe keeping the whole marketplace secret (in combination with #1) until we find an impact investor or a retro funder who we trust and then invite them.
Read every certificate either before or shortly after it is published. (In combination with exposé certificates in case we make a mistake.)
Let’s say #3 is a given. Do you think the marketplace would fulfill your safety requirements if only #1, only #2, or both were added to it?
It involves explaining that. What we wrote was to argue that Attributed Impact is not as complicated as it may sound but rather quite intuitive.
If you want to open a bazaar, one of your worries could be that people will use it to sell stolen goods. Currently these people sell the stolen goods online or on other bazaars, and the experience may be a bit clunky. By default these people will be happy to use your bazaar for their illegal trade because it makes life slightly easier for them. Slightly easier could mean that they get to sell a bit more quickly and create a bit more capacity for more stealing.
But if you enact some security measures to keep them out, you quickly reach the point where the bazaar is less attractive than the alternatives. At that point you already have no effect anymore on how much theft there is going on in the world in aggregate.
So the trick is to tune the security measures just right that they make the place less attractive than alternatives to the thieves and yet don’t impose prohibitively high costs on the legitimate sellers.
My intent so far was to focus on text that is accessible online, e.g., articles, papers, some books. There may be other classes of things that are similarly strong candidates. Outreach seems like a bad fit to me. I’ve so far only considered it once when someone (maybe you) brought it up as something that’d be a bad fit for an impact market and I agreed.
Also very bad fit for an impact market as we envision it. To be a good fit, the object needs to have some cultural rights along the lines of ownership or copyright associated with it so market participants can agree on an owner. It needs to have a start and an end in time. It need so generate a verifiable artifact. Finally, it should not try super hard to try to fit something into that mold that doesn’t fit. There are a bunch of examples in my big post. So a paper, article, book, etc. (a particular version of it) is great. Something ongoing like starting an org is not a good fit. Something where you influence others and most of your impact is leveraging behavior change of others is really awkward because you can’t credibly assign an owner.
An impact market with invite-only access for issuers and investors seems safer than otherwise. But will that be a temporary phase after which our civilization ends up with a decentralized impact market that nobody can control or shut down, and people are incentivized to recruit as many new retro funders as they can? In the Toward Impact Markets post (March 2022) you wrote:
That came after the sentence “A web2 solution like that would have a few advantages too:”, after which you listed three advantages that have nothing to do with safety.
I don’t think the analogy works. Right now, there seems to be no large-scale retroactive funding mechanisms for anthropogenic x-risk interventions. Launching an impact market can change that. An issuer/investor/funder who will use your impact market would probably not use Twitter or anything else to deal with retroactive funding if you did not launch your impact market. The distribution mismatch problem applies to those people. (In your analogy there’s a dichotomy of good people vs. thieves, which has no clear counterpart in the domain of retroactive funding.) Also, if your success inspires others to launch/join competing impact markets, you can end up increasing the number of people who use the other markets.