If someone thinks a net-negative project is being traded on (or run at all), how about posting about it on the forum?
As we wrote in the post, even if everyone believes that a certain project is net-negative, its certificates may be traded for a high price due to the chance that the project will end up being beneficial. For example, consider OpenAI (I’m not making here a claim that OpenAI is net-negative, but it seems that many people in EA think it is, and for the sake of this example let’s imagine that everyone in EA think that). It’s plausible that OpenAI will end up being extremely beneficial. Therefore, if a naive impact market had existed when OpenAI was created, it’s likely that the market would have helped in funding its creation (i.e. OpneAI’s certificates would have been traded for a high price).
Also, it seems that people in EA (and in academia/industry in general) usually avoid saying bad things publicly about others’ work (due to reasons that are hard to nullify). Another point to consider is that saying that a project is net-negative publicly can sometimes in itself be net-negative due to drawing attention to info hazards. (e.g. “The experiment that Alice is working on is dangerous!”)
The problem of funding net-negative projects exists also now.
As I already wrote in a reply to Austin, impact markets can incentivize/fund net-negative projects that are not currently of interest to for-profit investors. For example, today it can be impossible for someone to make a huge amount of money by launching an aggressive outreach campaign to make people join EA, or publishing a list of “the most dangerous ongoing experiments in virology that we should advocate to stop”; which are interventions that may be net-negative. (Also, in cases where both impact markets and existing mechanisms incentivize a project, one can flip your argument and say that the solution to funding net-positive projects already exist and so we don’t need impact markets. To be clear, I’m not making that argument, I’m just trying to show that the original one is wrong.)
This is a kind of project that we can stop or change if we want to. There is a lot of human discretion. This is not like adding a government regulation that will be very hard to change, or launching a blockchain that you can pretty much never take back no matter what you do.
Shutting down an impact market, if successful, functionally means burning all the certificates that are owned by the market participants, who may have already spent a lot of resources and time in the hope to profit from selling their certificates in the future. Obviously, that may not be an easy action for the decision makers to take. Also, if the decision makers have conflicts of interest with respect to shutting down the market, things are even more problematic (which is an important topic that is discussed in the post.) [EDIT: Also, my understanding is that there was (and perhaps still is) an intention to launch a decentralized impact market (i.e. Web3 based), which can be impossible to shut down.]
“OpenAI is expected to be negative on net even if it ends up randomly working [...] So retro funders, if you agree with this, please don’t retro-fund OpenAI, even if it works”
I expect this will reduce the price at which OpenAI is traded
People in EA (and in academia/industry in general) usually avoid saying bad things publicly about others’ work
Yeah. Anonymous comments? Messaging FTX (or who ever does the retro funding) ? This seems like the kind of thing we’d learn on the fly, no?
Or an item on the checklist before retro-funding?
This problem already exists today when you fund someone for work that they’re planning to do, no?
launching an aggressive outreach campaign to make people join EA
I bet EA has this problem today already and CEA deals with it somehow. Wanna ask them?
(In other words, doesn’t seem like a new probelm. No?)
Publishing a list of “the most dangerous ongoing experiments in virology that we should advocate to stop”
I assume CEA+LW already remove such infohazards from the forum(s) and would reach out to EAs who publish this stuff elsewhere, if it comes to their attention.
= probably not a new problem.
And you could probably ask CEA/LW what they think since it’s their domain (?) They might say this is totally a huge problem and making it worse is really bad, I don’t know
One can flip your argument and say that the solution to funding net-positive projects already exist and so we don’t need impact markets
Nice! (I like this kind of argument <3 )
I missed what you’re replying to though. Is it the “The problem of funding net-negative projects exists also now.” ?
I’m pointing at “we’re not creating any new problem [that we have no mechanism to solve]”. (I’ll wait for your reply here since I suspect I missed your point)
Human discretion / Shutting down an impact market
What I actually mean is not that the people running the market will shut it down.
What I mean is that the retro funders can decide which projects to fund or not to fund and they have a ton of flexibility around this. (I can say more)
I expect this will reduce the price at which OpenAI is traded
But an impact market can still make OpenAI’s certificates be worth $100M if, for example, investors have at least 10% credence in some future retro funder being willing to buy them for $1B (+interest). And that could be true even if everyone today believed that creating OpenAI is net-negative. See the “Mitigating the risk is hard” section in the OP for some additional reasons to be skeptical about such an approach.
I missed what you’re replying to though. Is it the “The problem of funding net-negative projects exists also now.” ?
Yes. You respond to examples of potential harm that impact markets can cause by pointing out that these things can happen even without impact markets. I don’t see why these arguments should be more convincing than the flipped argument: “everything that impact markets can fund can already be funded in other ways, so we don’t need impact markets”. (Again, I’m not saying that the flipped argument makes sense.)
Your overall view seems to be something like: we should just create an impact market and if it causes harm then the retro funders will notice and stop buying certificates (or they will stop buying some particular certificates that are net-negative to buy). I disagree with this view because:
There is a dire lack of feedback signal in the realm of x-risk mitigation. It’s usually very hard to judge whether a given intervention was net-positive or net-negative. It’s not just a matter of asking CEA / LW / anyone else what they think about a particular intervention, because usually no one on Earth can do a reliable, robust evaluation. (e.g. is the creation of OpenAI/Anthropic net positive or net negative?) So, if you buy the core argument in the OP (about how naive impact markets incentivize people to carry out interventions without considering potential outcomes that are extremely harmful), I think that you shouldn’t create an impact market and rely on some unspecified future feedback signals to make retro funders stop buying certificates in a net-negative way at some unspecified point in the future.
As I argued in the grandparent comment, we should expect the things that people in EA say about the impact of others in EA to be positively biased.
All the above assumes that by “retro funders” here you mean a set of carefully appointed Final Buyers. If instead we’re talking about an impact market where anyone can become a retro funder, and retro funders can resell their impact to arbitrary future retro funders, I think things would go worse in expectation (see the first three points in the section “Mitigating the risk is hard” in the OP).
Shutting down an impact market, if successful, functionally means burning all the certificates that are owned by the market participants, who may have already spent a lot of resources and time in the hope to profit from selling their certificates in the future.
It could be done a bit more smoothly by (1) accepting no new issues, (2) completing all running prize rounds, and (3) declaring the impact certificates not burned and allowing people some time to export their data. (I don’t think it would be credible for the marketplace to declare the certs burned since it doesn’t own them.)
Also, my understanding is that there was (and perhaps still is) an intention to launch a decentralized impact market (i.e. Web3 based), which can be impossible to shut down.
My original idea from summer 2021 was to use blockchain technology simply for technical ease of implementation (I wouldn’t have had to write any code). That would’ve made the certs random tokens among millions of others on the blockchain. But then to set up a centralized, curated marketplace for them with a smart and EA curation team.
We’ve moved away from that idea. Our current market is fully web2 with no bit of blockchain anywhere. Safety was a core reason for the update. (But the ease-of-implementation reasons to prefer blockchain also didn’t apply so much anymore. We have a doc somewhere with all the pros and cons.)
For our favored auction mechanisms, it would be handy to be able to split transactions easily, so we have thought about (maybe, at some point) allowing users to connect a wallet to improve the user experience, but that would be only for sending and receiving payments. The certs would still be rows in a Postgres database in this hypothetical model. Sort of like how Rethink Priorities accepts crypto donations or a bit like a centralized crypto exchange (but that sounds a bit pompous).
But what do you think about the original idea? I don’t think it’s so different from a fully centralized solution where you allow people to export their data or at least not prevent them from copy-pasting their certs and ledgers to back them up.
My greatest worries about crypto stem less from the technology itself (which, for all I know, could be made safe) but from the general spirit in the community that decentralization, democratization, ungatedness, etc. are highly desirable values to strive for. I don’t want to have to fight against the dominant paradigms, so that doing it on my server was more convenient. But then again big players in the Ethereum space have implemented very much expert-run systems with no permissionless governance tokens and such. So I hope (and think) that there are groups that can be convinced that an impact market should be gated and curated by trusted experts only.
But even so, a solution that is crypto-based beyond making payments easier is something that I consider more in the context of joining existing efforts to make them safer rather than actions that would influence their existence.
(3) declaring the impact certificates not burned and allowing people some time to export their data.
That could make it easier for another team to create a new impact market that will seamlessly replace the impact market that is being shut down.
My original idea from summer 2021 was to use blockchain technology simply for technical ease of implementation (I wouldn’t have had to write any code). That would’ve made the certs random tokens among millions of others on the blockchain. But then to set up a centralized, curated marketplace for them with a smart and EA curation team.
[…]
But what do you think about the original idea? I don’t think it’s so different from a fully centralized solution where you allow people to export their data or at least not prevent them from copy-pasting their certs and ledgers to back them up.
If a decentralized impact market gains a lot of traction, I don’t see how the certificates being “tokens among millions of others” helps. A particular curated gallery can end up being ignored by some/most market participants (and perhaps be outcompeted by another, less scrupulous curated gallery).
Allowing people to make backups: You’d rather make it as hard as possible to make backups, e.g., by using anti-screenscraping tools and maybe hiding some information about the ledger in the first place so people can’t easily back it up.
Web3: Seems about as bad as any web2 solution that allows people to easily back up their data.
Web3: Seems about as bad as any web2 solution that allows people to easily back up their data.
I think that a decentralized impact market that can’t be controlled or shut down seems worse. Also, a Web3 platform will make it less effortful for someone to launch a competing platform (either with or without the certificates from the original platform).
As we wrote in the post, even if everyone believes that a certain project is net-negative, its certificates may be traded for a high price due to the chance that the project will end up being beneficial. For example, consider OpenAI (I’m not making here a claim that OpenAI is net-negative, but it seems that many people in EA think it is, and for the sake of this example let’s imagine that everyone in EA think that). It’s plausible that OpenAI will end up being extremely beneficial. Therefore, if a naive impact market had existed when OpenAI was created, it’s likely that the market would have helped in funding its creation (i.e. OpneAI’s certificates would have been traded for a high price).
Also, it seems that people in EA (and in academia/industry in general) usually avoid saying bad things publicly about others’ work (due to reasons that are hard to nullify). Another point to consider is that saying that a project is net-negative publicly can sometimes in itself be net-negative due to drawing attention to info hazards. (e.g. “The experiment that Alice is working on is dangerous!”)
As I already wrote in a reply to Austin, impact markets can incentivize/fund net-negative projects that are not currently of interest to for-profit investors. For example, today it can be impossible for someone to make a huge amount of money by launching an aggressive outreach campaign to make people join EA, or publishing a list of “the most dangerous ongoing experiments in virology that we should advocate to stop”; which are interventions that may be net-negative. (Also, in cases where both impact markets and existing mechanisms incentivize a project, one can flip your argument and say that the solution to funding net-positive projects already exist and so we don’t need impact markets. To be clear, I’m not making that argument, I’m just trying to show that the original one is wrong.)
Shutting down an impact market, if successful, functionally means burning all the certificates that are owned by the market participants, who may have already spent a lot of resources and time in the hope to profit from selling their certificates in the future. Obviously, that may not be an easy action for the decision makers to take. Also, if the decision makers have conflicts of interest with respect to shutting down the market, things are even more problematic (which is an important topic that is discussed in the post.) [EDIT: Also, my understanding is that there was (and perhaps still is) an intention to launch a decentralized impact market (i.e. Web3 based), which can be impossible to shut down.]
OpenAI
I recommend that early on someone posts:
“OpenAI is expected to be negative on net even if it ends up randomly working [...] So retro funders, if you agree with this, please don’t retro-fund OpenAI, even if it works”
I expect this will reduce the price at which OpenAI is traded
People in EA (and in academia/industry in general) usually avoid saying bad things publicly about others’ work
Yeah. Anonymous comments? Messaging FTX (or who ever does the retro funding) ? This seems like the kind of thing we’d learn on the fly, no?
Or an item on the checklist before retro-funding?
This problem already exists today when you fund someone for work that they’re planning to do, no?
launching an aggressive outreach campaign to make people join EA
I bet EA has this problem today already and CEA deals with it somehow. Wanna ask them?
(In other words, doesn’t seem like a new probelm. No?)
Publishing a list of “the most dangerous ongoing experiments in virology that we should advocate to stop”
I assume CEA+LW already remove such infohazards from the forum(s) and would reach out to EAs who publish this stuff elsewhere, if it comes to their attention.
= probably not a new problem.
And you could probably ask CEA/LW what they think since it’s their domain (?) They might say this is totally a huge problem and making it worse is really bad, I don’t know
One can flip your argument and say that the solution to funding net-positive projects already exist and so we don’t need impact markets
Nice! (I like this kind of argument <3 )
I missed what you’re replying to though. Is it the “The problem of funding net-negative projects exists also now.” ?
I’m pointing at “we’re not creating any new problem [that we have no mechanism to solve]”. (I’ll wait for your reply here since I suspect I missed your point)
Human discretion / Shutting down an impact market
What I actually mean is not that the people running the market will shut it down.
What I mean is that the retro funders can decide which projects to fund or not to fund and they have a ton of flexibility around this. (I can say more)
But an impact market can still make OpenAI’s certificates be worth $100M if, for example, investors have at least 10% credence in some future retro funder being willing to buy them for $1B (+interest). And that could be true even if everyone today believed that creating OpenAI is net-negative. See the “Mitigating the risk is hard” section in the OP for some additional reasons to be skeptical about such an approach.
Yes. You respond to examples of potential harm that impact markets can cause by pointing out that these things can happen even without impact markets. I don’t see why these arguments should be more convincing than the flipped argument: “everything that impact markets can fund can already be funded in other ways, so we don’t need impact markets”. (Again, I’m not saying that the flipped argument makes sense.)
Your overall view seems to be something like: we should just create an impact market and if it causes harm then the retro funders will notice and stop buying certificates (or they will stop buying some particular certificates that are net-negative to buy). I disagree with this view because:
There is a dire lack of feedback signal in the realm of x-risk mitigation. It’s usually very hard to judge whether a given intervention was net-positive or net-negative. It’s not just a matter of asking CEA / LW / anyone else what they think about a particular intervention, because usually no one on Earth can do a reliable, robust evaluation. (e.g. is the creation of OpenAI/Anthropic net positive or net negative?) So, if you buy the core argument in the OP (about how naive impact markets incentivize people to carry out interventions without considering potential outcomes that are extremely harmful), I think that you shouldn’t create an impact market and rely on some unspecified future feedback signals to make retro funders stop buying certificates in a net-negative way at some unspecified point in the future.
As I argued in the grandparent comment, we should expect the things that people in EA say about the impact of others in EA to be positively biased.
All the above assumes that by “retro funders” here you mean a set of carefully appointed Final Buyers. If instead we’re talking about an impact market where anyone can become a retro funder, and retro funders can resell their impact to arbitrary future retro funders, I think things would go worse in expectation (see the first three points in the section “Mitigating the risk is hard” in the OP).
It could be done a bit more smoothly by (1) accepting no new issues, (2) completing all running prize rounds, and (3) declaring the impact certificates not burned and allowing people some time to export their data. (I don’t think it would be credible for the marketplace to declare the certs burned since it doesn’t own them.)
My original idea from summer 2021 was to use blockchain technology simply for technical ease of implementation (I wouldn’t have had to write any code). That would’ve made the certs random tokens among millions of others on the blockchain. But then to set up a centralized, curated marketplace for them with a smart and EA curation team.
We’ve moved away from that idea. Our current market is fully web2 with no bit of blockchain anywhere. Safety was a core reason for the update. (But the ease-of-implementation reasons to prefer blockchain also didn’t apply so much anymore. We have a doc somewhere with all the pros and cons.)
For our favored auction mechanisms, it would be handy to be able to split transactions easily, so we have thought about (maybe, at some point) allowing users to connect a wallet to improve the user experience, but that would be only for sending and receiving payments. The certs would still be rows in a Postgres database in this hypothetical model. Sort of like how Rethink Priorities accepts crypto donations or a bit like a centralized crypto exchange (but that sounds a bit pompous).
But what do you think about the original idea? I don’t think it’s so different from a fully centralized solution where you allow people to export their data or at least not prevent them from copy-pasting their certs and ledgers to back them up.
My greatest worries about crypto stem less from the technology itself (which, for all I know, could be made safe) but from the general spirit in the community that decentralization, democratization, ungatedness, etc. are highly desirable values to strive for. I don’t want to have to fight against the dominant paradigms, so that doing it on my server was more convenient. But then again big players in the Ethereum space have implemented very much expert-run systems with no permissionless governance tokens and such. So I hope (and think) that there are groups that can be convinced that an impact market should be gated and curated by trusted experts only.
But even so, a solution that is crypto-based beyond making payments easier is something that I consider more in the context of joining existing efforts to make them safer rather than actions that would influence their existence.
That could make it easier for another team to create a new impact market that will seamlessly replace the impact market that is being shut down.
If a decentralized impact market gains a lot of traction, I don’t see how the certificates being “tokens among millions of others” helps. A particular curated gallery can end up being ignored by some/most market participants (and perhaps be outcompeted by another, less scrupulous curated gallery).
Okay, but to keep the two points separate:
Allowing people to make backups: You’d rather make it as hard as possible to make backups, e.g., by using anti-screenscraping tools and maybe hiding some information about the ledger in the first place so people can’t easily back it up.
Web3: Seems about as bad as any web2 solution that allows people to easily back up their data.
Is that about right?
I think that a decentralized impact market that can’t be controlled or shut down seems worse. Also, a Web3 platform will make it less effortful for someone to launch a competing platform (either with or without the certificates from the original platform).