I’m working on Impact Markets – markets to trade nonexcludable goods.
If you’re also interested in less directly optimific things – such as climbing around and on top of boulders or amateurish musings on psychology – then you may enjoy some of the posts I don’t cross-post from my blog, Impartial Priorities.
Pronouns: Ideally they or she, but the others are fine too. I also go by Dawn now.
First of all, what we’ve summarized as “curation” so far could really be distinguished as follows:
Making access for issuers invite-only, maybe keeping the whole marketplace secret (in combination with #2) until we find someone who produces cool papers/articles and who we trust and then invite them.
Making access for investors/retro funders invite-only, maybe keeping the whole marketplace secret (in combination with #1) until we find an impact investor or a retro funder who we trust and then invite them.
Read every certificate either before or shortly after it is published. (In combination with exposé certificates in case we make a mistake.)
Let’s say #3 is a given. Do you think the marketplace would fulfill your safety requirements if only #1, only #2, or both were added to it?
Does your current plan not involve explaining to all the retro funders that that they should consider the ex-ante EV as an upper bound?
It involves explaining that. What we wrote was to argue that Attributed Impact is not as complicated as it may sound but rather quite intuitive.
How does the potential harm that other people can cause via Twitter etc. make launching a certain impact market be a better idea than it would otherwise be?
If you want to open a bazaar, one of your worries could be that people will use it to sell stolen goods. Currently these people sell the stolen goods online or on other bazaars, and the experience may be a bit clunky. By default these people will be happy to use your bazaar for their illegal trade because it makes life slightly easier for them. Slightly easier could mean that they get to sell a bit more quickly and create a bit more capacity for more stealing.
But if you enact some security measures to keep them out, you quickly reach the point where the bazaar is less attractive than the alternatives. At that point you already have no effect anymore on how much theft there is going on in the world in aggregate.
So the trick is to tune the security measures just right that they make the place less attractive than alternatives to the thieves and yet don’t impose prohibitively high costs on the legitimate sellers.
Do you intent to allow people to profit from outreach interventions that attract new retro funders? (i.e. by allowing people to sell certificates of such outreach interventions.)
My intent so far was to focus on text that is accessible online, e.g., articles, papers, some books. There may be other classes of things that are similarly strong candidates. Outreach seems like a bad fit to me. I’ve so far only considered it once when someone (maybe you) brought it up as something that’d be a bad fit for an impact market and I agreed.
I disagree. I think this risk can easily materialize if the description of the certificate is not very specific (and in particular if it’s about starting an organization, without listing specific interventions.)
Also very bad fit for an impact market as we envision it. To be a good fit, the object needs to have some cultural rights along the lines of ownership or copyright associated with it so market participants can agree on an owner. It needs to have a start and an end in time. It need so generate a verifiable artifact. Finally, it should not try super hard to try to fit something into that mold that doesn’t fit. There are a bunch of examples in my big post. So a paper, article, book, etc. (a particular version of it) is great. Something ongoing like starting an org is not a good fit. Something where you influence others and most of your impact is leveraging behavior change of others is really awkward because you can’t credibly assign an owner.
Okay, but to keep the two points separate:
Allowing people to make backups: You’d rather make it as hard as possible to make backups, e.g., by using anti-screenscraping tools and maybe hiding some information about the ledger in the first place so people can’t easily back it up.
Web3: Seems about as bad as any web2 solution that allows people to easily back up their data.
Is that about right?
Shutting down an impact market, if successful, functionally means burning all the certificates that are owned by the market participants, who may have already spent a lot of resources and time in the hope to profit from selling their certificates in the future.
It could be done a bit more smoothly by (1) accepting no new issues, (2) completing all running prize rounds, and (3) declaring the impact certificates not burned and allowing people some time to export their data. (I don’t think it would be credible for the marketplace to declare the certs burned since it doesn’t own them.)
Also, my understanding is that there was (and perhaps still is) an intention to launch a decentralized impact market (i.e. Web3 based), which can be impossible to shut down.
My original idea from summer 2021 was to use blockchain technology simply for technical ease of implementation (I wouldn’t have had to write any code). That would’ve made the certs random tokens among millions of others on the blockchain. But then to set up a centralized, curated marketplace for them with a smart and EA curation team.
We’ve moved away from that idea. Our current market is fully web2 with no bit of blockchain anywhere. Safety was a core reason for the update. (But the ease-of-implementation reasons to prefer blockchain also didn’t apply so much anymore. We have a doc somewhere with all the pros and cons.)
For our favored auction mechanisms, it would be handy to be able to split transactions easily, so we have thought about (maybe, at some point) allowing users to connect a wallet to improve the user experience, but that would be only for sending and receiving payments. The certs would still be rows in a Postgres database in this hypothetical model. Sort of like how Rethink Priorities accepts crypto donations or a bit like a centralized crypto exchange (but that sounds a bit pompous).
But what do you think about the original idea? I don’t think it’s so different from a fully centralized solution where you allow people to export their data or at least not prevent them from copy-pasting their certs and ledgers to back them up.
My greatest worries about crypto stem less from the technology itself (which, for all I know, could be made safe) but from the general spirit in the community that decentralization, democratization, ungatedness, etc. are highly desirable values to strive for. I don’t want to have to fight against the dominant paradigms, so that doing it on my server was more convenient. But then again big players in the Ethereum space have implemented very much expert-run systems with no permissionless governance tokens and such. So I hope (and think) that there are groups that can be convinced that an impact market should be gated and curated by trusted experts only.
But even so, a solution that is crypto-based beyond making payments easier is something that I consider more in the context of joining existing efforts to make them safer rather than actions that would influence their existence.
I love the insurance idea because compared to our previous ideas around shorting with hedge tokens that compound automatically to maintain a −1x leverage, collateral, etc. (see Toward Impact Markets), the insurance idea also has the potential of solving the incentive problems that we face around setting up our network of certificate auditors! (Strong upvotes to both of you!)
(The insurances would function a bit like the insurances in Robin Hanson’s idea for a tort law reform.)
Dawn’s (Denis’s) Intellectual Turing Test Red-Teaming Impact Markets
[Edit: Before you read this, note that I failed. See the comment below.]
I want to check how well I understand Ofer’s position against impact markets. The “Imagined Ofer” below is how I imagine Ofer to respond (minus language – I’m not trying to imitate his writing style though our styles seem similar to me). I would like to ask the real Ofer to correct me wherever I’m misunderstanding his true position.
I currently favor using the language of prize contests to explain impact markets unless I talk to someone intimately familiar with for-profit startups. People seem to understand it more easily that way.
My model of Ofer is informed by (at least) these posts/comment threads.
Dawn: I’m doing these prize contests now where I encourage people to help each other (monetarily and otherwise) to produce awesome work to reduce x-risks and finally I reward all participants in the best ones of the projects. I’m writing software to facilitate this. I will only reward them in proportion to the gains from moral trade that they’ve generated, and I’ll use my estimate of their ex ante EV as a ceiling for my overall evaluation of a project.
This has all sorts of benefits! It’s basically a wide-open regrantor program where the quasi-regrantors (the investors) absorb most of the risk. It scales grantmaking up and down – grantmakers have ~ 10x less work and can thus scale their operation up by 10x, and the investors can be anyone around the world, so they can draw on their existing networks for their investments, so they can consider many more much smaller investments or investments which require very niche knowledge or access. Many more ideas will get tried, and it’ll be easier for people to start projects even when they still lack personal contact to the right grantmakers.
Imagined Ofer: That seems very dangerous to me. What if someone else also offers a reward and also encourages people to help each other with the projects but does not apply your complicated ex ante EV ceiling? Someone may create a flashy but extremely risky project and attract a lot of investors for it.
Dawn: But they can do that already? All sorts of science prizes, all the other EA-related prizes, Bountied Rationality, new prizes they promote on Twitter, etc.
Imagined Ofer: Okay, but you’re building a software to make it easier, so presumably you’ll thereby increase the number of people who will offer such prizes and increase the number of people who will attract investments in advance because the user experience and networking with investors is smoother and because they’re encouraged to do so.
Dawn: That’s true. We should make our software relatively unattractive to such prize offerers and their audiences, for example by curating the projects on it such that only the ones that are deemed to be robustly positive in impact are displayed (something I proposed from the start, in Aug. 2021). I could put together a team of experts for this.
Imagined Ofer: That’s not enough. What if you or your panel of experts overlook that a project was actually ex ante net-negative in EV, for example because it has already matured and so happened to turn out good? You’d be biased in a predictably upward direction in your assessment of the ex ante EV. In fact, people could do a lot of risky projects and then only ever submit the ones that worked out fine.
Dawn: Well, we can try really hard… Pay bounties for spotting projects that were negative in ex ante EV but slipped through; set up a network of auditors; make it really easy and effortless to hold compounding short positions on projects that manage their −1x leverage automatically; recruit firm like Hindenburg Research (or individuals with similar missions) to short projects and publish exposés on them; require issuers to post collateral; set up a mechanisms whereby it becomes unlikely that there’ll be other prizes with any but a small market share (such as the “pot”); maybe even require preregistration of projects to avoid the tricks you mention; etc. (All the various fixes I propose in Toward Impact Markets.)
Imagined Ofer: Those are only unreliable patches for a big fundamental problem. None of them is going to be enough, not even in combination. They are slow and incomplete. Ex ante negative projects can slip through the cracks or remain undetected for long enough to cause harm in this world or a likely counterfactual world.
Dawn: Okay, so one slips through, attracts a lot of investment, gets big, maybe even manages to fool us into awarding it prize money. It or new projects in the same reference class have some positive per-year probability of being found out due to all the safety mechanisms. Eventually a short-seller or an exposé-bounty poster will spot them and make a lot of money for doing so. We will react and make it super-duper clear going forward that we will not reward projects in that reference class ever again. Anyone who wants to get investments will need to make the case that their project is not in that reference class.
Imagined Ofer: But by that time the harm is done, be it to a counterfactual world. Next time the harm will be done to the factual world. Besides, regardless of how safe you actually make the system, what’s important is that there can always be issuers and investors who believe (be it wrongly believe) that they can get their risky project retro-funded. You can’t prevent that no matter how safe you make the system.
Dawn: But that seems overly risk averse to me because prospective funders can also make mistakes, and current prizes – including prizes in EA – are nowhere near as safe. Once our system is safer than any other existing methods, the bad actors will prefer the existing methods.
Imagined Ofer: The existing methods are much safer. Prospective funding is as safe as it gets, and current prizes have a time window of months or so, so by the time the prizes are awarded, the projects that they are awarded to are still very young, so the prizes are awarded on the basis of something that is still very close to ex ante EV.
Dawn: But retroactive funders can decide when to award prizes. In fact, we have gone with a month in our experiment. But admittedly, in the end I imagine that cycles of a year or two are more realistic. That is still not that much more. (See this draft FAQ for some calculations. Retro funders will pay out prizes of up to 1000% in the success case, but outside the success case investors will lose all or most of their principal. They are hits-based investors, so their riskless benchmark profit is probably much higher than 5% per year. They’ll probably not want to stay in certificates for more than a few years even at 1000% return in the success case.)
Imagined Ofer: A lot more can happen in a year or two than in a month. EA, for example, looked very different in 2013 compared to 2015, but it looked about the same in January vs. February 2015. But more importantly, you write about tying the windfall clauses of AGI companies to retro funding with enormous budgets, budgets that surely offset even the 20 years that it may take to get to that point and the very low probability.
Dawn: The plan I wrote about has these windfalls reward projects that were previously already rewarded by our regular retro funders, no more.
Imagined Ofer: But what keeps a random, unaligned AGI company from just using the mechanism to reward anyone they like?
Dawn: True. Nothing. Let’s keep this idea private. I can unpublish my EA Forum post too, but maybe that’s the audience that should know about it if anyone should. As an additional safeguard against uncontrolled speculation, how about we require people to always select one or several actual present prize rounds when they submit a project?
Imagined Ofer: That might help, but people could just churn out small projects and select whatever prize happens to be offered at the time whereas in actuality they’re hoping that one of these prizes will eventually be mentioned in a windfall clause or their project will otherwise be retro funded through a windfall clause or some other future funder who ignore the setting.
Dawn: Okay, but consider how far down the rabbit hole we’ve gone now: We have a platform that is moderated; we have relatively short cycles for the prize contest (currently just one month); we explicitly offer prizes for exposés; we limit our prizes to stuff that is, by dint of its format, unlikely to be very harmful; we even started with EA Forum posts, a forum that has another highly qualified moderation team. Further, we want to institute more mechanisms – besides exposés – that make it easy to short certificates to encourage people to red-team them; mechanisms to retain control of the market norms even if many new retro funders enter; even stricter moderation; etc. We’re even considering requiring preregistration, mandatory selection of present prize rounds (even though it runs counter to how I feel impact markets should work), and very narrow targets set by retro funders (like my list of research questions in our present contest). Compare that to other EA prize contests. Meanwhile, the status quo is that anyone with some money and Twitter following can do a prize contest, and anyone can make a contract with a rich friend to secure a seed investment that they’ll repay if they win. All of our countless safeguards should make it vastly easier for unaligned retro funders and unaligned project founders to do anything other than use our platform. All that remains is that maybe we’re spreading the meme that you can seed-invest into potential prize winners, but that’s also something that is already happening around the world with countless science prizes. What more can we do!
Imagined Ofer: This is not an accusation – we’re all human – but money and sunk time-cost fallacy corrupt. For all I know this could be a motte-and-bailey type of situation: The moment a big crypto funder offers you a $1m grant, you might throw caution to the wind and write a wide-open ungated blockchain implementation of an impact market.
Dawn: I hope I’ve made clear in my 20,000+ words of writing on impact market safety that were unprompted by your comments (other than the first one in 2021) that my personal prioritization has long rested on robustness over mere positive EV. I’ve just quit my well-paid ETG job as software engineer in Switzerland to work on this. If I were in it for the money, I wouldn’t be. (More than what I need for my financial safety.) Our organization is also set up with a very general purview so that we can pivot easily. So if I should start work on a more open version of the currently fully moderated, centralized implementation, it’s because I’ve come to believe that it’s more robustly positive than I currently think it is. (Or it may well be possible to find a synthesis of permissionlessness and curation.) The only thing that can convince me otherwise are evidence and arguments.
Imagined Ofer: I think that most interventions that have a substantial chance to prevent an existential catastrophe also have a substantial chance to cause an existential catastrophe, such that it’s very hard to judge whether they are net-positive or net-negative (due to complex cluelessness dynamics that are caused by many known and unknown crucial considerations). So the typical EA Forum post with sufficient leverage over our future to make a difference at all is about equally likely to increase or to decrease x-risk.
Dawn: I find that to be an unusual opinion. CEA and others try to encourage people to post on the EA Forum rather than discourage them. That was also the point of the CEA-run EA Forum contest. Personally, I also find it unintuitive that that should be the case: For any given post, I try to think of pathways along which it could be beneficial and detrimental. Usually there are few detrimental pathways, and if there are any, there are strong social norms around malice and government institutions such as the police in the way of pursuing the paths. A few posts come to mind that are rare, unusual exceptions to this theme, but it’s been several years since I read one of those. Complex cluelessness also doesn’t seem to make a difference here because it applies equally to any prospective funding, to prizes after one month, and to prizes after one year. Do you think that writing on high-leverage topics such as x-risks should generally be discouraged rather than encouraged on the EA Forum?
Imagined Ofer: Even if you create even a very controlled impact market that is safer than the average EA prize contest you are still creating a culture and a meme regarding retroactive funding. You could inspire someone to post on Twitter “The current impact markets are too curated. I’m offering a $10m retro prize for dumping 500 tons of iron sulfate into the ocean to solve climate change.” If someone posted this now no one would take them seriously. If you create an impact market with tens of millions of dollars flowing through it and many market actors, it will become believable to some rouge players that this payout is likely real.
We’ve considered a wide range of mechanisms and ended up most optimistic about this one.
When it comes to prediction markets on funding decisions, I’ve thought about this in two contexts in the past:
During the ideation phase, I found that it was already being done (by Metaculus?) and not as helpful because it doesn’t provide seed funding.
In Toward Impact Markets, I describe the “pot” safety mechanism that, I surmised, could be implemented with a set of prediction markets. The implementation that I have in mind that uses prediction markets has important gaps, and I don’t think it’s the right time to set up the pot yet. But the basic idea was to have prediction markets whose payouts are tied to decisions of retro funders to buy a particular certificate. That action resolves the respective market. But the yes votes on the market can only be bought with shares in the respective cert or by people who also hold shares in the respective cert and in proportion to them. (In Toward Impact Markets I favor the product of the value they hold in either as determinant of the payout.)
But maybe you’re thinking of yet another setup: Investors buy yes votes on a prediction market (e.g. Polymarket, with real money) about whether a particular project will be funded. Funders watch those prediction markets and participants are encouraged to pitch their purchases to funders. Funders then resolve the markets with their actual grants and do minimal research, mostly trust the markets. Is that what you envisioned?
I see some weaknesses in that model. I feel like it’s rather a bit over 10x as good as the status quo vs. our model, which I think is over 100x as good. But it is an interesting mechanism that I’ll bear in mind as a fallback!
Going Forward
We will convene a regular working group to more proactively iterate and improve the mechanism design focused on risk mitigation. We intend for this group to function for the foreseeable future. Anyone is welcome to join this group via our Discord.
We will attempt to gain consultation from community figures that have expressed interest in impact markets (Paul Christiano, Robin Hanson, Scott Alexander, Eliezer Yudkowsky, Vitalik Buterin). This should move the needle towards more community consensus.
We will continue our current EA Forum contest. We will not run another contest in July.
We will do more outreach to other projects interested in this space (Gitcoin, Protocol Labs, Optimism, etc.) to make sure they are aware of these issues as well and we can come up with solutions together.
Do we think that impact markets are net-negative?
We – the Impact Markets team of Denis, Dony, and Matt – have been active EAs for almost a combined 20 years. In the past years we’ve individually gone through a prioritization process in which we’ve weighed importance, tractability, neglectedness, and personal fit for various projects that are close to the work of QURI, CLR, ACE, REG, CE, and others. (The examples are mostly taken from my, Denis’s, life because I’m drafting this.) We not only found that impact markets were net-positive but have become increasingly convinced (before we started working on them) that they are the most (positively!) impactful thing in expectation that we can do.
We have started our work for impact markets because we found that it was the best thing that we could do. We’ve more or less dedicated our lives to maximizing our altruistic impact – already a decade ago. We were not nerdsniped into it and adjusted our prioritization to fit.
We’re not launching impact certificates to make ourselves personally wealthy. We want to be able to pay the rent, but once we’re financially safe, that’s enough. Some of us have previously moved countries for earning to give.
Why do we think impact markets are so good?
Impact markets reduce the work of funders – if a (hits-based) funder hopes for 10% of their grantees to succeed, then they cut down on the funders’ work by 10x. The funders pay out correspondingly higher rewards which incentivize seed investors to pick up the slack. This pool of seed investors can be orders of magnitude larger than current grant evaluators and would be made up of individuals from different cultures, with different backgrounds, and different networks. They have access to funding opportunities that the funders would not have learned of, they can be confident in these opportunities because they come out of their existing networks, and they can make use of economies of scale if the projects they fund have similar needs. These opportunities can also be more and smaller than opportunities that it would’ve been cost-effective for a generalist funder to evaluate.
Thus impact markets solve the scaling problem of grantmaking. We envision that the result will be an even more vibrant and entrepreneurial EA space that makes maximal use of the available talent and attracts more talent as EA expands.
What do we think about the risks?
The risks are real – we’ve spent June 2021 to March 2022 almost exclusively thinking about the downsides, however remote, to position us well to prevent them. But abandoning the project of impact markets because of the downsides seems about as misguided to us as abandoning self-driving cars because of adversarial-example attacks on street signs.
A wide range of distribution mismatches can already happen due to the classic financial markets. Where an activity is not currently profitable, these don’t work, but there have been prize contests for otherwise nonprofitable outcomes for a long time. We see an impact market as a type of prize contest.
Other things being equal, simpler approaches are easier to communicate …
Attributed Impact may look complicated but we’ve just operationalized something that is intuitively obvious to most EAs – expectational consequentialism. (And moral trade and something broadly akin to UDT.) We may sometimes have to explain why it sets bad incentives to fund projects that were net-negative in ex ante expectation to start, but the more sophisticated the funder is, the less likely it is that we need to expound on this. There’s also probably a simple version of the definition that can be easily understood. Something like: “Your impact must be seen as morally good, positive-sum, and non-risky before the action takes place.”
If there is no way to prevent anyone from becoming a retro funder …
We already can’t prevent anyone from becoming a retro funder. Anyone with money and a sizable Twitter following can reward people for any contributions that they so happen to want to reward them for – be it AI safety papers or how-tos for growing viruses.
Even if we hone Attributed Impact to be perfectly smooth to communicate and improve it to the point where it is very hard to misapply it, that hypothetical person on Twitter can just ignore it. Chances are they’ll never hear of it in the first place.
The price of a certificate tracks the maximum amount of money that any future retro funder will be willing to pay for it …
The previous point applies here too. Anyone on Twitter with some money can already outbid others when it comes to rewarding actions.
An additional observation is that the threshold for people to seed-invest into projects seems to be high. We think that very few investors will put significant money into a project that is not clearly in line with what major retro funders already explicitly profess to want to retro-fund only because there may later be someone who does.
Suppose that a risky project that is ex-ante net-negative ends up being beneficial …
There are already long-running prize contests where the ex ante and the ex post evaluation of the expected impact can deviate. These don’t routinely seem to cause catastrophes. If they are research prizes outside EA, it’s also unlikely that the prize committees will always be sophisticated enough that contenders will trust them to evaluate their projects according to its ex ante impact. Even the misperception that a prize committee would reward a risky project is enough to create an incentive to start the project.
And yet we very much do not want our marketplace to be used for ex ante net-negative activities. We are eager to put safeguards in place above and beyond what any other prize contest in EA has done. As soon as any risks appear to emerge, we are ready to curate the marketplace with an iron fist, to limit the length of resell chains, to cap the value of certificates, to consume the impact we’re buying, and much more.
What are we actually doing?
We are not currently working on a decentralized impact marketplace. (Though various groups in the Ethereum space are, and there is sporadic interest in the EA community as well.)
This is our marketplace. It is a React app hosted on an Afterburst server with a Postgres database. We can pull the plug at any time.
We can hide or delete individual certificates. We’re ready to make certificates hidden by default until we approve them.
You can review the actual submissions that we’ve received to decide how risky the average actual submission is.
We would be happy to form a curation committee and include Ofer and Owen now or when the market grows past the toy EA Forum experiment we have launched so far.
This is our current prize round.
We have allowed submissions that are directly related to impact markets (and received some so that we don’t want to back down from our commitment now), but we’re ready to exclude them in future prize rounds.
We would never submit our own certificates to a prize contest that we are judging, but we’d also be open to not submitting any of our impact market–related work to any other prize contests if that’s what consensus comes to.
An important safety mechanism that we have already started implementing is to reward solutions to problems with impact markets. A general ban on using such rewards would remove this promising mechanism.
We don’t know how weak consensus should be operationalized. Since we’ve already launched the marketplace, it seems to us that we’ve violated this requirement before it was put into place. We would welcome a process by which we can obtain a weak consensus, however measured, before our next prize round.
Miscellaneous notes
Attributed Impact also addresses moral trade.
“A naive implementation of this idea would incentivize people to launch a safe project and later expand it to include high-risk high-reward interventions” – That would have to be a very naive implementation because if the actual project is different from the project certified in the certificate, then the certificate does not describe it. It’s a certificate for a different project that failed to happen.
Yeah, that sounds perfectly plausible to me.
“A bit confused” wasn’t meant to be any sort of rhetorical pretend understatement or something. I really just felt a slight surprise that caused me to check whether the forum rules contain something about ad hom, and found that they don’t. It may well be the right call on balance. I trust the forum team on that.
Maybe, but I find it important to maintain the sort of culture where one can be confidently wrong about something without fear that it’ll cause people to interpret all future arguments only in light of that mistake instead of taking them at face value and evaluating them for their own merit.
The sort of entrepreneurialness that I still feel is somewhat lacking in EA requires committing a lot of time to a speculative idea on the off-chance that it is correct. If it is not, the entrepreneur has wasted a lot of time and usually money. If additionally it has the social cost that they can’t try again because people will dismiss them because of that past failure, it makes it just so much less likely still that anyone will try in the first place.
Of course that’s not the status quo. I just really don’t want EA to move in that direction.
I’ve nevertheless downvoted this post because it seems like it’s making claims that are significantly too strong, based on a methodology that I strongly disendorse.
I agree, and I’m a bit confused that the top-level post does not violate forum rules in its current form. There is a version of the post – rephrased and reframed – that I think would be perfectly fine even though I would still disagree with it.
And I say that as someone who loved Paul’s response to Eliezer’s list!
Separately, my takeaway from Ben’s 80k interview has been that I think that Eliezer’s take on AI risk is much more truth-tracking than Ben’s. To improve my understanding, I would turn to Paul and ARC’s writings rather than Eliezer and MIRI’s, but Eliezer’s takes are still up there among the most plausible ones in my mind.
I suspect that the motivation for this post comes from a place that I would find epistemically untenable and that bears little semblance to the sophisticated disagreement between Eliezer and Paul. But I’m worried that a reader may come away with the impression that Ben and Paul fall into one camp and Eliezer into another on AI risk when really Paul agrees with Eliezer on many points when it comes to the importance and urgency of AI safety (see the list of agreements at the top of Paul’s post).
A bit of a tangent, but:
Sometimes funders try to play 5d chess with each other to avoid funging each other’s donations, and this results in the charity not getting enough funding.
That seems like it could be a defection in a moral trade, which is likely to burn gains of trade. Often you can just talk to the other funder and split 50:50 or use something awesome like the S-Process.
But I’ve been in the situation where I wanted to make a grant/donation (I was doing ETG), knew of the other donor, but couldn’t communicate with them because they were anonymous to me. Hence I resorted to a bit of proto-ECL: There are two obvious Schelling points, (1) both parties each fill half of the funding gap, or (2) both parties each put half of their pre-update budget into the funding gap. Point 2 is inferior because the other party knows, without even knowing me, that more likely than not my donation budget is much smaller than half the funding gap, and because the concept of the funding gap is subjective and unhelpful anyway. Point 1 should thus be the compromise point of which it is relatively obvious to both parties that is should be obvious to both parties. Hence I donated half my pre-update budget.
There’s probably a lot more game theory that can be done on refining this acausal moral trade strategy, but I think it’s pretty good already, probably better than the status quo without communication.
Maybe something along the lines of: Thinking in terms of individual geniuses, heroes, Leviathans, top charities implementing vertical health interventions, central charity evaluators, etc. might go well for a while but is a ticking time bomb because these powerful positions will attract newcomers with narcissistic traits who will usurp power of the whole system that the previous well-intentioned generation has built up.
The only remedy is to radically democratize any sort of power, make sure that the demos in question is as close as possible to everyone who is affected by the system, and build in structural and cultural safeguards against any later attempts of individuals to try to usurp absolute power over the systems.
But I think that’s better characterized as a libertarian critique, left or right. I can’t think of an authoritarian-left critique. I wouldn’t pass an authoritarian-left intellectual Turing test, but I have thought of myself as libertarian socialist at one point in my life.
Strong upvote. My personal intuitions are suffering focused, but I’m currently convinced that I ought to do whatever evidential cooperation in large worlds (ECL) implies. I don’t know exactly what that is, but I find it eminently plausible that it’ll imply that extinction and suffering are both really, really bad, and s-risks, especially according to some of the newer, more extreme definitions, even more so.
Before ECL, my thinking was basically: “I know of dozens of plausible models of ethics. They contradict each other in many ways. But none of them is in favor of suffering. In fact, a disapproval of many forms of suffering seems to be an unusually consistent theme in all of them, more consistent than any other theme that I can identify.[1] Methods to quantify tradeoffs between the models are imprecise (e.g., moral parliaments). Hence I should, for now, focus on alleviating the forms of suffering of which this is true.”
Reducing suffering – in all the many cases where doing so is unambiguously good across a wide range of ethical systems – still strikes me as at least as robust as reducing extinction risk.
Some variation on universalizability, broadly construed, may be a contender.
Here are four more things that I’m somewhat skeptical of and would like someone with more time on their hands and the right brain for the topic to see whether they hold water:
Evidential cooperation in large worlds is ridiculously underexplored considering that it might “solve ethics” as I like to epitomize it. AI safety is arguably more urgent, but maybe it can even inform that discipline in some ways. I have spent about a quarter of a year thinking about ECL, and have come away with the impression that I can almost ignore my own moral intuitions in favor of what little I think I can infer about the compromise utility function. More research is needed.
There is a tension between (1) the rather centralized approach that the EA community has traditionally taken and that is still popular, especially outside key organizations like CEA, and the pervasive failures of planned economies historically, and between (2) the much greater success of Hayakian approaches and the coordination that is necessary to avert catastrophic coordination failures that can end our civilization. My cofounders and I have started an EA org to experiment with market mechanisms for the provision of public and common goods, so we are quite desperate for more thinking of how we and EAs in general should resolve those tensions.
80k and others have amassed evidence that it’s best for hundreds or thousands of people to apply for each EA job, e.g., because the difference between the best and second best candidate are arguably large. I find this counterintuitive. Counterintuitive conclusions are interesting and the ones we’re likely to learn most from, but they are also more often than not wrong. In particular, my intuition is that, as a shallow heuristic, people will do more good if they focus on what is most neglected, all else equal. It seems suspicious that EA jobs should be an exception to this rule. I wonder whether it’s possible to make a case against along the lines of this argument, quantitatively trading off the expected difference between the top and the second best candidate against the risk of pushing someone (the second best candidate 0 to several hops removed) out of the EA community and into AI capabilities research (e.g., because they run out of financial runway), or simply by scrutinizing the studies that 80k’s research is based on.
I think some EAs are getting moral cooperation wrong. I’ve very often heard about instances of this but I can’t readily cite any. A fictional example is, “We can’t attend this workshop on inclusive workplace culture because it delays our work by one hour, which will cause us to lose out on converting 10^13 galaxies into hedonium because of the expansion of space.” This is, in my opinion, what it is like to get moral cooperation a bit less wrong. Obviously, all real examples will be less exaggerated, more subtle, and more defensible too.
Alternative models for distributing funding are probably better and are definitely under-explored in EA
I’m particularly optimistic about “impact markets” here, where you get:
countless mostly profit-oriented investors that use their various kinds of localized knowledge (language, domain expertise, connections, flat mates) to fund promising projects and compete with each other on making the best predictions, and
retroactive funders who reward investors and projects that, after some time (say, one or two years), look successful.
That model promises to greatly cut down on the work that funders have to do, and separates the concern “priorities research” or “What does success look like?” from the concern “due diligence” or “What sort of author or entrepreneur is likely to achieve such success and how to find them?”
The SFF is using a similar system internally.
Fittingly, we, Good Exchange, have received funding through the FTX Regrantor Program and are running our first MVP here.
Note that we started Good Exchange because we were already optimistic about this approach, and that it’s likely the most impactful thing that we can do with our time.
Some other solution concepts that come to mind:
Retrox – an experiment in “democratizing” retroactive funding, where the electorate is one of select experts.
Using delegated voting and PageRank to determine weights of experts in votes on funding decisions (Matt Goldenberg and Justin Shovelain have thought more about this)
For example, if funders hire a part-time teacher to develop a curriculum, then they attribute 100% of the development.
Do you mean that if someone is already hired and paid to do a job, that job should not, by default, be considered additionally rewarded through an impact allocation? Or at least if someone is paid at market rate?
Anyone can suggest a non-claimant, who then can choose to be notified.
That seems useful. I would like to err on the side of making sure that more rather than fewer people need to be explicitly asked for their consent for this. Otherwise it’s too easy for someone to miss the call. That’s sort of the opening pages of the Hitchhiker’s Guide to the Galaxy. xD
Maybe, people could select which ones work best for them or form teams considering all others’ expertise.
Oh yes, defs! :-)
But yes, for the total impact that an issuer makes even the non-purchased certificates’ counterfactuals should be used.
Agreed. Preregistration also solves a lot of problems with our version of p-hacking (i-hacking ^.^).
Hmm… yes, but maybe actually some organizers would not take the risk, so some projects would get unfunded or delayed and many more projects could be run but eventually not purchased (which could reduce some persons’ ‘risk or learning’ budgets or be counterfactually beneficial).
I think we’d need more assumptions for this to happen. When it comes to business ventures, some are better funded from investments and some from loans. But if we imagine a world without investments, with only loans, and then offer investments, it’s not straightforward to me that that would harm the businesses that are better suited for loans. Similarly, I’d think (at first approximation at least) that the projects that are not suited for retro funding will still apply for and receive prospective funding at almost the same rate that it is available today. The bottleneck is currently more the investment of funding than the availability of funding.
So, (1)*(2)+(3)*[4]
Right, those are multiplicative too… That makes them a bit not-robust (volatile?). Would it also be valid to conceive of each of those as a multiplier on the current counterfactual impact, e.g., 10x increase in seed funding, 10x improvement in the allocation of seed funding, 10x through economies of scale, etc.? Here’s a sample Guesstimate. But it feel to me like we’re much too likely to err in upward direction this way even if it’s just an estimate of the success (non–black swan) case. Sort of like the multi-stage fallacy give too low estimates if you multiply many probabilities, so I feel like multiplying all these multipliers probably also ignores lots of bottlenecks.
But that said, almost all the factors are multiplicative here except for two components of the allocation – better allocation thanks to knowledge of more languages and cultures and thanks to being embedded in more communities. (I suppose a person is in about as many communities regardless which language/s they speak.) The language component may seem unusually low, which is because so much of the world speaks English and because the US are such a big part of the world economy.
I’m assuming that more entrepreneurial people will join the fray who are not sufficiently altruistic to already be motivated to start EA charities but who will do a good job if they can get at least somewhat rich from it.
Finally, I’m assuming that retro funders reinvest all their savings into the retro funding and priorities research, and that priorities research still has < 10x room for improvement, which seems modest to me (given evidential cooperation in large worlds for example).
I’d be interested what you think of this version – any other factors that need to be considered, or anything that should be additive instead of multiplicative? But we can discuss all that in our call if you like. No need to reply here.
1. surveys make sense but why anonymous—you could get greater sincerity in a (‘safe’ group) conversation, perhaps.
Perhaps, but I’m worried that they may be incentivized to fib about it because it makes their impact seem more valuable and because it makes it more likely that we’ll continue to program (from which they want to benefit again), since we might discontinue it if we don’t think it’s sufficiently impactful.
Hm, not really, maybe like 126 for an RCT if you observe a greater than 50% difference
Thanks! Still a lot. I’ve updated away from this being an important factor to measure… I’d rather find a way to compare the success rates of investors vs. current prospective funders. If none of them are consistently better, they may lose interest in the system.
Ask funders: Estimate the impact of our framework compared to its non-existence. Explain your reasoning. Then, summarize quotes and see what people critique.
Yeah, that sounds good in both cases. It’s probably not easy to talk to them but it would be valuable.
But why would you weigh by the extent of profit-orientation?
If we’re just reallocating EA money, we’re not adding more. A big source of the impact stems from attracting for-profit money.
Looking forward to our call! Feel free to just respond verbally then. No need to type it all out. :-D
Thank you for engaging with this question as well!
So, to terribly oversimplify, your idea is to form an independent committee of sorts that has some sort of voting procedure (and maybe a veto period?) and all sorts of useful features like that and that votes or otherwise agrees on how impact in a project should be allocated?
That seems feasible for really big impacts. Dony imagines Stanislav Petrov’s impact certificate for saving the world from WW3. It seems sensible to form a committee that is trusted by everyone who was around Stanislav Petrov at the time and might’ve also had an influence and that then allocates the impact among all these “participants.”
In many other cases it’s going to be both easier and harder: A archetypal case may be that of a workshop. There is no strong cultural norm around how impact should be allocated among organizers, funders, and maybe some pro-bono instructors, but still the set of people who need to be considered is fairly limited. Typically, for example, the parents of an instructor won’t make a claim. In that sense it’s easier, but it’s also harder in that a single workshop (well, the median workshop) won’t be significant enough that we can assemble a trusted committee to make the decision.
Also, typically, the reason for the autonomous allocation is that the person who wants to know their shared doesn’t want to impose on the time of the others to settle the matter. So if otherwise they would’ve had a 30-minute call with the other n-1 participants, and scheduling the call takes 5 minutes for each of them, then you get a time cost of 35*(n-1) minutes for everyone else. So an alternative method needs to be cheaper than 35*(n-1) minutes to be worth it. (I’ll assume for simplicity that the person’s own time doesn’t matter because they’re probably ready to invest > 10x the amount of time, but of course there are hypothetical methods that are so complex that even that person would not deem them to be worth their time.)
Hence why I was thinking that there may be some norms that we could set that will establish for some subset of projects (say, all since the norm was established) that impact can be assumed to be allocated according to some simple rule unless otherwise specified… But I haven’t thought of anything.
I’d be delighted to know if you have any more ideas that could help us here! :-D
Oh, thank you for engaging my our question! We were delighted to see and read your post! (Have you joined our Discord already or would you be interested in having a call with us? Your input would be great to have, for example when we’re brainstorming something!)
When I’m considering how I would apply your methodology in practice, I mostly run into the hurdle that it’s hard to assess counterfactuals. In our competition we’ve come up with the trick that we suggest particular topics that we think it would be very impactful for someone to work on, a topic so specific that we can be fairly sure that the person wouldn’t otherwise have picked it.
That allows us to assess, for example, that you would most likely have investigated something else in the time that you spent to write this post. But (also per your model) that could still result in a negative impact if what you would’ve otherwise investigated would’ve been more important to investigate. But that depends on lots of open questions from priorities research and on your personal fit for the investigations, so it’s something that it’s hard for us take into account.
There is also the problem that issuers may be biased about their actual vs. counterfactual impact because they don’t want to believe that they’ve made a mistake and they may be afraid that their certificate price will be lower.
One simplification that may be appropriate in some cases is to assume that the same n projects are (1) funded prospectively or (2) founded retroactively. That way, we can ignore the counterfactual of the funders’ spending. Since retro funding will hopefully be used for the sorts of projects where it is appropriate, it’s probably by and large a valid approximation that the same projects get funded, just differently (and hopefully faster and smarter).
But a factor that makes it more complicated again is, I think, that it’ll be important to make explicit the number of investors. In the prospective case, the investors are just the founders who invest time and maybe a bit of money. But in the retrospective case, you get profit-seeking investors who would otherwise invest into pure for-profit ventures. The degree to which we’re able to attract these is an important part of the success of impact markets.
So ideally, the procedure should be one where we can measure something that is a good proxy for (1) additional risk capital that flows into the space, (2) effort that would not otherwise have been expended on (at all) impactful things, and (3) time saved for retro funders.
Our current metric can capture 2 to a very modest extent, but our market doesn’t yet support 1 and and we’re the only EA retro funders at the moment, and we wouldn’t otherwise have done prospective funding.
Some random ideas that come to mind (and sorry for rambling! ^.^’):
Anonymous surveys asking people what they would’ve done without impact markets – where they would’ve invested, what they would’ve investigated, etc. (Sadly, we can’t ask them what they actually did, or it might deanonymize them. But we can probably still average over the whole group.)
Recruit a group of people, ask them what they’re planning to do, then offer them the promise of retro funding, and then observe what they actually end up doing.
Tell people that they’ll randomly get either the small prospective or the large, conditional retrospective funding if they preregister their projects. See in which group there is more follow-through.
These all have weaknesses – the first relies on people’s memory and honesty, the second is costly and we can’t really prevent people from self-selecting into the group if they prefer retro funding, and the last one is similar. I’m also worried that small methodological mistakes can ruin the analysis for us. And a power analysis will probably show that we’d need to recruit hundreds of participants to be able to learn anything from the results.
So yeah, I’d love some sort of quick and dirty metric that is still informative and not biased to an unknowable extent.
What do you think? Do any solutions come to mind?
Oh, in mature markets retro funders will try to estimate at what “age” an investment into impact might break even with some counterfactual financial market investment of an investor. I have some sample calculations here. Retro funders can then announce that they’ll buy-or-not-buy at a point in time that’ll allow the investors to still make a big profit. But randomly, they can announce much later times. The investors who invest in the short term but less so in the longer term can be assumed to be mostly profit-oriented. If we can identify investors on this market, we can sum up the invested amounts weighed by the degree to which they are profit-oriented. But that’s all not viable yet at all…
Well, that’s all my thoughts. xD I’d be very curious what you think!
Very cool formalization! What do you think of the following way of applying it:
Hiring managers that are looking for similar candidates meet (e.g., online) to hash out a single standardized application process for all the similar open positions.
When the applications are in and they have narrowed them down to the set candidates any of them find at all interesting, they start the process.
Is there a strong reason why they would need to agree on relative impact scores for their organizations? I imagine they’ll find it hard to agree on those. Maybe they can just assume a vector of [1, 1, 1, …] for all the impact scores?
They add more pseudo-organizations to the mix, which could have an impact of −100 in spaces like AGI (representing the average non-safety AGI lab) or 0 in most other spaces. (They don’t have control over the choice between other orgs, so I don’t think it makes sense to add several different values, but there need to be as many pseudo-organizations as there are candidates, in case all organizations decide not to hire.)
They generate the candidate-organization matrix but also include columns for “no candidate at org n,” because not hiring is also an option. (“No candidate” can “work” for multible or all organizations in parallel, so this gets a bit complicated.)
I think in most cases they can assume that everyone will be full time, which could simplify the process in those cases.
Then they pick the row that maximizes the impact.
Step 5 seems like the one that’ll require a lot more work in practice. There could be an independent team of forecasters that does the estimation or all hiring managers estimate this for all orgs, or the hiring manages of org A makes the case for/against each candidate at org A, and then all other hiring manages estimate the impact.
There could be the risk that if one org was wrong and their eventual pick soon quits or doesn’t do a good job, that then the remaining allocation is not optimal anymore because that person would’ve been a good fit at another org that now doesn’t have capacity to hire them anymore.