I’m working on impact markets – markets to trade nonexcludable goods. (My profile.)
I have a conversation menu and a Calendly for you to pick from!
If you’re also interested in less directly optimific things – such as climbing around and on top of boulders or amateurish musings on psychology – then you may enjoy some of the posts I don’t cross-post from my blog, Impartial Priorities.
Pronouns: Ideally she or they. I also still go by Denis and Telofy in various venues.
Proportional prizes for prescient philanthropists
Effective Altruism, Economic Growth, Empowering Excetional People
A low-tech alternative to my proposal for impact markets is to offer regular, reliable prizes for early supporters of exceptionally impactful charities. These can be founders, advisors, or donors. The prizes would not only go to the top supporters but proportionally to almost anyone who can prove that they’ve contributed (or where the charity has proof of the contribution), capped only at a level where the prize money is close to the cost of the administrative overhead.
Donors may be rewarded in proportion to the aggregate size of their donations, advisors may be rewarded in proportion to their time investment valued at market rates, founders may be rewarded in proportion to the sum of both.
If these prizes are awarded reliably, maybe by several entities, they may have some of the same benefits as impact markets. Smart and altruistic donors, advisors, and charity serial entrepreneurs can accumulate more capital that they can use to support their next equally prescient project.
Please let us know if there is anything we at GoodX can do to help. Our main project is to build an impact marketplace, but ultimately we want to get resources to where they are needed (as efficiently as possible).
(E.g., it wouldn’t be my first time running an emergency fundraiser to bail out customers of a failed venture.)
Scandals don’t just happen in the vacuum. You need to create the right conditions for them. So I suggest:
We spread concern about the riskiness of all altruistic action so that conscientious people (who are often not sufficiently scandal-prone) self-select out of powerful positions and open them up to people with more scandal potential.
We encourage more scathing ad-hom attacks on leadership so that those who take any criticism to heart self-select out of leadership roles.
We make these positions more attractive to scandal-prone people by abandoning cost-effectiveness analyses and instead base strategy and grantmaking on vibes and relationships.
We further improve the cushiness of these positions by centralizing power and funding around them to thwart criticism and prevent Hayekian diversity and experimentation.
We build stronger relationships with powerful, unscrupulous people and companies by, e.g., helping them with their hiring.
We emphasize in-person networking and move the most valuable networks to some of the most expensive spots in the world. That way access to the network comes with even greater dependency on centralized funding, making it easier to control.
[Meta: I’m not claiming anyone is doing these things on purpose! It would be nice, though, if more people were trying to counter these risk factors for scandals and generally bad epistemics.]
Sorry if this is a bit of a tangent but it seems possible to me to frame a lot of the ideas from the paper as wholly uncontroversial contributions to priorities research. In fact I remember a number of the ideas being raised in the spirit of contributions by various researchers over the years, for which they expected appreciation and kudos rather than penalty.
(By “un-/controversial” I mean socially un-/controversial, not intellectually. By socially controversial I mean the sort of thing that will lead some people to escalate from the level of a truth-seeking discussion to the level of interpersonal conflict.)
It think it’s more a matter of temperament than conviction that I prefer the contribution framing to a respectful critique. (By “respectful” I mean respecting feelings, dignity, and individuality of the addressees, not authority/status. Such a respectful critique can be perfectly irreverent.) Both probably have various pros and cons in different contexts.
But one big advantage of the contribution framing seems to be that it makes the process of writing, review, and publishing a lot less stressful because it avoids antagonizing people – even though they ideally shouldn’t feel antagonized either way.
Another is evident in this comment section: The discussion is a wild mix of threads about community dynamics and actual object-level responses to the paper. Similarly I had trouble deciding whether to upvote or to strong-upvote the post: The paper touches on many topics, so naturally my thoughts about its object-level merits are all over the place. But I feel very strongly that, as community norms go, such respectful critiques are invaluable and ought to be strongly encouraged. Then again Simon Grimm’s suggestion to make a separate link-post for the object-level discussion would also address that.
But the critique framing seems to have the key advantage of a signal boost. I think Logan and Duncan observed that social posts generated a lot more engagement on Less Wrong than epistemic posts (which I haven’t tried to confirm), and Scott Alexander’s “Toxoplasma of Rage,” though a much more extreme case, seems to feed on similar dynamics. So maybe there are particular merits to the critique framing, at least when the content is so important that the community gains from the signal boost; it’s probably one of those powerful tools that ought to be handled with great care.
Are there heuristics for when a critique framing is warranted or even better than a contribution framing? Was it the correct choice to go for a critique framing in this case?
I can host the adjacent astroturfing meetup! I’ll reserve one seat at a restaurant.
Going Forward
We will convene a regular working group to more proactively iterate and improve the mechanism design focused on risk mitigation. We intend for this group to function for the foreseeable future. Anyone is welcome to join this group via our Discord.
We will attempt to gain consultation from community figures that have expressed interest in impact markets (Paul Christiano, Robin Hanson, Scott Alexander, Eliezer Yudkowsky, Vitalik Buterin). This should move the needle towards more community consensus.
We will continue our current EA Forum contest. We will not run another contest in July.
We will do more outreach to other projects interested in this space (Gitcoin, Protocol Labs, Optimism, etc.) to make sure they are aware of these issues as well and we can come up with solutions together.
Do we think that impact markets are net-negative?
We – the Impact Markets team of Denis, Dony, and Matt – have been active EAs for almost a combined 20 years. In the past years we’ve individually gone through a prioritization process in which we’ve weighed importance, tractability, neglectedness, and personal fit for various projects that are close to the work of QURI, CLR, ACE, REG, CE, and others. (The examples are mostly taken from my, Denis’s, life because I’m drafting this.) We not only found that impact markets were net-positive but have become increasingly convinced (before we started working on them) that they are the most (positively!) impactful thing in expectation that we can do.
We have started our work for impact markets because we found that it was the best thing that we could do. We’ve more or less dedicated our lives to maximizing our altruistic impact – already a decade ago. We were not nerdsniped into it and adjusted our prioritization to fit.
We’re not launching impact certificates to make ourselves personally wealthy. We want to be able to pay the rent, but once we’re financially safe, that’s enough. Some of us have previously moved countries for earning to give.
Why do we think impact markets are so good?
Impact markets reduce the work of funders – if a (hits-based) funder hopes for 10% of their grantees to succeed, then they cut down on the funders’ work by 10x. The funders pay out correspondingly higher rewards which incentivize seed investors to pick up the slack. This pool of seed investors can be orders of magnitude larger than current grant evaluators and would be made up of individuals from different cultures, with different backgrounds, and different networks. They have access to funding opportunities that the funders would not have learned of, they can be confident in these opportunities because they come out of their existing networks, and they can make use of economies of scale if the projects they fund have similar needs. These opportunities can also be more and smaller than opportunities that it would’ve been cost-effective for a generalist funder to evaluate.
Thus impact markets solve the scaling problem of grantmaking. We envision that the result will be an even more vibrant and entrepreneurial EA space that makes maximal use of the available talent and attracts more talent as EA expands.
What do we think about the risks?
The risks are real – we’ve spent June 2021 to March 2022 almost exclusively thinking about the downsides, however remote, to position us well to prevent them. But abandoning the project of impact markets because of the downsides seems about as misguided to us as abandoning self-driving cars because of adversarial-example attacks on street signs.
A wide range of distribution mismatches can already happen due to the classic financial markets. Where an activity is not currently profitable, these don’t work, but there have been prize contests for otherwise nonprofitable outcomes for a long time. We see an impact market as a type of prize contest.
Other things being equal, simpler approaches are easier to communicate …
Attributed Impact may look complicated but we’ve just operationalized something that is intuitively obvious to most EAs – expectational consequentialism. (And moral trade and something broadly akin to UDT.) We may sometimes have to explain why it sets bad incentives to fund projects that were net-negative in ex ante expectation to start, but the more sophisticated the funder is, the less likely it is that we need to expound on this. There’s also probably a simple version of the definition that can be easily understood. Something like: “Your impact must be seen as morally good, positive-sum, and non-risky before the action takes place.”
If there is no way to prevent anyone from becoming a retro funder …
We already can’t prevent anyone from becoming a retro funder. Anyone with money and a sizable Twitter following can reward people for any contributions that they so happen to want to reward them for – be it AI safety papers or how-tos for growing viruses.
Even if we hone Attributed Impact to be perfectly smooth to communicate and improve it to the point where it is very hard to misapply it, that hypothetical person on Twitter can just ignore it. Chances are they’ll never hear of it in the first place.
The price of a certificate tracks the maximum amount of money that any future retro funder will be willing to pay for it …
The previous point applies here too. Anyone on Twitter with some money can already outbid others when it comes to rewarding actions.
An additional observation is that the threshold for people to seed-invest into projects seems to be high. We think that very few investors will put significant money into a project that is not clearly in line with what major retro funders already explicitly profess to want to retro-fund only because there may later be someone who does.
Suppose that a risky project that is ex-ante net-negative ends up being beneficial …
There are already long-running prize contests where the ex ante and the ex post evaluation of the expected impact can deviate. These don’t routinely seem to cause catastrophes. If they are research prizes outside EA, it’s also unlikely that the prize committees will always be sophisticated enough that contenders will trust them to evaluate their projects according to its ex ante impact. Even the misperception that a prize committee would reward a risky project is enough to create an incentive to start the project.
And yet we very much do not want our marketplace to be used for ex ante net-negative activities. We are eager to put safeguards in place above and beyond what any other prize contest in EA has done. As soon as any risks appear to emerge, we are ready to curate the marketplace with an iron fist, to limit the length of resell chains, to cap the value of certificates, to consume the impact we’re buying, and much more.
What are we actually doing?
We are not currently working on a decentralized impact marketplace. (Though various groups in the Ethereum space are, and there is sporadic interest in the EA community as well.)
This is our marketplace. It is a React app hosted on an Afterburst server with a Postgres database. We can pull the plug at any time.
We can hide or delete individual certificates. We’re ready to make certificates hidden by default until we approve them.
You can review the actual submissions that we’ve received to decide how risky the average actual submission is.
We would be happy to form a curation committee and include Ofer and Owen now or when the market grows past the toy EA Forum experiment we have launched so far.
This is our current prize round.
We have allowed submissions that are directly related to impact markets (and received some so that we don’t want to back down from our commitment now), but we’re ready to exclude them in future prize rounds.
We would never submit our own certificates to a prize contest that we are judging, but we’d also be open to not submitting any of our impact market–related work to any other prize contests if that’s what consensus comes to.
An important safety mechanism that we have already started implementing is to reward solutions to problems with impact markets. A general ban on using such rewards would remove this promising mechanism.
We don’t know how weak consensus should be operationalized. Since we’ve already launched the marketplace, it seems to us that we’ve violated this requirement before it was put into place. We would welcome a process by which we can obtain a weak consensus, however measured, before our next prize round.
Miscellaneous notes
Attributed Impact also addresses moral trade.
“A naive implementation of this idea would incentivize people to launch a safe project and later expand it to include high-risk high-reward interventions” – That would have to be a very naive implementation because if the actual project is different from the project certified in the certificate, then the certificate does not describe it. It’s a certificate for a different project that failed to happen.
CLR (and previously FRI) regularly came out at the top of my prioritized list of charities to donate to when I was in ETG, and I wish I still had a sizable donation budget to invest today!
Thanks for this incredibly comprehensive overview!
I’d like to use this chance to position our GoodX project “Impact Markets” in this context. (Thanks for linking to our MVP post!) I’m trying to write this in a way that is also understandable to people who haven’t thought about impact markets much, so sorry in advance if some of it is unnecessarily verbose.
Our main selling point is simply that altruists are more likely to have some sort of investor in their network (because there are countless ones ) than to have a funder in their network who we would trust (because there are like half a dozen or so – Open Phil, the Future Fund, the EA Funds, the SFF, et al.). Hence, to a funder, the altruist is going to succeed with the baseline probability of some suitably chosen reference class – say 10%. But to the investor, who has more knowledge of their altruistic friend, they are more likely to succeed – say 20%. There are also reasons to think that the investor can make the altruist succeed at a lower cost than the funder. The result is that the funder and the investor can find a reward size that saves the funder money and time (they don’t need to negotiate this in each case) and yet earns the investor a profit.
An added benefit is that funders can hire a wider range of staff. These staff still need to be excellent at priorities research or at applying priorities research, but they don’t need to be excellent at startup picking anymore.
What Is The Basic Format Of The Market?
We want to implement these in a sensible order, from simple to complex. We haven’t considered “block” impact certificates outside an early experiment, so they’re all considered to be fractional in our case.
If “block” certs are legally easier, that could be a reason to rethink that! Our vision for where impact markets might go is that they integrate with the classic financial market: Think tanks (incorporated as, say, public benefit corporations) issue impact certificates, generate a profit from their sale, and control their stock price with the profits, e.g., by paying dividends or doing buybacks. All the investment happens on the level of the classic stock of the think tanks. The impact certificates are just extremely simple products, consumables, that are so simple that it is dead obvious to the SEC that they are not securities. Our (Owen’s) metaphor is (consumable) bottles of wine.
This seems to us like it would make impact markets hard to use for independent researchers (the main audience we have in mind for issuers) because they would have to first start a corporation for their blogs. But maybe there are ways around that. (Maybe accredited investors can invest into the stock of unincorporated associations.)
Note that this consideration (the previous two paragraphs) is much more fundamental than the question whether certs should be fractional or not. It just relates to the question if non-fractional certs are legally easier. But if the investment happens on the level of classic stock anyway, then impact certs could also be issued retroactively, which makes them legally unproblematic according to what we’ve heard from the lawyers we talked to.
Note also that one of our cofounders, Dony, has thought a lot about dominant assurance contracts in the past. I remember him thinking about how they might combine with impact markets. That’s an interesting angle (dominant or not), but not one we’ve prioritized yet.
What Is Being Sold?
We want to keep it simple and so only focus on the core idea of making the “startup picking” part of funding delegatable. For that, the finer points of the metaphysical significance of impact purchases don’t matter. It just matters that there is a trusted retro funder that provides a prize pool and makes good decisions on what projects to reward. We want to leave the philosophy to philosophers or culture to figure out over time. (I can see that these are interesting questions, but I don’t think that that’s where the big counterfactual impact is.)
This sounds similar to what you write under “Credit for funding the project.” But I at least have moved away from the term “moral credit” for this because while some people interpret it simply as “I can get money for holding this contract,” others again interpret metaphysical significance into it (and in various conflicting ways too).
The funders that we envision serving with our solution are funders like the EA Funds, Open Phil, the Future Fund, et al., who are only interested in the counterfactual impact and not in being seen as virtuous by others or by God or some other variation.
We call what our certs certify a “right to retroactive funding,” so a right that is completely limited to only the entitlement to some share of the prize pool to the extent to which it is awarded to the project. (We want to eventually make it possible for people to attach other rights to their certificates, such as “bragging rights” if they so choose, but it’ll be up to them to define how that should work. This agnosticism about the rights that people attach to certs is just an implication of our using the Hypercertificate standard. The author of the Hypercertificates standard, David Dalrymple, and Protocol Labs are working on a solid, standards-based implementation of impact markets that we want to be compatible with.)
Another way to think about it is that if currently someone enters an article that they’ve written into a prize contest, then they can win a prize. If they’ve collaborated with five others on the article, they can agree to split the prize among themselves if they win it. They can (if they all so choose) make that agreement without ever clarifying the metaphysical significance of their agreement.
How Should The Market Handle Projects With Variable Funding Needs?
We’re going for the second solution from your list here because we put a strong emphasis on the precision and verifiability of impact certificates. This has various reasons:
we want impact certificates to be like products that charities can sell and that investors can preorder,
we want to make it easy for buyers to verify that the impact is not doubly sold under different framings, and
we want to make it easy for buyers to verify that the set of issuers is complete according to strong cultural norms.
So “cure malaria in Senegal” is not a definition of an impact cert that would work on our market. It doesn’t have enough details about the concrete actions that’ll be taken, it doesn’t make clear that the issuers really owns the cert to the extent that they claim, etc.
Instead something like this could work: “We want to collaborate with Concern Universal on a distribution of 5 million long-lasting insecticide-treated bednets in the Matam region of Senegal between Jan. 1, 2023 and Apr. 1, 2023. We currently own 60% of the certificate; Concern Universal owns 40%. The census, the distribution, and the follow-up surveys are conducted by paid staff of Concern Universal who have forfeited any claim to the certificate in their work contracts. No one else, to our knowledge, can make a legitimate claim to the certificate.”
This clarifies ownership, timeframe, location, scope, etc. The organization would sell scores or hundreds of such certificates that can all follow a standard template.
Unrelatedly, I don’t think this impact certificate would work: LLIN distributions are an intervention that is highly likely to work. Funders may assign a 90% chance to their success; a highly involved investor might assign a 95% chance to the success. That gives the investor a very, very minimal edge over the funder. The funder would likely consider it unnecessary overhead to use impact markets for this, or, if they do end up using them, they’ll pay very little over the counterfactual prospective funding for the very slightly reduced risk. That’ll not be enough for the investor to beat some counterfactual investment like a standard ETF. That’s in contrast to hits-based giving where funders often assign a 10% chance of success, which leave a lot of (multiplicative) room for highly involved investors to be more optimistic.
Should Founders Be Getting Rich Off Impact Certificates?
We haven’t thought about this much and would love to be convinced one way or the other.
But if there are projects that everyone recognizes as good, then an impact marketplace will shift the surplus from final oracular funders (who are forced to pay prices more commensurate with their benefits) to someone else. If we let founders get rich, it shifts the surplus to founders. If we don’t let founders get rich, it shifts the surplus to fast investors who may not have added any value—that is, to the first people who snapped up the obviously-underpriced certificates. See Section 7 for more on this problem.
I don’t have a good solution to this latter issue other than either not using impact markets, or allowing founders to get rich. I sketch a hack-ish solution in Section 11C.
My take is basically the first, i.e. that impact markets won’t be used for projects where it doesn’t make sense for both sides (investors/founders and funders) to use them. These will just continue to use prospective funding. This just seems like a brute fact of the preferences of the market actors and not like something that is under the control of the marketplace at all.
Some more elaboration in case it’s not clear what I mean (but it probably is): Funders will pay as little as they can get away with paying. They may start by offering prizes that are just slightly more than what they would’ve paid prospectively for a single project (with, say, a 10% chance of success). Maybe no investor will be interested. Then they’ll raise their bid slightly again and again until investors start to be interested and the first few projects actually get funded from investor money. But this is still strictly less than what the funders would’ve otherwise paid in total for prospective funding. Sure, some of that money goes to investors who can buy yachts from it, but the counterfactual is not that that money would’ve gone to beneficiaries but that it (and more) would’ve gone to failed projects.
Or in made-up numbers: In the prospective-funding world, a funder pays out a total of $1m to 100 projects ($10k each), 10 of which succeed. In the retrospective world, the funder pays out $800k to 10 projects ($80k each), all of which have succeeded. These $80k are maybe split $40k to the founders and $40k to the investors, so that one could say that $40k or more have gone to waste in some sense. But that’s compared to a counterfactual that was never attainable in the first place. More realistically, the funder achieved the same impact while saving $200k that can now go to more projects to help beneficiaries.
Unrelatedly, established charities (as opposed to independent researchers or startup charities) often have track records that make impact markets redundant (see above) or have 18+ months of runway that retro funding would funge with, so that, one way or another, funders are probably not very interested in using impact markets for them.
If impact markets were “a success”, they might have to more than dectuple the amount they pay this person, with no change to their output.
Realistically, I think, a funder wouldn’t just dectuple the investment but would try to pay as little as possible that is just enough to incentivize the creation of this researcher. By construction, the six-figure salary (say, $200k) is not enough to create an additional engineer, but maybe $300k is enough, in which case the funder would bid $300k, not $10m. Of course it could just so happen that the only price that incentivizes the creation of another researcher is no less than $10m, but then the counterfactual is not another researcher for $200k but no researcher at all.
Again the funder is well-advised to keep their true valuation of the impact (if they can come up with such a thing) a secret and just bid as low as they can get away with. (Just like when you buy a second-hand laptop. You may privately decide that you’ll bid up to $1k for it and yet start by bidding $100 because it might just be enough to convince the seller to part with the laptop.)
Admittedly, the current AI safety researchers may get greedy and try to use whatever leverage they have to get more money, but they could do that already.
How Do We Kickstart The Existence Of An Impact Market?
The Committed Pot of Money is the main or most fundamental solution we have in mind. The lower-down solutions are more powerful and they may be the desirable end states but they also come with risks that we’re wary of.
If I put myself into the shoes of an early investor on the Committed Pot of Money impact market, I don’t model myself as judging my chances to break even as a random draw. Rather, by buying in early I can buy more at a lower price, so that I later have an outsized share in the expected returns compared to later investors. But I also have an opinion on what the retro funder will value, and that model has a much stronger effect on my subjective expected returns than the amount that has been invested in total in all projects.
(If I buy SOL tokens at $5 because I think Solana is a great technology with a high success chance, I don’t start to think that Solana is going to be much less successful only because there is heavy investment into Solana and many other technologies. I’ll Aumann-update a bit on all the investments into Avalanche, Cardano, etc., but not overwhelmingly much.)
Should The Market Use Cryptocurrency?
My thoughts: Crypto-literate reviewers say that crypto exchanges take fiat and handle transactions within themselves in a non-crypto way—but then the crypto exists in case someone wants to bring it to a different exchange. This seems like a best of both worlds scenario.
Agreed. Plus we can just start with simple non-crypto solutions and keep all other options open in case there are eventually strong arguments to transition to something more like FTX (centralized) or Serum (decentralized).
How Should The Market Price IPOs?
We have an auction process that A Donor developed for us (see A Donor’s comment) that, we think, solves B neatly. We haven’t implemented it yet. (It is based on the view that it is desirable to reward people for aiding a fast price discovery.)
How Should The Oracular Funder Buy Successful Projects?
We’ve considered both – buying shares immediately if possible or creating the funding floor of limit orders. For now we’re going with the first solution simply for simplicity, but I definitely see the appeal of the second.
What Should The Final Oracular Funder’s Decision Process Be?
My thoughts: Guess this one is up to the final oracular funders.
Agreed. And by implication I think it’ll be an auction where the funders bid as little as they can get away with paying and only bid more if they don’t observe enough projects getting started or invested in.
Who Are We Expecting To Have As Investors?
My thoughts: Institutional types probably won’t bite for a little while, so we’ll need to be prepared for ordinary people.
Agreed. Plus, institutional types are again fewer and have to optimize more for scale, which causes them to suffer from some of the same problems that funders currently suffer from. Investors should rather be the sort of people who are sufficiently many and sufficiently unconstrained by scale that every independent researcher or startup entrepreneur can so happen to know one or two of them quite well.
It’s apparently now possible to become an accredited investor by passing an exam, which lowers the bar. But impact certs as we conceive of them can be consumed (or “dedicated” or “burned”) by altruists, which plausibly turns them into consumables. Consumables, such as wine, can also be preordered and resold. I find it plausible that impact certs are so much more like wine and so much less like stock that they are not securities in the eye of the law or the SEC. Plus, all of this is easier in many countries other than the US and UK.
Conclusion: What Kind Of Impact Market Should We Have?
I think the above has elucidated the design that we’re interested in. It’s similar to the “Maximum Capitalism” design but without the metaphysical bits about “losing credit” and such. The founders have done their work, and if they (maybe unwisely) elected to sell 100% of their impact cert, they’ve still done their work and can get their recognition and kudos for it. (For comparison, if I start a successful startup and make an exit, I can still claim to have started a successful startup and get the reputational benefits.)
Issues Around Unregistered Securities
We have talked with lawyers about this and people with more experience with quasi impact markets. The devil is in the details. There are various ways to avoid the impression that impact certs are securities – as mentioned above, we conceive of them like bottles of wine that can be preordered, stored, resold, and consumed – but we don’t yet have full legal clarity on the issue.
Issues Around Tax-Deductability
In our view, investing in impact certs should not be tax-deductible. It’s done with a profit motive. But consuming the certificate is something that is altruistic, and it would make sense to see to it that that is tax deductible. We envision one potential system where you have the option to do all your cert trades through a nonprofit. The nonprofit just executes all your deals on your behalf. So long as you haven’t consumed your certs, the nonprofit does nothing else. But once you consume a cert, that nonprofit (not the issuer of the cert) writes the donation receipt for you.
Issues Around Oracular Funders’ Nonprofit Status
There may be larger prize contests (such as the XPrize) organized by nonprofits. One could research those to look for legal precedents.
Issues Around Governance And Principal-Agent Problems
Agreed.
Issues Around Middlemen Holding Money
We’re currently planning to let people transact directly with each other to avoid these questions for the time being to be lean and all.
Accidentally Encouraging High-Risk Negative-In-Expectation Projects
This is getting at some of the big crucial problems for us that we’ve been mulling over for a year or so. Here is a list of all the current risks and mitigation mechanisms that we see.
Attributed Impact (mentioned in the linked document ) also addresses moral trade, the lack of which I consider an even greater problem as well as a superset that contains the problem of the distribution mismatch.
Using Impact Certificates For Evil
The above document also addresses this.
Incentivizing Reward-Hacking
That is an interesting failure mode to look out for.
People Could Lose A Lot Of Money
I think we should promote a norm that people shouldn’t invest any money in impact certificates that they don’t want to lose. Otherwise, I’m not sure this is any worse than eg crypto, which already lets small investors lose all their money quickly.
Agreed. A marketplace could also set a default limit on investments and only raise it for investors who have somehow proven that they are financial savvy. Only allowing accredited investors to invest would have that effect, though it may be a bit too exclusive.
Saram is an organization in the area that I trust. You can for example get in touch with Nicolai Sprekels (nicolai.sprekels@saram-nk.org), who is an expert in the area.
I’m much less up-to-date here, and I don’t know this particular organization (afaik, there are several that help escapees), but the ones I looked into back in the day weren’t able to help people escape from NK but instead helped them flee from China to South Korea. The northern border is the one that most escapees manage/d to cross, but China tries to catch them and returns them to NK if they can’t escape from China before that happens.
If escapees leave families behind, the families often claim that the person has died. So they don’t suffer consequences at first, but if the person is then returned from China, it becomes clear that they lied, and the escapee and three generations of the family are sent to concentration camps or killed. So helping them to escape from China saves the whole family from that fate, which increases the cost-effectiveness estimate.
So if this is still true and true of this organization (Nicolai may know more), this flips the sign of points 2 and (separately) changes some other points subtly, because the organization doesn’t affect who or how many people escape in the first place.
Point 4 is probably a very small effect because of how few people escape compared to the general population, and how little power they have to organize or even move around in the country. The people who escape are probably mostly from the north because few people who are not from the north can travel there. Conversely, to influence something in the country politically, you’d probably have to be in the top caste and in Pyongyang.
Point 5 is potentially important. They may execute people in particularly public, gruesome ways in response to rescue attempts to deter foreign organizations.
A “Red Team” to rigorously explore possible futures and advocate against interventions that threaten to backfire
Research That Can Help Us Improve, Effective Altruism, Epistemic Institutions, Values and Reflective Processes
Motivation. There are a lot of proposals here. There are additional proposals on the Future Fund website. There are additional proposals also on various lists I have collected. Many EA charities are already implementing ambitious interventions. But really we’re quite clueless about what the future will bring.
This week alone I’ve discussed with friends and acquaintances three decisions in completely different contexts that might make the difference between paradise and hell for all sentient life, and not just in the abstract in the way that cluelessness forces us to assign some probability to almost any outcome but in the sense were we could point to concrete mechanisms along which the failure might occur. Yet we had to decide. I imagine that people in more influential positions than mine have to make similar decisions on almost a daily basis and on hardly any more information.
As a result, the robustness of an intervention has been the key criterion for prioritization for me for the past six years now. It’s something like the number and breadth of scenarios that trusted, impartial people have thought through along which the intervention may have any effect, especially unintended effects, divided by the number of bad failure modes that they’ve found and haven’t been able to mitigate. I mostly don’t bother to think about probabilities for this exercise, though that would be even better.
Tools. Organizations should continue to do their own red-teaming in-house, but that is probably always going to be less rigorous and systematic than what a dedicated red-teaming team could do.
I hear that Policy Horizons Canada (h/t Jaques Thibodeau), RAND (h/t Christian Tarsney), and others have experience in eliciting such scenarios systematically. The goal is often to have a policy response ready for every eventuality. The focus may need to be different for effective alturism and especially any interventions to do with existential and suffering risks: We can probe our candidate interventions using scenario planning or ensample simulations and discard them if they seem too risky.
I imagine that you could also set up a system akin to a prediction market platform but with stronger incentives to create new markets and to conditionally chain markets, including the UI/UX to make voting on conditional predictions frictionless, without much mental math.
Jaques Thibodeau is also thinking about using machine learning tools to aid with the elicitation of scenarios.
Organization. The final “Red Team” organization needs to have strong social buy-in from EA-branded organizations to offset the social awkwardness of being a perpetual critic. It’ll probably need to recruit the sort of people who thrive in the role of a devil’s advocate. But it’ll also need to hold itself to its own high standards and disband if it finds that its own mission is at risk of backfiring badly.
More options:
Save the child but then let your clothes dry, thereby killing thousands of zooplankton, rotifers, and nematodes.
Never go outside without your Uber Walking Buddy, so that arguably they would’ve saved the child if you hadn’t.
Don’t worry about it and save the child because the optimal (infinite EV) action would’ve been to give your expensive shoes to a normie in return for being your counterparty in a St. Petersburg Game.
Offset every saved child with a donation to a marine habitat conservation charity.
Save the child but tell it that there is no God thereby defecting in an acausal moral trade with Calvinists centuries ago.
Save all children but on the shore cunningly arrange them in a spatiotemporal pattern that actually reduces overall value according to the value-density and hyperreal remedies to infinite ethics.
Save children who eat meat.
Corollary: Complex cluelessness means the EV of all actions is undefined/unknowable, so you’re free to save all the children you want.
Don’t worry about it because whatever you think is maximally good probably isn’t because of the Optimizer’s Curse.
Counterfactuals are logically inconsistent in a deterministic world, so you might as well save the child and compare that action against a counterfactual where you prevented them from falling into the pond in the first place.
Save the child but don’t tell them about cryonics.
Buy the expensive shoes to ruin at an unnecessarily high price.
Drench your clothes in a mild poison every morning so that by jumping into the water you contaminate it and expose the child to a low chance of death by poisoning.
Save the child while keeping your eyes open or not plugging your ears – you thereby force the simulation to simulate the stressful drowning process in detail, which is not optimal.
Use a quantum random number generator for all sorts of decisions every day to decorrelate yourself from your selves on other Everett branches: Now you’re free to save the child because on many branches you’re not even passing by the pond.
Assume Omega predicted your actions: If it predicted that you would save the child, it put the child in the pond; if it predicted you wouldn’t save the child, it let the child play by the shore in peace. By being the sort of person who would save the child, you’ve already caused the child a lot of unnecessary stress, so saving them now doesn’t risk being optimal anymore.