Building a successful economy for collaborative cognitive work with high externalities


[Epistemic sta­tus: Quite con­fi­dent. Lots of this seems ob­vi­ous from first prin­ci­ples. Though it’s far from ex­haus­tive. Wary of car­ry­ing costs and the plan­ning fal­lacy, I pub­lish this post rough and in­com­plete, rather than not at all.]

Global mar­kets are cur­rently only (some­what) effi­cient in in­cen­tivis­ing prob­lem-solv­ing in ar­eas where the benefits can be in­ter­nal­ised, such as by earn­ing a profit from the product one has built. ^[Ig­nor­ing var­i­ous mar­ket failures such as long-time hori­zons, large co­or­di­na­tion prob­lems, high ini­tial costs, and more.]

Sev­eral peo­ple in the EA com­mu­nity have sug­gested that we should be able to use mon­e­tary mechanisms to gain similar benefits in ar­eas with large ex­ter­nal­ities. Suggested mechanisms in­clude prizes, boun­ties ([1], [2]), im­pact cer­tifi­cates and grants. (I will not fo­cus on grants in this post, as there’s a ton of con­tent about them on this site already.)

This spread­sheet sum­marises these efforts, in or­der to 1) al­low peo­ple look­ing to do free­lance cog­ni­tive work to find good op­por­tu­ni­ties, and 2) al­low peo­ple in­ter­ested in mak­ing prizes work to sur­vey the his­tory of ap­proaches and why they failed/​suc­ceeded.

The cur­rent post is an at­tempt to analyse what is needed to make prizes work—that is, to effec­tively change some peo­ple’s be­havi­our in a way which di­rectly op­ti­mises for im­prov­ing the long-term fu­ture (or some other goal we care about).

Ex­am­ples of such be­havi­our changes in­clude:

  • Some­one spend­ing a year liv­ing off of one’s sav­ings, learn­ing how to sum­marise com­ment threads, with the ex­pec­ta­tion that peo­ple will pay well for this abil­ity in the fol­low­ing years

  • A com­pe­tent liter­a­ture-re­viewer gath­er­ing 5 friends to teach them the skill, in or­der to scale their re­view­ing ca­pac­ity to earn more prize money

  • A col­lege stu­dent build­ing up a strong fore­cast­ing track-record and then be­ing paid enough to do fore­cast­ing for a few hours each week that they can pur­sue their own pro­jects in­stead of hav­ing to work full-time over the summer

  • A col­lege stu­dent drop­ping out to work full-time on an­swer­ing ques­tions on LessWrong, ex­pect­ing this to provide a sta­ble fund­ing stream for 2+ years

  • A pro­fes­sional with a sta­ble job and fam­ily and a hard time mak­ing changes to their life-situ­a­tion, tak­ing 2 hours/​week off from work to do skil­led cost-effec­tive­ness analy­ses, while be­ing fairly compensated

  • Some peo­ple start­ing a “Prize VC” or “Prize mar­ket maker”, which at­tempts to find po­ten­tial prize win­ners and con­nect them with prizes (or vice versa), while tak­ing a cut somehow

Etc. etc. (I ex­pect the above to be a small sub­set of the space of ex­cit­ing op­ti­mi­sa­tion that emerges when you man­age to get the in­cen­tives right.)

There are at least four main ways in which in­cen­tives af­fect be­havi­our:

  1. Con­scious mo­ti­va­tion: peo­ple de­liber­ately change their be­havi­our to benefit from the incentives

  2. Re­in­force­ment learn­ing: peo­ple un­con­sciously change their be­havi­our in line with the in­cen­tives, due to the pos­i­tive re­in­force­ment that gives them

  3. Selec­tion effects: peo­ple whose be­havi­our al­igns with the in­cen­tives will tend to be more suc­cess­ful and in­fluen­tial than peo­ple whose be­havi­our does not, ab­sent any ac­tual changes in a sin­gle per­son’s behaviour

  4. Mimet­ics: peo­ple or en­tire com­mu­ni­ties share tips, tricks, memes, norms and more to en­able oth­ers to benefit from the incentives

(I have writ­ten more about these here.)

Each have sep­a­rate im­pli­ca­tions for how to make prizes suc­cess­ful. I don’t think I have ex­hausted each sub-mechanism and look for­ward to col­lab­o­ra­tively mak­ing more progress in the com­ments.


1. Con­scious motivation

How can one en­sure that in­di­vi­d­u­als will con­sciously choose strate­gies to op­ti­mise for win­ning the prizes?

Clar­ity: It must be clear to peo­ple what they are op­ti­mis­ing for

This prop­erty fails at the tails. Some­times it’s bet­ter to have the prize giver be some­one more ra­tio­nal than the prize taker, mean­ing that it’s too hard for the lat­ter to good­hart on the de­sires of the former, in­stead leav­ing them to sim­ply do the best they can and treat­ing the prize sig­nal as an ob­jec­tive eval­u­a­tion of good work

Sta­bil­ity: Peo­ple must be able to change their be­havi­our in ex­pec­ta­tion

It’s not suffi­cient that I get a one-time prize for some­thing I did a year ago. I must ex­pect that there will be a sta­ble fund­ing stream in the fu­ture, such that I can con­di­tion my fu­ture plans on this (e.g. drop­ping out of col­lege, not satis­fic­ing on a job offer when the al­ter­na­tive is prize work, skill-build­ing for a skill in de­mand by prize-givers) -- while at the same time have prize-givers con­di­tion their ac­tions on my availa­bil­ity (e.g. set aside re­sources to man­age ap­pli­ca­tions, give feed­back, build sup­port­ing in­fras­truc­ture, and mak­ing sure funds are available)

One might split this into:

  1. Precom­mit­ments/​re­li­able ex­pec­ta­tions of fu­ture fund­ing

  2. Com­mon knowl­edge (be­tween both parts of the two-fac­tor mar­ket): for a com­mu­nity to build in­fras­truc­ture and make plans rest­ing upon prizes, the ex­is­tence and broad rules of the prizes should be com­mon knowl­edge. This en­ables would-be pro­duc­ers (e.g. po­ten­tial col­lege dropouts) and would-be con­sumers (e.g. EA orgs in­vest­ing time into turn­ing re­search ques­tions into an out­source­able for­mat) to move in-lock­step to the Nash equil­ibria where they suc­cess­fully trade re­sources via prizes.


2. Re­in­force­ment learn­ing (un­con­scious mo­ti­va­tion)

How can one en­sure that prizes un­con­sciously af­fect be­havi­our?

Quick and smooth payout

In­so­far as hu­mans are hy­per­bolic dis­coun­ters (re­gard­less of whether we want to or not), avoid­ing ir­ri­tat­ing pa­per­work and long de­lays might be helpful.

Clear credit assignment

In or­der for some­one to do more of what worked, they need to have a good sense of what as­pect of their work is be­ing re­warded (“Did I write good com­ments? Did I make good pre­dic­tions? Was the sum­mary good? Was the re­search topic novel and in­ter­est­ing?” etc.)

In­cen­tivise ex­plo­ra­tion (avoid an overly sparse re­ward sig­nal), oth­er­wise prize work­ers won’t learn the most effec­tive strategies

Effec­tively bal­ance in­trin­sic and ex­trin­sic motivation

Sec­tion 1.5 of Kraut and Res­nick’s “Build­ing suc­cess­ful on­line com­mu­ni­ties” has a use­ful chap­ter on this, in­clud­ing this di­a­gram from a 2001 meta-anal­y­sis about when ex­trin­sic re­wards harm (-) vs en­hance (+) in­trin­sic re­wards:

[di­a­gram—I can’t get the “add image” op­tion to work]


3. Selec­tion effects

How can one those with a com­par­a­tive ad­van­tage in do­ing cer­tain prize work will tend to be the ones do­ing that work?

I be­lieve that to a large ex­tent se­lec­tion effects will be pre­sent whether one wants them to or not, so the main ques­tion is rather whether there are effec­tive ways of choos­ing which se­lec­tion effects one wants to am­plify.

Here are some ex­am­ples of un­wanted se­lec­tion effects:

  • The peo­ple who win prizes are those most ea­ger to work for prizes, not the ones who had the high­est com­par­a­tive ad­van­tage in do­ing so

  • The in­cen­tive dy­nam­ics are set by the prize givers most gen­er­ous in giv­ing out prizes, rather than those with the best ideas of what should be funded (e.g. sup­pose foun­da­tion X are less care­ful in think­ing about the op­por­tu­nity costs of money than foun­da­tion Y, and so de­cide to award 5x as much prize money, which on the mar­gin dis­in­cen­tives the more valuable work preferred by Y)

  • The peo­ple who work for a par­tic­u­lar prize are those who thought it was in­ter­est­ing/​a good idea (e.g. award­ing a prize for re­sponses to “Does God ex­ist?” and only hav­ing the­olo­gians put in the work, al­most all of whom an­swer “Yes”)

Beyond list­ing these, I am un­cer­tain about what ac­tion-guid­ing ad­vice there is here.


4. Memetics

How can one en­sure that peo­ple suc­cess­fully com­mu­ni­cate things like: the ex­is­tence of prizes, good strate­gies and heuris­tics for prize work, promis­ing prize work­ers, norms and best prac­tices for prize de­sign, etc.?

As with se­lec­tion effects, there are likely sev­eral ad­verse effects that memet­ics have on a prize (for ex­am­ple, mes­sages will tend to lose nu­ance when be­ing shared be­tween many peo­ple, as there are more ways to mi­s­un­der­stand a claim than to un­der­stand it); and it might be hard to de­liber­ately in­ter­vene to pre­vent them.

“Me­meify­ing” prizes/​cre­at­ing a con­cep­tual handle

Hav­ing a sim­ple name for some­thing en­ables the differ­ence be­tween:

Without meme

Alice: “Hey, you seem pretty well off lately, but I haven’t no­ticed you get­ting a job or any­thing? What hap­pened?”

Bob: “Oh, it’s be­cause of [20 minute ex­pla­na­tion of the ideas be­hind hav­ing a mar­ket for im­pact via prizes]”

And

With meme

Alice: “Hey, you seem pretty well off lately, but I haven’t no­ticed you get­ting a job or any­thing? What hap­pened?”

Bob: “Yeah, I did some cog­ni­tive prize work!”

Us­ing this con­cep­tual han­dle, Alice can now quickly ask other friends “Do you know any­thing about how to be suc­cess­ful at ‘cog­ni­tive prize work’?”, she can eas­ily Google or search her favourite blogs for posts about “cog­ni­tive prize work”, and more.

For a real life ex­am­ple, Wei Dai writes:

For both of the AI al­ign­ment re­lated boun­ties, when a friend or ac­quain­tance asks me about my “work”, I can now talk about these prize that I re­cently won, which sounds a lot cooler than “oh, I par­ti­ci­pate on this on­line dis­cus­sion fo­rum”. :)

Pro­duc­ing sharable ma­te­rial

En­sur­ing there is a key refer­ence pub­lic write-up of memes one want to in­cen­tivise the spread of, e.g. what traits caused cer­tain prize win­ners to re­ceive their prizes (“X has the skill of be­ing both brief and ac­cu­rate”, “Y used a Guessti­mate model in a helpful way”) as well as on­go­ing pub­lic dis­cus­sion of the work (“Three things I did to im­prove as a prize worker”, “OpenPhil recom­men­da­tions for as­piring prize work­ers”).

A great ex­am­ple here is the April 2019 EA Long-term fu­ture fund write-up.