Studying behaviour and interactions of boundedly rational agents, AI alignment and complex systems.
Research fellow at Future of Humanity Institute, Oxford. Other projects: European Summer Program on Rationality. Human-aligned AI Summer School. Epistea Lab.
Obviously the toy model is wrong in describing reality: it’s one end of the possible spectrum, where you have complete randomness. On the other you have another toy model: results in a field neatly ordered by cognitive difficulty, and the best person at a time picks all the available fruit. My actual claims roughly are
reality is somewhere in between
it is field-dependent
even in fields more toward the random end, there actually would be differences like different speeds of travel among prospectors
It is quite unclear to me where on this scale the relevant fields are.
I believe your conclusion, that the power law distribution is all due to the properties of the peoples cognitive processes, and no to the randomness of the field, is not supported by the scientometric data for many research fields.
Thanks for a good preemptive answer :) Yes if you are good enough in identifying the “golden” cognitive processes. While it is clear you would be better than random chance, it is very unclear to me how good you would be. *
I think its worth digging into an example in detail: if you look a at early Einstein, you actually see someone with an unusually developed geometric thinking and the very lucky heuristic of interpreting what the equations say as the actual reality. Famously special relativity transformations were written first by Poincare. “All” what needed to be done was to take it seriously. General relativity is a different story, but at that point Einstein was already famous and possibly one of the few brave enough to attack the problem.
Continuing with the same example, I would be extremely doubtful if Einstein would be picked by selection process similar to what CEA or 80k hours will be probably running, before he become famous. 2nd grade patent clerk? Unimpressive. Well connected? No. Unusual geometric imagination? I’m not aware of any LessWrong sequence which would lead to picking this as that important :) Lucky heuristic? Pure gold, in hindsight.
(*) At the end you can take this as an optimization problem depending how good your superior-cognitive-process selection ability is. Let’s have a practical example: You have 1000 applicants. If your selection ability is great enough, you should take 20 for individual support. But maybe its just good, and than you may get better expected utility if you are able to reach 100 potentially great people in workshops. Maybe you are much better than chance, but not really good… than, maybe you should create online course taking in 400 participants.
For scientific publishing, I looked into the latest available paper[1] and apparently the data are best fitted by a model where the impact of scientific papers is predicted by Q.p, where p is “intrinsic value” of the project and Q is a parameter capturing the cognitive ability of the researcher. Notably, Q is independent of the total number of papers written by the scientist, and Q and p are also independent. Translating into the language of digging for gold, the prospectors differ in their speed and ability to extract gold from the deposits (Q). The gold in the deposits actually is randomly distributed. To extract exceptional value, you have to have both high Q and be very lucky. What is encouraging in selecting the talent is the Q seems relatively stable in the career and can be usefully estimated after ~20 publications. I would guess you can predict even with less data, but the correct “formula” would be trying to disentangle interestingness of the problems the person is working on from the interestingness of the results.
(As a side note, I was wrong in guessing this is strongly field-dependent, as the model seems stable across several disciplines, time periods, and many other parameters.)
Interesting heuristics about people :)
I agree the problem is somewhat different in areas not that established/institutionalized where you don’t have clear dimensions of competition, or the well measurable dimensions are not that well aligned with what is important. Loooks like another understudied area.
[1] Quantifying the evolution of individual scientific impact, Sinatra et.al. Science, http://www.sciencesuccess.org/uploads/1/5/5/4/15543620/science_quantifying_aaf5239_sinatra.pdf
After thinking about it for a while I’m still a bit puzzled by the rated-100 or rated-1000 plan changes, and their expressed value in donor dollars. What exactly is here the counterfactual? As I read it, it seems based just on comparing “the person not changing their career path”. However, with some of the examples of the most valued changes, leading to people landing in EA organizations it seems the counterfactual state “of the world” would be “someone else doing a similar work in a central EA organization”. As AFAIK recruitment process for positions at places like central EA organizations is competitive, why don’t count as the real impact just the marginal improvement of the 80k hours influenced candidate over the next best candidate?
Another question is how do you estimate your uncertainty in valuing something rate-n?
Sure, first 80k thought your counterfactual impact is “often negligible” due to replaceability, then they changed position toward replaceability being “very uncertain” in general. I don’t think you can just remove it from the model completely.
I also don’t think in the particular case of central EA organizations hiring the uncertainty is as big as in general / I’m uncertain about this, but my vague impression was there is a usually a selection of good candidates to choose from when they are hiring.
As many others have expressed, the concern with systemic changes is you are often dealing with complex poorly understood systems.
Let’s take the example of EU’s Common Agricultural Policy. It is most likely evil toward world’s poor, but it’s not clear for example whether it works toward or against EU unity. It is plausible it is somehow important for EU unity, either because being a form of fiscal transfer or a way how to corrupt important voter block… So we should include possible political consequences in the utility calculation, and the problem becomes really tricky.
On the other hand I agree systemic interventions are worth considering, and e.g. change in drug policy seems to be an excellent candidate, as the action has been tested, we understand most of the consequecnes.
Thanks for writing it.
Here are my reasons for the belief wild animal/small minds/… suffering agenda is based mostly on errors and uncertainties. Some of the uncertainties should warrant research effort, but I do not believe the current state of knowledge justifies prioritization ofany kind of advocacy or value spreading.
1] The endeavour seems to be based on extrapolating intuitive models far outside the scope for which we have data. The whole suffering calculus is based on extrapolating the concept of suffering far away from the domain for which we have data from human experience.
2] Big part of it seems arbitrary. When expanding the moral circle toward small computational processes and simple systems, why not expand it toward large computational processes and complex systems? E.g. we can think about the DNA based evolution as about large computational/optimization process—suddenly “wild animal suffering” has a purpose and traditional environmnet and biodiversity protection efforts make sense.
(Similarly we could argue much “human utility” is in the larger system structure above individual humans)
3] We do not know how to measure and aggregate utility of mind states. Like, really don’t know. E.g. it sems to me completely plausible the utility of 10 people reaching some highly joyful mindstates is the dominanat contribution over all human and animal minds.
4] Part of the reasoning usually seems contradictory. If the human cognitive processes are in the priviledged position of creating meaning in this universe … well, then they are in the priviledged postion, and there _is_ a categorical difference between humans and other minds. If they are not in the priviledged positions, how it comes humans should impose their ideas about meaning on other agents?
5] MCE efforts directed toward AI researchers with the intent of influencing values of some powerful AI may increase x-risk. E.g. if the AI is not “speciist” and gives the same weight to satysfing prefrences of all humans and all chicken, the chicken would outnumber humans.
I think the part which is lacking in your understanding is part of MIRIs intellectual DNA.
In it you can find lot of Eliezer Yudkowsky’s thought—I would recommend reading his latest book ” Inadequate equilibria” where he explains some of the reasons why the normal reserach environment may be inadequte in some respects.
MIRI is explicitly founded on the premise to free some people from the “publish or perish” pressure, which is severely limiting what people in normal academia work on and care about. If you give enough probability to this beeing approach worth taking, it does make sense to base decisions on funding MIRI on different criteria.
What I would do when evaluating potentially high-impact, high uncertainty “moonshot type” research project would be to ask some trusted highly knowledgeable researcher to assess the thing. I would not evaluate publication output, but whether the effort looks sensible, people working on it are good, and some progress is made (even if in discovering things which do not work)
I just published a short history of creating effective altruism movement in the Czech Republic http://effective-altruism.com/ea/1ls/introducing_czech_association_for_effective/ and I think it is highly relevant to this discussion
Compared to Ben’s conclusions I would use it as a data-point showing
it can be done
it may not be worth delaying
there are intermediate forms of communication in between “mass outreach” and “person-to-person outreach”
you should consider more complex model of communication than just (personal vs. mass media): specifically, a viable model in new country could be something like “very short message in mass media, few articles translated in national language to lower the bareer and point in the right direction, much larger amount transmitted via conferences & similar
Putting too much weight into “person to person” interaction runs into the problem you are less likely to find the right persons (consider how such connections may be created)
Btw it seems to me the way e.g. 80k hours and CEA works are inadeqate in creating the required personal connections in new countries, so it’s questionable if it makes sense to focus on it
(I completely agree China is extremely difficult, but I don’t think China should be considered a typical example—considering mentality it’s possibly one of the most remote countries from from Eurpoean POV)
Hi Ben,
1) I understand your concerns.
On the other hand I’m not sure if you take into account the difficuties
e.g. going to EAG could require something like “heroic effort”. If my academic job was my only source of income, going to EAGxOxford would have meant spending more than my monthly disposable income on it (even if EAGxOxford organizers were great in offering me big reduction in fees)
I’m not sure if you are modelling correctly barriers in building connections to the core of the community(*). Here is my guess at some problems people from different countries ol cultures may run into when trying to make connections, e.g. by applying for internships, help, conferences, etc.
1) People unconsciously take language sophistication as a proxy for inteligence. By not being proficient in English you loose points.
2) People are evaluated on proxies like “attending prestigeous university”. Universities outside of US or UK are generally discounted as regional
3) People are often in fact evaluated based on social network distance; this leads to “rich gets richer” dynamics
4) People are evaluated based on previous “EA achievements” which are easier to achieve in places where EA is allready present.
(*) you may object e.g. Ales Flidr is a good counterexample, but Harvard alumni are typically relatively “delocalized”, in demand everywhere, and may percieve greater value in working in the core than spreading ideas from core to emerging local groups. (Prestige and other incentives also point that way)
one risk I see in your article is it may influence the people who would be best at mitigating risks of “wrong local EA movements” being founded to not work on it at all.
I dont think “the barriers” should be zero, as such barriers in a way help the selection of motivated people. Just in my impression they may be higher than they appear from inside. Asking people to first build conections, while building such connections is not systematically supported, may make the barriers higher than optimal.
Btw yes the core of the group can read English materials. Also it could do research in machine learning, found a startup, work in quantitive finance, in part get elected to the parliament, move out of the country, and more. What I want to point at, if you imagine members of a group of people working on “translation” of EA into new culture you would like, they are likely paying huge opportunity costs in doing so. It may be hard to keep them at some state of waiting and building connections.
In our case, counterfactually it seems plausible “waiting even more” could have also led to the group not working, or worse organization beeing created, as the core people would loose motivation / pursue other opportunities.
2) In counting long-term impact and the lock-in effect you should consider the chance movements in new languages and cultures develop in some respects better versions of effective altruism, and beneficial mutations can than spread, even back to the core. More countries may mean more experiments running and faster evolution of better versions of EA. It’s unclear to me how these terms (lock in, more optimization power) add up but both should be counted. One possible resolution may be to prioritize founding movements in smaller countries where you create experience, but the damage is limited.
To your questions
i] with one exception, reasonably well taken from the viewpoint of strategic communication (communicating easily communicable concepts, eg impact and effectivity). I don’t think the damage is great, and part of misconceptions is unavoidable givent the name “effective altruism”.
ii] it has a distribution… IMO the understanding at the moment is highly correlated with the enagagement in the group. Which may be more important than criteria like “average understanding” or “public understanding”
iii] yes, it’s complicated, depends on some misalignments, and I dont want to disscuss it publicly.
Thanks! I think the algorithm may be explained something like
1] take the most up-to-date reasoning and conclusions from central places … like prioritization outputs of CEA, FHI, CSER, GiveWell, OpenPhil, etc.
2] try to think what are the local differences, and which of them would most influence the outcomes.
3] think in terms of comparative advantages and disadvantages (e.g., if you are in Geneva, you have access to all that world organizations and people working in them … so you have advantage in policy)
4] think in terms of the movement optimizing as a unified organization, like “if there was an EA Inc. what units it would create here” (sometimes its possible to learn from actual multinationals)
Possible caveat… doing this may be much less work than creating the original models, but may need good understanding of the models, and good grasp of concepts like parameter sensitivity.
We’re starting a prioritization projects attempting to quantitatively model this (factor in local considerations). It will also probably try to model the important question “stay or move”
I agree it may seem to point toward some “person-affecting views” which many EAs consider to be wrong.
Possibly the aim was to describe the motivation is altruistic?
The disadvantage of ‘do as much good as possible’ may be it would associtate EA with utilitarianism even more than it is.
I think about EA as a movement trying to answer a question “how to change the world for better most effectively with limited resources” in a rational way, and act on the answer. Which seems to me a tiny bit more open than ‘do as much good as possible’ as it requires just some sort of comparison on world-sates, while ‘as much good as possible’ seems to depend on more complex structure.
Good point, I think the “effective altruism as an academic field” metaphor is useful—but possibly having too much influence.
Effective altruism should be not only about finding answers to the central question but also about acting on them, at some point. The action part is IMO what makes it different from academia, may be brings it closer to “technology” than “science” and may benefit from a different structure or tighter coordination. Or maybe not, that’s part of the question.
To me the basic variance seems to be in the level of centralization (hierarchy with a centre vs. loose cluster), hierarchicality, type of connections between organizations (formal vs. informal).
Possibly closer example than academia or corporations may be “environmental movement” … where you have research, action, and policy components, and I would guess there may be things to learn from.
So if I understand correctly your answer to “How much should geography correlate with function?” is “strong correlation is good”. Do you see some limits, or the stronger, the better?
From my observations, the biggest problem in current EA funding ecosystem is structural bottlenecks.
It seems difficult to get relatively modest funding for a promising project if you are not well connected in the network / early stage projects in general (?).
Why?
While OpenPhil has an abundance of resources, they are at the moment staff limited, unlikely to grant to subjects they don’t know directly, and unlikely to grant to small projects ($10k)
EA funds seem to also be staff limited and also not capable of giving small grants.
In theory, EA Grants should fill this gap, but the program also seems staff limited (I’m familiar with one grant application where since Nov 2017 the term when the grant program will be open is pushed into future at a rate 1 month per month)
Part of the early-stage projects grant support problem is it generally means investing into people. Investing in people needs either trust or lot of resources to evaluate the people (which is in some aspects more difficult than evaluating projects which are up and running)
Trust in our setting usually comes via network links in the social network, which is quite limiting resource.
So my conclusion is, the efficient allocation is structurally limited by 1] lack of staff in grant-making organizations 2] insufficient size of the “trust network” allowing investment in promising projects based on theor founders
Individual EAs have good opportunities to get more impact from their donations than by donating to EA funds if they’re overcoming the structural bottlenecks by their funding. That may mean
a] donating to projects which are under the radar of OpenPhil and EA Funds
b] using their personal knowledge of people to support early stage efforts
The articles has quite a lot of arguments, so to pick just one manageable topic: I think it’s worth to dispute, on empirical grounds, the claim that cryptocurrencies lead to a redistribution of wealth which is somehow less centralized.
According to https://howmuch.net/articles/bitcoin-wealth-distribution about 4% of addresses own above 96% of wealt.
In comparison, in relatively strong European states the top 10% own just above 37% of income. (http://wir2018.wid.world/executive-summary.html)
I would be also worried. Homophily is of the best predictors of links in social networks, and factors like being member of the same social group, having similar education, opinions, etc. are known to bias selection processes again toward selecting similar people. This risks having the core of the movement be more self encapsulated that it is, which is a shift in bad direction.
Also I would be worried with 80k hours shifting also more toward individual coaching, there is now a bit overemphasis on “individual” approach and too little on “creating systems”.
Also it seems lot of this would benefit from knowledge from the fields of “science of success”, general scientometry, network science, etc. E.g. when I read concepts like “next Peter Singer” or a lot of thinking along the line “most of the value is created by just a few peple”, I’m worried. While such thinking is intuitively appealing, it can be quite superficial. E.g., a toy model: Imagine a landscape with gold scattered in power-law sized deposits. And prospectors, walking randomly, and randomly discovering deposits of gold. What you observe is the value of gold collected by prospectors is also power-law distributed. But obviously the attempts to emulate “the best” or find the “next best” would be futile. It seems open question (worth studying) how much some specific knowledge landscape resembles this model, or how big part of the success is attributable to luck.