Studying behaviour and interactions of boundedly rational agents, AI alignment and complex systems.
Research fellow at Future of Humanity Institute, Oxford. Other projects: European Summer Program on Rationality. Human-aligned AI Summer School. Epistea Lab.
[comment I’m likely to regret writing; still seems right]
It seems lot of people are reacting by voting, but the karma of the post is 0. It seems to me up-votes and down-votes are really not expressive enough, so I want to add a more complex reaction.
It is really very unfortunate that the post is framed around the question whether Will MacAskill is or is not honest. This is wrong, and makes any subsequent discussion difficult. (strong down-vote) (Also the conclusion (“he is not”) is not really supported by the evidence.)
It is (and was even more in the blog version) over-zealous, interpreting things uncharitably, and suggesting extreme actions. (downvote)
At the same time, it seems really important to have an open and critical discussion, and culture where people can challenge ‘canonical’ EA books and movement leaders. (upvote)
Carefully going through the sources and checking if papers are not cherry-picked and represented truthfully is commendable. (upvote)
Having really good epistemics is really important, in particular with the focus on long-term. Vigilance in this direction seems good. (upvote)
So it seems really a pity the post was not framed as a question somewhere in the direction “do you thing this is epistemically good?”
If I try to imagine something like “steel-maned version of the post”, without questioning honesty, and without making uncharitable inferences, the reaction could have been some useful discussion.
It seems to me
“Doing Good Better” is sometimes more on the “explaining & advocacy of ideas” side than “dispassionate representation of research”.
Given the genre, I would bet the book is in the top quartile on the metric of representing research correctly.
In some of the examples, it seems adding more caveats and reporting in more detail would have been better for readers interested in precision. Likely at the cost of making the text more dry.
Some emotions sometimes creep in: In case of the somewhat uncharitable part about Charity Evaluator, I remembered their much more uncharitable / misrepresenting text attacking effective altruism and GiveWell. Also while they talk about importance of other things, what they actually measure is actually wrong, and criticized correctly. In case of the whole topic… well a lot of evidence points toward things like 100x multiplier being true, meaning that yes, actually it is possible to save many more people. It seems hard to not to have some passion.
Given that several books about long-term future are now written, the update I would take from that is the books mostly about long-term should err more the side of caveating and describing disagreements and explaining uncertainty more, but my feeling is shift in this direction already happened between 2014 and 2018.
For me, unfortunately, the discourse surrounding Wytham Abbey, seems like a sign of epistemic decline of the community, or at least on the EA forum.
The amount of attentions spent on this seems to be a textbook example of bikeshedding.
Quoting Parkinson :”The time spent on any item of the agenda will be in inverse proportion to the sum [of money] involved.” A reactor is so vastly expensive and complicated that an average person cannot understand it (see ambiguity aversion), so one assumes that those who work on it understand it. However, everyone can visualize a cheap, simple bicycle shed, so planning one can result in endless discussions because everyone involved wants to implement their own proposal and demonstrate personal contribution.
In case of EAs, there are complicated, high-stakes things, for example what R&D efforts to support around AI. This has scale of billions of dollars now, much higher stakes in the future, and there is a lot to understand.
In contrast, absolutely anyone can easily form opinions about appropriateness of a manor house purchase, based on reading a few tweets.
Repeatedly, the tone of the discussion is a bit like “I’ve read a tweet by Émile Torres, I got upset, and I’m writing on EA forum”. Well, tweets by Émile Torres are known to be an unreliable source of information, and in contrast are often optimized to create negative emotional response. (To not single out Torres, this is also true for many other tweets.)
The discussion often almost completely misses the direct, object-level, even if just at back-of-the-envelope estimate way. On this level venue purchases were sensible:
the amount of EA x-risk-reduction money per person in early 2022 was pretty high
organising events and setting up venues is labour intensive, often more difficult to delegate than people assume, and often constrained on time of people who have high opportunity costs
on the margin, you can often trade money, time, work … this trade seems to make sense
apparently, multiple people reached similar conclusion; apart from Wytham, there was for example a different venue purchased in the Bay area, and other near Prague (note: I’m leading the org fiscally sponsoring the later project)
In contrast, large fraction attention in the discussion seems spent on topics which are both two steps removed from the actual thing , and very open to opinions. Where by one step removed I mean e.g. “how was this announced” or “how was this decided”, and two steps removed is e.g. “what will be the impact of how was this announced on the sentiment of the twitter discussion”. While I do agree such considerations can have large effect, driving decisions by this type of reasoning in my view moves people and orgs into the sphere of pure PR, spin and appearance.
FWIW … in my opinion, retaining the property might have been a more beneficial decision.
Also, I think some people working in the space should not make an update against plans like “have a permanent venue”, but plausibly should make some updates about the “major donors”. My guess this almost certainly means Open Philanthropy, and also likely they had most of the actual power in this decision.
Before delving further, it’s important to outline some potential conflicts of interest and biases:
- I did co-organize or participated at multiple events at Wytham. For example, in 2023, ACS organized a private research retreat aimed at increasing the surface area between Active Inference and AI Alignment communities. The event succeeded to attracts some of the best people from both sides and was pretty valuable for the direction of alignment research I do care about, and the Oxford location was very useful for that. I regret running events like that in future will be more difficult.
- I have friends in all orgs or sides involved—Wytham project, Open Phil, EV, EAs who disapproved the purchase,...
- I lead an org funded by Open Philanthropy—
I also lead an org which was fiscally sponsoring a different project of venue purchase, which was funded by FTX regrant (won’t comment on that for legal reasons)
Also, without more details published, my current opinion is personal speculation, partially based on my reading of the vibes.
My impression-from-a-distance is part of the decision is driven by a factor which I think should not be given undue weight, and a factor where I likely disagree
Factor where I possibly disagree is aesthetics. As far as I can tell, the current preferred EA aesthetics is something more similar to how recent EAG Bay looked like. At EAG Bay, my impression of the vibes of venue was… quite dystopian—the main space is a giant hall with unpleasant artificial lights, no natural light, no colours, endless rows of identical black tables utilized by people having endless rows of 1:1s. In some vague aesthetic space, nearby vibes vectors are faceless bureaucracies, borgs and the scifi portrayals of heart-less technocratic baddies. Also something about naive utilitarianism and army.
Wytham seemed to stand in stark contrast to this aesthetics: the building was old and full of quirks. The vibes were more like an old Oxford college.
Factor which I would guess was part of the decision and I suspect had weight was PR concerns. Wytham definitely got some negative media coverage both in traditional media, social media, and on this forum.
What I dislike about this, these concerns often seemed to be mostly on the Simulacra levels 3 and 4, detached from the reality of running events in Oxford, or actual concern about costs. (Why I do think so? Because of approximately zero amount of negative PR, forum criticisms etc. anyone and anything in the ecosystem is getting for renting properties, even if they are more expensive per day or per person.)
To be clear—
I don’t think these were the only or main(?) factors.
- * I would expect somewhere also exists some spreadsheet with some estimates of “value” of events at Wytham. If this is the case, I probably also disagree about some of the generative opinions about what’s valuable.
Still, given the amount of speculative criticism the purchase of Wytham generated on the forum, it seems good for transparency to also express critical view about the sale.
This is a bit tangential/meta, but looking at the comment counter makes me want to express gratitude to the Community Health Team at CEA.
I think here we see a ‘practical demonstration’ of the counterfactuals of their work:
- insane amount of attention sucked by this
- the court of public opinions on fora seems basically strictly worse at all relevant dimensions like fairness, respect of privacy or compassion to people involved
As ‘something like this’ would be quite often the counterfactual to CH to trying to deal with stuff …it makes it clear how much value they are creating by dealing with these problems, even if their process is imperfect
Both “EA translation service” and “EA spaces everywhere” seem like ideas which can take good, but also many bad or outright harmful forms.
A few years ago, I tried to describe how to to establish a robustly good “local effective altruism” in a new country or culture (other than Anglosphere).
The super brief summary is
1. it’s not about translating a book, but about “transmission of a tradition of knowledge”
2. what is needed is a highly competent group of people
3. who can, apart from other things, figure out what the generative question of EA means in the given context (which may be quite different from Oxford or Bay)
Point #2 is the bottleneck. Without it, efforts to have “country/language X EA” will be often unsuccessful or worse. Doing Good Better was translated to e.g. japanese. It is doing a bit worse on Amazon.co.jp than the english version on Amazon.co.uk, but not by many orders. Yet you aren’t seeing a lot of content and projects from EA Japan.
So: One good version of “translation service” seems to be basically giving functional national EA groups money to get as good translations as needed.
One bad version is a “centralized project” trying to translate centrally selected content to centrally selected languages by hiring professional contractors to do it.
Similarly EA spaces:
Hiring offices in 100 cities is easy, and you can do that centrally. Running a space with a good culture and gatekeeping is harder, and bottlenecked on “#2″.
Good version of the project seems to be basically giving functional city EA groups money to get good coworking spaces, which is probably doable via CEA group support.
Many bad versions are somewhere close to “centralized megaproject to run EA spaces everywhere”.
In other words: it isn’t easy to get around the bottleneck ‘you need people on the ground’.
Footnote: there are many variations on this theme. The basic pattern roughly is:
“notice that successful groups are doing activity X” (e.g. they have a coworking space, or materials in the local language, or they organize events,...).
the next step, in which this goes wrong, is, “let’s start a project P(X) that will try to make activity X happen everywhere”.
Also: The chances of bad or explicitly harmful outcomes increase roughly in proportion to the combination of cultural, network and geographic distance from the “centre” from which such activity should be directed. Eg project which will try to run spaces in Stanford from Berkeley seem fine
I wanted to provide some context on the recently purchased events venue near Prague that has been discussed here in one of the subthreads. The subthread contains some misleading content, and is not clearly visible, so I am posting this as a top level comment to provide more information on the project. It is extremely long, sorry.
1. My relationship to the project
2. Context for Forum readers
3. Context for outsiders
4. About the project
5. About the venue
6. Communication timelines
1. My relationship to the project
The project is led by Irena Kotikova and was something she started thinking about and working on multiple years ago, before the FTX Future Fund existed.
I am not the recipient of the grant, nor am I responsible for the funding decision: I’m involved in the project as chairman of ESPR, which is a Czech non-profit housing part of the Hostacov project. The ESPR association decided to fiscally sponsor the project because it aligned with our mission (improving epistemics) and we already had the necessary “equivalency determination” to receive tax-exempt funding from a US foundation. However, the Hostacov project is independent and the property is owned by a separate legal entity. The ESPR association is not involved in the operation of the property.
I personally believe the project is good and I’m happy to defend it in my private capacity. The following comments are mine and not made on behalf of the project or its funder. I expect Irena to comment on this as well. (Even more context: I am friends with Irena and have collaborated with her on many projects.)
2. Context for EAs
Irena and I were fairly involved with the Czech government’s Covid response. Our efforts were at least partially successful and likely contributed to preventing thousands of death and preventing some economic damage. However, it also made us quite unpopular among anti-vax and anti-lockdown activists. (In the US context, you can imagine these groups as falling somewhere between the Great Barrington Declaration and QAnon.)
Based on one online media report, these circles noticed the venue purchase and are trying to frame our involvement in a false negative light, ranging from “me and Irena were aware of fraud within FTX” to “both we and FTX are part of an Illuminati conspiracy, we likely knew about the pandemic before it happened, our covid response efforts were part of the plot, and the grant is an expression of gratitude from the Illuminati”. Obviously bizarre, but sadly not harmless.
As a result, please note that the Czech media and these Covid sceptics might well find this discussion. So, anything you write on this post could easily be selectively quoted or otherwise used against you or me. I am not trying to prevent people expressing their opinions, but would ask that any speculation (or, for example, digging for the physical addresses of vaguely related entities) be done with the above in mind.
3. Context for outsiders (you can skip if you have followed the forum in past month)
In case you’re not familiar with FTX and its ill-fated philanthropy: FTX was a large crypto exchange that went bankrupt in early November. Soon after it was revealed that its founder, Sam Bankman-Fried, lied about the state of their assets and may have used client money in a fraudulent way. I recommend Spencer Greenberg’s podcast for more details.
FTX founded a charity, the FTX Future Fund / Foundation. The foundation was small compared to FTX, but quickly became a significant funder in several areas that I find important—AI safety, pandemic preparedness, catastrophic risks, tools to support collective epistemology, teaching critical thinking, etc.
Regardless of what exactly happened at FTX, the result is extremely bad for everyone. First of all for FTX customers—it is unclear what fraction will be returned to them in the bankruptcy proceedings. But it is also bad for people who worked at the FTX Foundation, grant recipients, and those who were promised grants but received nothing.
According to its website, the FTX Foundation has given out hundreds of grants, more than $130M by summer 2022. If they continued their pace, by November this was likely well over $200M. The group of recipients is diverse, from research groups at about half of the top 10 universities in the relevant fields, to tiny non-profits and individuals. Personally, I think that most of the grants were beneficial.
What I want to emphasise is that the grant recipients have done nothing wrong and have nothing to be ashamed of or apologise for.
The implication “you got a grant from a foundation that was sponsored by FTX” ⇒ “what you are doing is suspicious” is nonsense.
Unfortunately, some parts of online space are attempting to paint grant recipients as themselves scandalous or immoral, and shaming them. From what I have seen, this is based on ignoring context or on a clearly wrong framing:
1. Suggestive framing like “they got a grant not long before bankruptcy”. The implication is that grantees should have done – in addition to basic due diligence about the FTX Foundation being a legitimate entity and usual anti-money laundering – some sort of brilliant investigation to discover that FTX was going to go bankrupt in a few months because of fraud inside it.
This is clearly an unreasonable demand. Ironically, if someone had reliable information proving the insolvency of FTX a few months ago, such information would be extremely valuable—it seems likely that the market worth of this info would be greater than total FTX Foundation donations.
2. Ignoring causality. For example, Irena was working on the discussed project for years. It isn’t the case the SBF conceived this.
3. Thinking the grant recipients can just send the money back now. Grants are governed by contracts; organizations are governed by laws and bylaws. What should someone who has received money from the FTX Foundation do? Intuitively, one might think it best to immediately send the money “back” to FTX creditors. In practice, at least for a Czech non-profit, this would be likely illegal: a non-profit governed under Czech law cannot just take a donation it received from the Foundation for a specific purpose and send it to a private entity in the Bahamas. The reasonable option seems to be to wait for how the court proceedings develop, and adhere to contracts and the law. (This is not a legal advice)
4. Liability. Both the law and commonsense protect you if you took an action in good faith, as the grantees did.
4. About the project
As I mentioned earlier, this project hasn’t been announced yet.
While I’m not speaking on behalf of the project, I can try to explain why it seems good, and something like this should exist, from my personal perspective.
I. First, I believe the venue should not be thought of as “EAs for EAs” but rather as something that aims to improve the world. How? By providing a space for topics that are adjacent to EA, such as good epistemics, sanity in times of rapid change, increasingly strong AI, and existential security.
A typical use I imagine is an ‘epistemics workshop for decision-makers’, with most of the guests not being EAs, but decision-makers or researchers. There are many possible events of the type ‘some people should meet and think about something’ or ‘some people should meet and learn something’ where the people meeting and learning is someone else than effective altruists, yet the whole event is quite impactful. In the above mentioned case, you can for example imagine someone working on e.g. regulation of AI in some non-EA capacity understanding better how to think in probability distributions and how to evaluate conflicting statements by various parties.
Overall, I think interventions of this type could be reasonably high-leverage.
II. Second, I believe that the space in which an event is held can make a significant impact on its success. It can easily contribute around 20% to some intuitive “goodness” metric. A stable and amazing venue can also create interesting feedback loops. For example, some people care more about research progress than money and if they have a positive experience at the venue, they may want to return even if monetary incentives point elsewhere.
III. Third, historically, EA and EA-adjacent organisations based in Prague have provided a lot of operational support for various events run by other organisations. This gives me data on the demand for this type of venue.
5. The venue
The venue is a castle located about an hour and a half from Vaclav Havel Airport. Previously, it was operated as a hotel and often hosted events such as weddings or corporate meetings. In its current state, it can host around 60 people, and there is potential for expansion.
The specific venue was selected after Irena scouted for venues for years and investigated dozens of alternatives. I have seen some of the alternatives and, in my opinion, the decisive factor was that the venue was ready to use.
Some of the alternatives were much cheaper but usually required years of expensive reconstruction. Running some of the events I would be excited to happen now seems much more impactful than doing them in 3 years.
Another question could be, why there, and not in some entirely different place. In my view, given the growth of the fields in question, there is a space for multiple venues in different locations, with different use cases.
Compared to the venues in Oxford and near Berkeley, this venue is significantly cheaper, somewhat larger and in better shape.
In terms of travel: the venue is well-located in continental Europe (roughly midway between Berlin and Vienna). Prague Airport has direct flights to most of Europe and travel time from almost any big European airport to the venue is less than 5 hours. In my view, this makes the venue viable for events in Europe longer than roughly 3 or 4 days, and basically globally, for week-long events or longer.
…also, to reiterate, contrary to rumour, the property is not owned by Irena Kotíková or Czech EA association or “EA Prague”. Also it’s probably worth understanding for people involved, the project is mostly a lot of work, not some sort of leisure spending.
6. Communication timelines
Buying a venue of this size takes time. In this case, it involved multiple checks by lawyers, accountants, and technicians. The grant agreement was signed in July, the transaction was not actually completed and visible in the registry until mid-October. It would have been odd to announce the purchase before the transaction was complete.
Nice post, but my rough take is there is
it’s relatively common markets are inefficient, but unexploitable; trading on “everyone dies” seems a clear case of hard-to-exploit inefficiency
markets are not magic; impacts of one-off events with complex consequences are difficult to price in, and what all the magical market aggregation boils down to are a bunch of human brains doing the trades; e.g. I was able to beat the market and get n-times return at point where markets were insane about covid; later, I talked about it with someone in one of the giant hedge funds, and the simple explanation is, while they were looking into it, at some point I knew more about covid than they were able to assemble
example of such hard-to-predict event are e.g. capabilities and impacts of a specific model
the dichotomy 30% growth / everyone dies is unrealistic for trading purposes
near term, there are various outcomes like “industry X gets disrupted” or “someone loses job due to automation” or “war”
if you anticipate fears of this type to dominate in next 10 years, you should price many people increasing their savings and borrowing less
meta: I considered commenting, but instead I’m just flagging that I find it somewhat hard to have an open discussion about the EA hotel on the EA forum in the fundraising context. The feeling part is
there is a lot of emotional investment in EA hotel,
it seems if the hotel runs out of runway, for some people it could mean basically loosing their home.
Overall my impression is posting critical comments would be somewhat antisocial, posting just positives or endorsements is against good epistemics, so the personally safest thing to do for many is not to say anything.
At the same time it is blatantly obvious there must be some scepticism about both the project and the outputs: the situation when the hotel seems to be almost out of runway repeats. While eg EA funds collect donations basically in millions $ per year, EA hotel struggles to collect low tens of $.
I think this equilibrium where
people are mostly silent but also mostly not supporting the hotel, at least financially
the the financial situation of the project is somewhat dire
talks with EA Grants and the EA Long Term Future Fund are in progress but the funders are not funding the project yet
is not good for anyone, and has some bad effects for the broader community. I’d be interested in ideas how to move out of this state.
I posted a short version of this, but I think people found it unhelpful, so I’m trying to post somewhat longer version.
I have seen some number of papers and talks broadly in the genre of “academic economy”
My intuition based on that is, often they seem to consist of projecting complex reality into a space of single-digit real number dimensions and a bunch of differential equations
The culture of the field often signals solving the equations is profound/important, and the how you do the projection “world → 10d” is less interesting
In my view for practical decision making and world-modelling it’s usually the opposite: the really hard and potentially profound part is the projection. Solving the maths is in often is some sense easy, at least in comparison to the best maths humans are doing
While I overall think the enterprise is worth to pursue, people should in my view have a relatively strong prior that for any conclusions which depends on the “world-> reals” projection there could be many alternatives leading to different conclusions; while I like the effort in this post to dig into how stable the conclusions are, in my view people who do not have cautious intuitions about the space of “academic economy models” could still easily over-update or trust too much the robustness
If people are not sure, an easy test could be something like “try to modify the projection in any way, so the conclusions do not hold”. At the same time this will usually not lead to an interesting or strong argument, it’s just trying some semi-random moves is the model space. But it can lead to a better intuition.
I tried to do few tests in a cheap and lazy way (eg what would this model tell me about running at night on a forested slope?) and my intuitions was:
I agree with the cautious the work in the paper represents very weak evidence for the conclusions that follow only from the detailed assumptions of the model in the present post. (At the same time it can be an excellent academic economy paper)
I’m more worried about other writing about the results, such as linked post on Phil’s blog , which in my reading signals more of “these results are robust” than it’s safe
Harder and more valuable work is to point to something like some of the most significant way in which the projection fails” (aspects of reality you ignored etc.). In this case this was done by Carl Shulman and it’s worth discussing further
In practice I do have some worries about some meme ‘ah, we don’t know, but given we don’t know, speeding up progress is likely good’ (as proved in this good paper) being created in the EA memetic ecosystem. (To be clear I don’t think the meme would reflect what Leopold or Ben believe)
I’m in favour to improving this coordination problem, but I think this particular solution is a bad idea and should not be started. The main problem is unilateralist’s curse. Suppose there is a really bad project which 19 out of 20 altruistically minded funders (and no professional grantmaker) would support. Your design of the structure would make it much more likely that it will get funded.
In general effective altruism has a lot of value in brand and goodwill and epistemic standards/culture (like in billions $). It seems relatively easy to create large negative impact by destroying part of this, which can be “achieved” even by relatively small project with modest funding. Public donor list seems to be literally the worst option for structure if we want to avoid bad projects.
I don’t think anyone should be trying to persuade IMO participants to join the EA community, and I also don’t think giving them “much more directly EA content” is a good idea.
I would prefer Math Olympiad winners to think about long-term, think better, and think independently, than to “join the EA community”. HPMoR seems ok because it is not a book trying to convince you to join a community, but mostly a book about ways how to think, and a good read.
(If they readers eventually become EAs after reasoning independently, it’s likely good; if they for example come to the conclusion there are mayor flaws in EA and it’s better to engage with the movement critically, it’s also good.)
Somewhat controversial personal opinion: when thinking about this space, my null hypothesis is talented people are actually abundant, and the bottleneck is on the side of EA organizations and possibly in culture.
Reasons could be various: for example, I can imagine
organizations are ops bottlenecked, and hiring is another ops task
historically, founder’s effect & difficulty of finding/identifying ops people if you have very different mindset
homophily&trust networks based hiring
misaligned filtering (potentially great ops people being filtered early on criteria like not having CVs impressive in the right way, in a later stage people with the most impressive CVs actually not being good fit, or not that interested in ops)
prestige
risk-aversion on org side
I don’t have a strong preference. There a some aspects in which longerism can be better framing, at least sometimes.
I. In a “longetermist” framework, x-risk reduction is the most important thing to work on for many orders of magnitude of uncertainty about the probability of x-risk in the next e.g. 30 years. (due to the weight of the long term future). Even if AI related x-risk is only 10ˆ-3 in next 30 years, it is still an extremely important problem or the most important one. In a “short-termist” view with, say, a discount rate of 5%, it is not nearly so clear.
The short-termist urgency of x-risk (“you and everyone you know will die”) depends on the x-risk probability being actually high, like of order 1 percent, or tens of percents . Arguments why this probability is actually so high are usually brittle pieces of mathematical philosophy (eg many specific individual claims by Eliezer Yudkowsky) or brittle use of proxies with lot of variables obviously missing from the reasoning (eg the report by Ajeya Cotra). Actual disagreements about probabilities are often in fact grounded in black-box intuitions about esoteric mathematical concepts. It is relatively easy to come with brittle pieces of philosophy arguing in the opposite direction: why this number is low. In fact my actual, action guiding estimate is not based on an argument conveyable by a few paragraphs, but more on something like “feeling you get after working on this over years”. What I can offer other is something like “an argument from testimony”, and I don’t think it’s that great.
II. Longermism is a positive word, pointing toward the fact that future could be large and nice. X-risk is the opposite.
Similar: AI safety vs AI alignment. My guess is the “AI safety” framing is by default more controversial and gets more of a pushback (eg “safety department” is usually not the most loved part of an organisation, with connotations like “safety people want to prevent us from doing what we want”)
In my view this text should come with multiple caveats.
- Beware ‘typical young researcher fallacy’. Young researchers are very diverse, and while some of them will benefit from the advice, some of them will not. I do not believe there is a general ‘formula for successful early-career research’. Different people have different styles of doing research, and even different metrics for what ‘successful research’ means. While certainly many people would benefit from the advice ‘your ideas are bad’, some young researchers actually have great ideas, should work on them, and avoid generally updating on research taste of most of the”senior researchers”.
- Beware ‘generalisation out of training distribution’ problems. Compared to some other fields, AI governance as studied by Allan Dafoe is relatively well decomposed into a hierarchy of problems and you can meaningfully scale it by adding junior people and telling them what to do (work on sub-problems senior people consider interesting). This seems more typical for research fields with established paradigms than for fields which are pre-paradigmatic, or fields in need of a change of paradigm.
- Large part of the described formula for success seems to be optimised for success in the direction getting attention of senior researchers, writing something well received, or similar. This is highly practical, and likely good for many people in fields like Ai governance; at the same time, it seems the best research outputs by early career researchers in eg AI safety do not follow this generative pattern, and seem to be motivated more by curiosity, reasoning from first principles, and ignoring authority opinions.
If we had incentivised whistle-blowers to come forward around shady things happening at FTX, would we have known about FTX fraud sooner and been less reliant on FTX funding? Very plausibly yes. She says “likely” which is obviously not particularly specific, but this would fit my definition of likely.
Why do you think so? Whistleblowers inside of FTX would have been protected under US law, and US institution like SEC offer them multi-million dollar bounties. Why would EA scheme create stronger incentive?
Also: even if the possible whistleblowers inside of FTX were EAs, whistleblowing about fraud at FTX not directed toward authorities like SEC, but toward some EA org scheme, would have been particularly bad idea. The EA scheme would not be equipped to deal with this and would need to basically immediately forward it to authorities, leading to immediate FTX collapse. Main difference would be putting EAs in the centre of the happenings?
If EA had diversified our portfolio to be less reliant on a few central donors, this would have also (quite obviously) mean the crash had less impact on EA overall, so this also seems true.
I think the ‘diversified our portfolio’ frame is subtly misleading, because it’s usually associated with investments or holdings, but here it is applied to ‘donors’. You can’t diversify donations the same way. Also: assume you recruit donors uniformly, no matter how wealthy they are. Most of the wealth will be with the wealthiest minority , basically because how the wealth distribution looks like. Attempt to diversify donation portfolio toward smaller donors … would look like GWWC?
Only real option how to have much less FTX money in EA was to not accept that much FTX funding. Which was a tough call at the time, in part because FTX FF seemed like the biggest step toward decentralized distribution of funding, and a big step toward diversifying from OP.
As a whole, I think effective altruism is currently more structurally bottlenecked and network bottlenecked than funding bottlenecked. Improving the structural and networking constrains is higher leverage than adding more money to the system (on average). Which is not say increasing funding is not valuable. I would expect this to depend a lot on individual circumstances.
If you look on the funding bottlenecks, they seem to be mostly the result of the structure of the funding, that of aggregate sum: imagine a counterfactual world in which OpenPhil has $1b more than it has. How much more effective actions you would expect to see in the world?
So also in funding, we need more structures. EtG is highly impactful as far as the giving is smart, with money directed toward alleviating the structural constrains of funding. I don’t think this scales “endlessly”.
Another point is, from a global perspective, EtG makes more sense in some places than others. For example, a philosophy postdoc in Prague can earn as little as £1100/m. Should such a person drop an academic career, and do EtG? Almost certainly not. What about EAs in India? They have likely very different comparative advantages than EtG.
(i.e. most people who are likely to update downwards on Yudkowsky on the basis of this post, seem to me to be generically too trusting, and I am confident I can write a more compelling post about any other central figure in Effective Altruism that would likely cause you to update downwards even more)
My impression is the post is somewhat unfortunate attempt to “patch” the situation in which many generically too trusting people updated a lot on AGI Ruin: A List of Lethalities and Death with Dignity and subsequent deference/update cascades.
In my view the deeper problem here is instead of disagreements about model internals, many of these people do some sort of “averaging conclusions” move, based on signals like seniority, karma, vibes, etc.
Many of these signals are currently wildly off from truth-tracking, so you get attempts to push the conclusion-updates directly.
For what it’s worth, my estimate of total current funding gap in the sector of “small and new projects motivated by long-term”, counting only “what has robustly positive EV by my estimate”, is >$1M.
In general, I think the ecosystem suffered from the spread of several over-simplified memes. One of them is “the field is talent constrained”, the other one “now with the OpenPhil money...”.
One way how to think about it* is projecting the space along two axes: “project size” and “risks/establishedness”. Relative abundance of funding in the [“medium to large”, “low risks / established”] sector does not imply much about the funding situation in the [“small”,”not established”] sector, or [“medium size”,”unproven/risky”] sector.
The [“medium to large”, “low risks / established”] sector is often constrained by a mix of structural limits how fast can organizations grow without large negative side-effects, bottlenecks in hiring, and yes, sometimes, by very specific talent needs. Much less by funding.
On the opposite side, the [“small”,”not established”] sector is probably funding constrained, plus constrained by a lack of advisors and similar support, and inadequacies in the trust network structure.
Long Term Future fund moving to fill part of the funding gap seems great news.
(*I have this from analysis how an x-risk funding organization can work by Karl Koch & strategy team at AI Safety Camp 1, non-public)
In my view this is a bad decision.
As I wrote on LW
Sorry but my rough impression from the post is you seem to be at least as confused about where the difficulties are as average of alignment researchers you think are not on the ball—and the style of somewhat strawmanning everyone & strong words is a bit irritating.
In particular I don’t appreciate the epistemic of these moves together
1. Appeal to seeing thinks from close proximity. Then I got to see things more up close. And here’s the thing: nobody’s actually on the friggin’ ball on this one!
2. Straw-manning and weakmaning what almost everyone else thinks and is doing
3. Use of an emotionally compelling words like ‘real science’ for vaguely defined subjects where the content may be the opposite of what people imagine. Is the empirical alchemy-style ML type of research what’s advocated for as the real science?
4. What overall sounds more like the aim is to persuade, rather than explain
I think curating this signals this type of bad epistemics is fine, as long as you are strawmanning and misrepresenting others in a legible way and your writing is persuasive. Also there is no need to actually engage with existing arguments, you can just claim seeing things more up close.
Also to what extent are moderator decisions influenced by status and centrality in the community...
… if someone new and non-central to the community came up with this brilliant set of ideas how to solve AI safety:
1. everyone working on it is not on the ball. why? they are all working on wrong things!
2. promising is to do something very close to how empirical ML capabilities research works
3. this is a type of problem where you can just throw money at it and attract better ML talent
… I doubt this would have a high chance of becoming curated.
I. It might be worth reflecting upon how large part of this seem tied to something like “climbing the EA social ladder”.
E.g. just from the first part, emphasis mine
Replace “EA” by some other environment with prestige gradients, and you have something like a highly generic social climbing guide. Seek cool kids, hang around them, go to exclusive parties, get good at signalling.
II. This isn’t to say this is bad . Climbing the ladder to some extent could be instrumentally useful, or even necessary, for an ability to do some interesting things, sometimes.
III. But note the hidden costs. Climbing the social ladder can trade of against building things. Learning all the Berkeley vibes can trade of against, eg., learning the math actually useful for understanding agency.
I don’t think this has any clear bottom line—I do agree for many people caring about EA topics it’s useful to come to the Bay from time to time. Compared to the original post I would probably mainly suggest to also consult virtue ethics and think about what sort of person you are changing yourself to, and if you, for example, most want to become “a highly cool and well networked EA” or e.g. “do things which need to be done”, which are different goals.