Director of Epoch, an organization investigating the future of Artificial Intelligence.
Currently working on:
Macroeconomic models of AI takeoff
Trends in Artificial Intelligence
Forecasting cumulative records
Improving forecast aggregation
I am also one of the coordinators of Riesgos Catastróficos Globales, a Spanish-speaking network of experts working on Global Catastrophic Risks.
I also run Connectome Art, an online art gallery where I host art I made using AI.
An encouraging update: thanks to the generous support of donors, we have raised $95k in funds to support our activities for six more months. During this time, we plan to 1) engage with the EU trilogue on the regulation of foundation models during the Spanish presidency of the EU council, 2) continue our engagement with policy markers in Argentina and 3) release a report on global risk management in latin america.
We nevertheless remain funding constrained. With more funding we would be able to launch projects such as:
A report on prioritizing and forecasting global catastrophic risk that could be used by stakeholders such as the regional UNNDR office or CEPAL.
Expand our previous work on mapping BSL-3 and BSL-4 labs in Latin America.
Develop a theory of change for how latin america could contribute through multilateral negotiations to the management of AI risk, covering a study of the historic role of latam countries in the nuclear risk treaties and the management of other global risks.
Further our relationship with Uruguay and other countries in the MERCOSUR to invite them to follow Argentina in the formulation of emergency plans for food security in case of nuclear winter.
And other priority projects to improve the management of global catastophic risk in Spanish-Speaking countries.
Each of these projects could cost between $30k and $80k to develop. You can support us with a donation to help us develop these projects. You may also reach out to me through a PM if you are considering donating and want more information.
I have grips with the methodology of the article, but I don’t think highlighting the geometric mean of odds over the mean of probabilities is a major fault. The core problem is assuming independence over the predictions at each stage. The right move would have been to aggregate the total P(doom) of each forecaster using geo mean of odds (not that I think that asking random people and aggregating their beliefs like this is particularly strong evidence).
The intuition pump that if someone assigns a zero percent chance then the geomean aggregate breaks is flawed:
There is an equally compelling pump the other way around: the arithmetic mean of probabilities defers unduly to people assigning a high chance. A single dissenter between 10 experts can bound the lower bound of the probability to their preferred up to a factor of 10.
And surely if anyone is assigning a zero percent chance to something, you can safely assume they are not taking the situation seriously and ignore them.
In ultimate instance, we can theorize all we want, but as a matter of fact the best performance when predicting complex events is achieved when taking the geometric mean of odds, both in terms of logloss and brier scores. Without more compelling evidence or a very clear theoretical reason that distinguishes between the contexts, it seems weird to argue that we should treat AI risk differently.
And if you are still worried about dissenters skewing the predictions, one common strategy is to winsorize, by clipping the predictions among the 5% and 95% percentile for example.
Yeah in hindisght that is probably about right.
It’d be interesting to look at some of these high profile journalists, and see if they are well supported to do impactful journalism or if they have to spend a lot of time on chasing trends to afford working on IJ pieces.
I looked into the impact of investigative journalism briefly a while back.
Here is an estimation of the impact of some of the most high profile IJ success stories I found:
How many investigative journalists exist? It’s hard to say, but the International Federation of Journalists has 600k members, so maybe there exists 6M journalists worlwide, of which maybe 10% are investigative journalists (600k IJs). If they are paid like $50k/year, that’s $30B used for IJ.
Putting both numbers together, that’s $2k to $20k per person affected. Let’s say that each person affected gains 0.1 to 10 QALYs for these high profile cases, then that’s $200 to $200k per QALY.
Seems to not be competitive with global health interventions, which are around $60/QALY IIUC, though of course this is neglecting that IJ has many important cultural effects (but then again, so does curing children from malaria!). I could also be grossly overestimating how much money goes to investigative journalism, and of course I am neglecting that the marginal dollar is probably much much less impactful than the average dollar.
Do not take this two minute exercise too seriously though! I’d be keen on seeing a more careful approach to it.
What are some open questions in your mind, for potential GCR priorities you haven’t had time to investigate?
What risks do you feel are particularly neglected by the EA community?
What opportunities are you most excited about for GCR mitigation outside the Anglosphere?
I don’t think EAs have any particular edge in managing harassment over the police, and I find it troublesome that they have far higher standards for creating safe spaces, specially in situations where the cost is relatively low such as inviting the affected members to separate EAG events while things cool down.
On another point, I don’t think this was Jeff’s intention, but I really dislike the unintentional parallel between an untreated schizophrenic and the CB who asked for the restraining order. I can assure you that this was not the case here, and I think the CB was sound of mind and fully justified in requesting it.
Whether it is easy to get a restraining order, IDK man. It is “easy” in the sense that there is a same-day procedure that only requires interrogating the requester. At least in the jurisdiction where this happened, it is also a gruelling process which requires you to consistently repeat your story to many police officials who keep asking you for details and are looking for contradictions, and takes a full work day to complete.
I agree that this is very valuable. I would want them to be explicit about this role, and be clear to community builders talking to them that they should treat them as if talking to a funder.
To be clear, in the cases where I have felt uncomfortable it was not “X is engaging in sketchy behaviour, and we recommend not giving them funding” (my understanding is that this happens fairly often, and I am glad for it. CHT is providing a very valuable function here, which otherwise would be hard to coordinate. If anything, I would want them to be more brazen and ready to recommend against people based on less evidence than they do now).
Is more cases like “CHT staff thinks that this subcommunity would work better without central coordination, and this staff is going to recommend against funding any coordinators going forward” or “CHT is pressuring me to make a certain choice such as not banning a community member I consider problematic, and I am afraid that if I don’t comply I won’t get renewed” (I’ve learned of situations like these happen at least thrice).
It is difficult to orient yourself towards someone who you are not sure whether your should treat as your boss or as neutral third party mediator. This is stressing for community builders.
I echo the general sentiment—I find the CHT to work diligently and be in most cases compassionate. I generally look up to the people who make it up, and I think they put a lot of thought into their decisions. From my experience, they helped prevent at least three problematic people from accruing power and access to funding in the Spanish Speaking community, and have invested 100s of hours into steering that sub community towards what they think is a better direction, including being always available for consultations.
I also think that they undervalue the work and wellbeing of community builders, that they have a lot of unaccountable influence on grant decisions and that they make some decisions that I don’t think an experienced HR / conflict mediation team would endorse and that causes very competent community builders to turn away from the job, more below
My overall impression is that the CEA community health team (CHT from now on) are well intentioned but sometimes understaffed and other times downright incompetent. It’s hard to me to be impartial here, and I understand that their failures are more salient to me than their successes. Yet I endorse the need for change, at the very least including 1) removing people from the CHT that serve as a advisors to any EA funds or have other conflict of interest positions, 2) hiring HR and mental health specialists with credentials, 3) publicly clarifying their role and mandate.
My impression is that the most valuable function that the CHT provides is as support of community building teams across the world, from advising community builders to preventing problematic community builders from receiving support. If this is the case, I think it would be best to rebrand the CHT as a CEA HR department, and for CEA to properly hire the community builders who are now supported as grantees, which one could argue is an employee misclassification.
I would not be comfortable discussing these issues openly out of concern for the people affected, but here are some horror stories:
A CHT staff pressured a community builder to put through with and include a community member with whom they weren’t comfortable interacting.
A CHT staff pressured a community builder to not press charges against a community member who they felt harassed by.
After a restraining order was set by the police in place in this last case, the CHT refused to liaison with the EA Global team to deny access to the person restrained, even knowing that the affected community builder would be attending the event.
My overall sense is that CHT is not very mindful of the needs of community builders in other contexts. Two very promising professionals I’ve mentored have dissociated from EA, and rejected a grant, in large part because of how they were treated by the CHT.
My impression is that the CHT staff undermines the legitimacy of local communities to make their own decisions. CEA is often perceived as a source of authority, and the CHT has a lot of sway in funding decisions. This makes it so that it is really hard for local groups to go against the wishes of the CHT, who is the main intermediary with groups. I wish this relation was more transparent, so they could be hold accountable for it.
To be clear, I think that these stories have a lot of nuance to them and are in each cases the result of the CHT making what they thought were the best decisions they could make with the tools they had, but in each of them I noticed that I ended up disagreeing with the decisions made and feeling very uncomfortable with how the whole community structure was set up.
Factor increase per year is the way we are reporting growth rates by default now in the dashboard.
And I agree it will be better interpreted by the public. On the other hand, multiplying numbers is hard, so it’s not as nice for mental arithmetic. And thinking logarithmically puts you in the right frame of mind.
Saying that GPT-4 was trained on x100 more compute than GPT-3 invokes GPT-3 being 100 times better, whereas I think saying it was trained on 2 OOM more compute gives you a better picture of the expected improvement.
I might be wrong here.
In any case, it is still a better choice than doubling times.
My sense is that pure math courses mostly make it easier to do pure math later, and otherwise not make much of a difference. In terms of credentials, I would bet they are mostly equivalent—you can steer your career later through a master or taking jobs in your area of interest.
Statistics seems more robustly useful and transferable no matter what you do later. Statistics degrees also tend to cover the very useful parts of math as well, such as calculus and linear algebra.
Within Epoch there is a months-long debate about how we should report growth rates for certain key quantities such as the amount of compute used for training runs.
I have been an advocate of an unusual choice: orders-of-magnitude per year (abbreviated OOMs/year). Why is that? Let’s look at other popular choices.
Doubling times. This has become the standard in AI forecasting, and its a terrible metric. On the positive, it is an intuitive metric that both policy makers and researchers are familiar with. But it is absolutely horrid to make calculations. For example, if I know that the cost of AI training runs is doubling every 0.6 years, and the FLOP/$ is doubling every 2.5 years, then the FLOP per training run is doubling every years, which is very difficult to solve in your head! [1] [2]
Percent growth. This is a choice often favoured in economics, where eg the growth rate of GDP is often reported as 3%, etc. Unlike doubling times, percent growth composes nicely—you just have to add them up! [3] However, I also find percent changes somewhat prone to confusion. For instance, when I tell people that model size has increased 200% since X years ago, I sometimes have had people misunderstand this as saying that it has increased by a factor of 2.
Ultimately, a very common operation that I find myself doing in my head is “if the effective FLOP used in AI training runs grows at a certain rate, how quickly will we traverse from the scale of current training runs (1e25 FLOP) to a certain threshold (eg 1e30)”. OOMs/year makes this computation easy, even if I need to account for multiple factors such as hardware increases, investment and algorithmic improvements. Eg if these grow respectively as 0.4 OOM/year, 0.1 OOM/year and 0.5 OOM/year, then I know the total effective growth is 1.0 OOM/year, and it will take 5 years to cross that 5 OOM scale gap. And if investment suddenly stopped growing, then I would be able to quickly understand that the pace would be halved, and the gap would then take 10 years to cross.
Sadly OOMs/year is uncommon, and both researchers and policy makers struggle to understand it. I think this is a missed opportunity, and that AI forecasting would be easier to reason about if we moved to it, or at the very least abandoned the very badly behaved doubling time framing.
What do you think? Do you agree we should move past doubling times to a better choice? Which choice would you favour?
I won’t enter into the technical details, but it also has some very unintuitive results when combining the results with uncertainty. Once we had a discussion because some doubling times looked like they had to be wrong. They spanned from days to years! But it turns out that doubling times are very sensitive to noise, which led to our intuitions being wrong.
I’d also argue that Christiano’s operationalization of slow takeoff is a terrible definition, and that a big part of that terribleness stems from doubling times being very unintuitive.
This is because have the useful property that for percent changes close to zero, which links percent growth to a straightforward model of growth such as . But this approximation breaks down for percent changes over 1 (which are often seen in AI forecasting).
I wrote a report on quantum cryptanalysis!
The TL;DR is that it doesn’t seem particularly prioritary, since it is well attended (the NIST has an active post quantum cryptography program) and not that impactful (there are already multiple proposals to update the relevant protocols, and even if these pan out we can rearrange society around them).
I could change my mind if I looked again into the NIST PQC program and learned that they are significantly behind schedule, or if I learned that qauntum computing technology has significantly accelerated since.
I mostly think of alignment as about avoiding deception or catastrophic misgeneralization outside of testing settings.
In general I also believe that AI companies have a massive incentive to align their systems with user intent. You can’t profit if you are dead.
This is a tough question, and I don’t really know the first thing about patents, so you probably should not listen to me.
My not very meditated first instinct is that one would set up a licensing scheme so that you commit to granting a use license to any manufacturer in exchange for a fee, preferrably as a portion of the revenue to incentivize experimentation from entrepreneurs.
Of course, this might dissuade the Tech Transfer office or other entrepreneurs from investing in a start-up if they won’t be guaranteed exclusive use.
I suppose that it would be relevant to investigate how succesful the Tech Transfer office has been in launching startups in the past, what types of permissive licenses they would be amenable to, and whether you know some other entrepreneurs or accelerators who would be keen to commercialize the idea (and what types of licenses they would consider working with).
In any case, good luck with the effort—this could be a very exciting opportunity!
As far as I can tell they are enabled—I see there is a cookie in storage for the intercom for example
Note that the AMF example does not quite work, because if each net has a 0.3% chance of preventing death, and all are independent, then with 330M nets you are >99% sure of saving at least ~988k people.