Director of Epoch, an organization investigating the future of Artificial Intelligence.
Currently working on:
Macroeconomic models of AI takeoff
Trends in Artificial Intelligence
Forecasting cumulative records
Improving forecast aggregation
I am also one of the coordinators of Riesgos Catastróficos Globales, a Spanish-speaking network of experts working on Global Catastrophic Risks.
I also run Connectome Art, an online art gallery where I host art I made using AI.
I agree that this is very valuable. I would want them to be explicit about this role, and be clear to community builders talking to them that they should treat them as if talking to a funder.
To be clear, in the cases where I have felt uncomfortable it was not “X is engaging in sketchy behaviour, and we recommend not giving them funding” (my understanding is that this happens fairly often, and I am glad for it. CHT is providing a very valuable function here, which otherwise would be hard to coordinate. If anything, I would want them to be more brazen and ready to recommend against people based on less evidence than they do now).
Is more cases like “CHT staff thinks that this subcommunity would work better without central coordination, and this staff is going to recommend against funding any coordinators going forward” or “CHT is pressuring me to make a certain choice such as not banning a community member I consider problematic, and I am afraid that if I don’t comply I won’t get renewed” (I’ve learned of situations like these happen at least thrice).
It is difficult to orient yourself towards someone who you are not sure whether your should treat as your boss or as neutral third party mediator. This is stressing for community builders.
I echo the general sentiment—I find the CHT to work diligently and be in most cases compassionate. I generally look up to the people who make it up, and I think they put a lot of thought into their decisions. From my experience, they helped prevent at least three problematic people from accruing power and access to funding in the Spanish Speaking community, and have invested 100s of hours into steering that sub community towards what they think is a better direction, including being always available for consultations.
I also think that they undervalue the work and wellbeing of community builders, that they have a lot of unaccountable influence on grant decisions and that they make some decisions that I don’t think an experienced HR / conflict mediation team would endorse and that causes very competent community builders to turn away from the job, more below
My overall impression is that the CEA community health team (CHT from now on) are well intentioned but sometimes understaffed and other times downright incompetent. It’s hard to me to be impartial here, and I understand that their failures are more salient to me than their successes. Yet I endorse the need for change, at the very least including 1) removing people from the CHT that serve as a advisors to any EA funds or have other conflict of interest positions, 2) hiring HR and mental health specialists with credentials, 3) publicly clarifying their role and mandate.
My impression is that the most valuable function that the CHT provides is as support of community building teams across the world, from advising community builders to preventing problematic community builders from receiving support. If this is the case, I think it would be best to rebrand the CHT as a CEA HR department, and for CEA to properly hire the community builders who are now supported as grantees, which one could argue is an employee misclassification.
I would not be comfortable discussing these issues openly out of concern for the people affected, but here are some horror stories:
A CHT staff pressured a community builder to put through with and include a community member with whom they weren’t comfortable interacting.
A CHT staff pressured a community builder to not press charges against a community member who they felt harassed by.
After a restraining order was set by the police in place in this last case, the CHT refused to liaison with the EA Global team to deny access to the person restrained, even knowing that the affected community builder would be attending the event.
My overall sense is that CHT is not very mindful of the needs of community builders in other contexts. Two very promising professionals I’ve mentored have dissociated from EA, and rejected a grant, in large part because of how they were treated by the CHT.
My impression is that the CHT staff undermines the legitimacy of local communities to make their own decisions. CEA is often perceived as a source of authority, and the CHT has a lot of sway in funding decisions. This makes it so that it is really hard for local groups to go against the wishes of the CHT, who is the main intermediary with groups. I wish this relation was more transparent, so they could be hold accountable for it.
To be clear, I think that these stories have a lot of nuance to them and are in each cases the result of the CHT making what they thought were the best decisions they could make with the tools they had, but in each of them I noticed that I ended up disagreeing with the decisions made and feeling very uncomfortable with how the whole community structure was set up.
Factor increase per year is the way we are reporting growth rates by default now in the dashboard.
And I agree it will be better interpreted by the public. On the other hand, multiplying numbers is hard, so it’s not as nice for mental arithmetic. And thinking logarithmically puts you in the right frame of mind.
Saying that GPT-4 was trained on x100 more compute than GPT-3 invokes GPT-3 being 100 times better, whereas I think saying it was trained on 2 OOM more compute gives you a better picture of the expected improvement.
I might be wrong here.
In any case, it is still a better choice than doubling times.
My sense is that pure math courses mostly make it easier to do pure math later, and otherwise not make much of a difference. In terms of credentials, I would bet they are mostly equivalent—you can steer your career later through a master or taking jobs in your area of interest.
Statistics seems more robustly useful and transferable no matter what you do later. Statistics degrees also tend to cover the very useful parts of math as well, such as calculus and linear algebra.
Within Epoch there is a months-long debate about how we should report growth rates for certain key quantities such as the amount of compute used for training runs.
I have been an advocate of an unusual choice: orders-of-magnitude per year (abbreviated OOMs/year). Why is that? Let’s look at other popular choices.
Doubling times. This has become the standard in AI forecasting, and its a terrible metric. On the positive, it is an intuitive metric that both policy makers and researchers are familiar with. But it is absolutely horrid to make calculations. For example, if I know that the cost of AI training runs is doubling every 0.6 years, and the FLOP/$ is doubling every 2.5 years, then the FLOP per training run is doubling every years, which is very difficult to solve in your head! [1] [2]
Percent growth. This is a choice often favoured in economics, where eg the growth rate of GDP is often reported as 3%, etc. Unlike doubling times, percent growth composes nicely—you just have to add them up! [3] However, I also find percent changes somewhat prone to confusion. For instance, when I tell people that model size has increased 200% since X years ago, I sometimes have had people misunderstand this as saying that it has increased by a factor of 2.
Ultimately, a very common operation that I find myself doing in my head is “if the effective FLOP used in AI training runs grows at a certain rate, how quickly will we traverse from the scale of current training runs (1e25 FLOP) to a certain threshold (eg 1e30)”. OOMs/year makes this computation easy, even if I need to account for multiple factors such as hardware increases, investment and algorithmic improvements. Eg if these grow respectively as 0.4 OOM/year, 0.1 OOM/year and 0.5 OOM/year, then I know the total effective growth is 1.0 OOM/year, and it will take 5 years to cross that 5 OOM scale gap. And if investment suddenly stopped growing, then I would be able to quickly understand that the pace would be halved, and the gap would then take 10 years to cross.
Sadly OOMs/year is uncommon, and both researchers and policy makers struggle to understand it. I think this is a missed opportunity, and that AI forecasting would be easier to reason about if we moved to it, or at the very least abandoned the very badly behaved doubling time framing.
What do you think? Do you agree we should move past doubling times to a better choice? Which choice would you favour?
I won’t enter into the technical details, but it also has some very unintuitive results when combining the results with uncertainty. Once we had a discussion because some doubling times looked like they had to be wrong. They spanned from days to years! But it turns out that doubling times are very sensitive to noise, which led to our intuitions being wrong.
I’d also argue that Christiano’s operationalization of slow takeoff is a terrible definition, and that a big part of that terribleness stems from doubling times being very unintuitive.
This is because have the useful property that for percent changes close to zero, which links percent growth to a straightforward model of growth such as . But this approximation breaks down for percent changes over 1 (which are often seen in AI forecasting).
I wrote a report on quantum cryptanalysis!
The TL;DR is that it doesn’t seem particularly prioritary, since it is well attended (the NIST has an active post quantum cryptography program) and not that impactful (there are already multiple proposals to update the relevant protocols, and even if these pan out we can rearrange society around them).
I could change my mind if I looked again into the NIST PQC program and learned that they are significantly behind schedule, or if I learned that qauntum computing technology has significantly accelerated since.
I mostly think of alignment as about avoiding deception or catastrophic misgeneralization outside of testing settings.
In general I also believe that AI companies have a massive incentive to align their systems with user intent. You can’t profit if you are dead.
This is a tough question, and I don’t really know the first thing about patents, so you probably should not listen to me.
My not very meditated first instinct is that one would set up a licensing scheme so that you commit to granting a use license to any manufacturer in exchange for a fee, preferrably as a portion of the revenue to incentivize experimentation from entrepreneurs.
Of course, this might dissuade the Tech Transfer office or other entrepreneurs from investing in a start-up if they won’t be guaranteed exclusive use.
I suppose that it would be relevant to investigate how succesful the Tech Transfer office has been in launching startups in the past, what types of permissive licenses they would be amenable to, and whether you know some other entrepreneurs or accelerators who would be keen to commercialize the idea (and what types of licenses they would consider working with).
In any case, good luck with the effort—this could be a very exciting opportunity!
As far as I can tell they are enabled—I see there is a cookie in storage for the intercom for example
Just a quick comment for the devs—I saw the “More posts like this” lateral bar and it felt quite jarring. I liked it way better when it was at the end. Having it randomly in the middle of a post felt distracting and puzzling.
ETA: the Give feedback button does not seem to work either. Also its purpose its unclear (give feedback on the post selection? on the feature?)
A pattern I’ve recently encountered twice in different contexts is that I would recommend a more junior (but with relevant expertise) staff for a talk and I’ve been asked if I could not recommend a more “senior” profile instead.
I thought it was with the best of intentions, but it still rubbed me the wrong way. First, this diminishes the expertise of my staff, who have demonstrated mastery over their subject matter. Second, this systematically denies young people of valuable experience and exposure. You cannot become a “senior” profile without access to these speaking opportunities!
Please give speaking opportunities to less experienced professionals too!
It’s really hard to say.
I am concerned that many projects are being set up in a way that does not follow good governance practices and leaves their members exposed to conflict and unpleasant situations. I would want to encourage people to try to embed themselves in a more formal structure with better resources and HR management. For example, I would be excited about community builders over the world setting a non-profit governance structure, where they are granted explicit rights as workers that encompass mental healthcare and a competent HR department to handle interpersonal conflict. This would arguably be an improvement over the current grant systems, that offers very little job security and care for workers.
Relatedly, I have updated very negatively on the ability of the EA community to solve interpersonal conflict and harassment. This means that I am now recommending aggravated community members to resort to external counsel and support, including psychologists and the police, than to resort to internal resources like the CEA Community Health team.
What is more, these past few months I have understood very viscerally that many people, specially women, are uncomfortable talking publicly about situations that have affected them. This is exacerbated in the case where they are applying for EA jobs or funding. And it hides the magnitude of the community problems.
Before motivating growth in the Spanish-speaking community I would like to see these problems addressed or at least considered. Major obstacles are that 1) there isn’t currently anyone I trust who has availability to take on that project and 2) there isn’t a lot of buy in at the moment for more organized efforts within the community. So I think we will likely just default to the current state of affairs, where we work as an ecosystem of related projects.
I’m pretty into the idea of putting more effort into concrete areas.
I think the biggest reason for is one which is not in your list: it is too easy to bikeshed EA as an abstract concept and fool yourself into thinking that you are doing something good.
Working on object level issues helps build expertise and makes you more cognizant of reality. Tight feedback loops are important to not delude yourself.
I mean rankings like https://www.metaculus.com/rankings/?question_status=resolved&timeframe=3month
I agree that the potential for this exists, and if it was an extended practice it would be concerning. Have you seen people who claim to have a good forecasting record engage with pseudonym exploitation though?
My understanding is that most people who claim this have proof records associated to a single pseudonym user in select platforms (eg Metaculus), which evades the problem you suggest.
Ranking the risks is outside the scope of our work. Interpreting the metaculus questions sounds interesting, though it is not obvious how to disentangle the scenarios that forecasters had in mind. I think the Forecasting Research Institute is doing some related work, surveying forecasters on different risks.
Forum style most definitely
I basically agree with the core message. I’ll go one step further and say that existential risk has unnecessary baggage—as pointed out by Carl Shulman and Elliot Thornley the Global Catastrophic Risk and CBA framing rescues most of the implications without resorting to fraught assumptions about the long term future of humanity.
I don’t think EAs have any particular edge in managing harassment over the police, and I find it troublesome that they have far higher standards for creating safe spaces, specially in situations where the cost is relatively low such as inviting the affected members to separate EAG events while things cool down.
On another point, I don’t think this was Jeff’s intention, but I really dislike the unintentional parallel between an untreated schizophrenic and the CB who asked for the restraining order. I can assure you that this was not the case here, and I think the CB was sound of mind and fully justified in requesting it.
Whether it is easy to get a restraining order, IDK man. It is “easy” in the sense that there is a same-day procedure that only requires interrogating the requester. At least in the jurisdiction where this happened, it is also a gruelling process which requires you to consistently repeat your story to many police officials who keep asking you for details and are looking for contradictions, and takes a full work day to complete.