Director of Epoch, an organization investigating the future of Artificial Intelligence.
Currently working on:
Macroeconomic models of AI takeoff
Trends in Artificial Intelligence
Forecasting cumulative records
Improving forecast aggregation
I am also one of the coordinators of Riesgos Catastróficos Globales, a Spanish-speaking network of experts working on Global Catastrophic Risks.
I also run Connectome Art, an online art gallery where I host art I made using AI.
I am not sure what this footnote means. The “cost of training per 1m tokens” is a very weird unit to talk about, since it depends on the model size and the GPU efficiency. I strongly suspect you meant to write something else and got mixed up.
What do you mean exactly? None of these maps have a discontinuity at .5
I find these analogies more reassuring than worrying TBH
My point is that our best models of economic growth and innovation (such as the semi-endogeneous growth model that Paul Romer won the Nobel prize for) straightforwardly predict hyperbolic growth under the assumptions that AI can substitute for most economically useful tasks and that AI labor is accumulable (in the technical sense that you can translate a economic output into more AI workers). This is even though these models assume strong diminishing returns to innovation, in the vein of “ideas are getting harder to find”.
Furthermore, even if you weaken the assumptions of these models (for example assuming that AI won’t participate in scientific innovation, or that not every task can be automated) you still can get pretty intense accelerated growth (up to x10 greater than today’s frontier economies).
Accelerating growth has been the norm for most of human history, and growth rates of 10%/year or greater have been historically observed in, e.g. 2000s China, so I don’t think this is an unreasonable prediction to hold.
I’ve only read the summary, but my quick sense is that Thorstad is conflating two different versions of the singularity thesis (fast takeoff vs slow but still hyperbolic takeoff), and that these arguments fail to engage with the relevant considerations.
Particularly, Erdil and Besiroglu (2023) show how hyperbolic growth (and thus a “singularity”, though I dislike that terminology) can arise even when there are strong diminishing returns to innovation and sublinear growth with respect to innovation.
(speculating) The key property you are looking for IMO is to which degree people are looking at different information when making forecasts. Models that parcel reality into neat little mutually exclusive packages are more amenable , while forecasts that obscurely aggregate information from independent sources will work better with geomeans.
In any case, this has little bearing on aggregating welfare IMO. You may want to check out geometric rationality as an account that lends itself more to using geometric aggregation of welfare.
Interesting case. I can see the intuitive case for the median.
I think the mean is more appropriate—in this case, what this is telling you is that your uncertainty is dominated by the possibility of a fat tail, and the priority is ruling it out.
I’d still report both for completeness sake, and to illustrate the low resilience of the guess.
Very much enjoyed the posts btw
Amazing achievements Mel! With your support, the group is doing a fantastic job, and I am excited about its direction.
>his has meant that, currently, our wider community lacks a clear direction, so it’s been harder to share resources among sub-groups and to feel part of a bigger community striving for a common goal.
I feel similarly! At the time being, it feels that our community has fragmented into many organizations and initiatives: Ayuda Efectiva, Riesgos Catastróficos Globales, Carreras con Impacto, EAGx LatAm, EA Barcelona. I would be keen on developing better the relationships between these pieces; for example I was enthused to have Guillem from RCG present in EA Barcelona. Would be cool to have more chats and find more links!
I have so many axes of disagreement that is hard to figure out which one is most relevant. I guess let’s go one by one.
Me: “What do you mean when you say AIs might be unaligned with human values?”
I would say that pretty much every agent other than me (and probably me in different times and moods) are “misaligned” with me, in the sense that I would not like a world where they get to dictate everything that happens without consulting me in any way.
This is a quibble because in fact I think if many people were put in such a position they would try asking others what they want and try to make it happen.
Consider a random retirement home. Compared to the rest of the world, it has basically no power. If the rest of humanity decided to destroy or loot the retirement home, there would be virtually no serious opposition.
This hypothetical assumes too much, because people outside care about the lovely people in the retirement home, and they represent their interests. The question is, will some future AIs with relevance and power care for humans, as humans become obsolete?
I think this is relevant, because in the current world there is a lot of variety. There are people who care about retirement homes and people who don’t. The people who care about retirement homes work hard toale sure retirement homes are well cared for.
But we could imagine a future world where the AI that pulls ahead of the pack is very indifferent about humans, while the AI that cares about humans falls behind; perhaps this is because caring about humans puts you at a disadvantage (if you are not willing to squish humans in your territory your space to build servers gets reduced or something; I think this is unlikely but possible) and/or because there is a winner-take-all mechanism and the first AI systems that gets there coincidentally don’t care about humans (unlikely but possible). Then we would be without representation and in possibly quite a sucky situation.
I’m asking why it matters morally. Why should I care if a human takes my place after I die compared to an AI?
Stop that train, I do not want to be replaced by either human or AI. I want to be in the future and have relevance, or at least be empowered through agents that represent my interests.
I also want my fellow humans to be there, if they want to, and have their own interests be represented.
Humans seem to get their moral values from cultural learning and emulation, which seems broadly similar to the way that AIs will get their moral values.
I don’t think AIs learn in a similar way to humans, and future AI might learn in a even more dissimilar way. The argument I would find more persuasive is pointing out that humans learn in different ways to one another, from very different data and situations, and yet end with similar values that include caring for one another. That I find suggestive, though it’s hard to be confident.
Our team at Epoch recently updated the org’s website.
I’d be curious to receive feedback if anyone has any!
What do you like about the design? What do you dislike?
How can we make it more useful for you?
Note that the AMF example does not quite work, because if each net has a 0.3% chance of preventing death, and all are independent, then with 330M nets you are >99% sure of saving at least ~988k people.
An encouraging update: thanks to the generous support of donors, we have raised $95k in funds to support our activities for six more months. During this time, we plan to 1) engage with the EU trilogue on the regulation of foundation models during the Spanish presidency of the EU council, 2) continue our engagement with policy markers in Argentina and 3) release a report on global risk management in latin america.
We nevertheless remain funding constrained. With more funding we would be able to launch projects such as:
A report on prioritizing and forecasting global catastrophic risk that could be used by stakeholders such as the regional UNNDR office or CEPAL.
Expand our previous work on mapping BSL-3 and BSL-4 labs in Latin America.
Develop a theory of change for how latin america could contribute through multilateral negotiations to the management of AI risk, covering a study of the historic role of latam countries in the nuclear risk treaties and the management of other global risks.
Further our relationship with Uruguay and other countries in the MERCOSUR to invite them to follow Argentina in the formulation of emergency plans for food security in case of nuclear winter.
And other priority projects to improve the management of global catastophic risk in Spanish-Speaking countries.
Each of these projects could cost between $30k and $80k to develop. You can support us with a donation to help us develop these projects. You may also reach out to me through a PM if you are considering donating and want more information.
I have grips with the methodology of the article, but I don’t think highlighting the geometric mean of odds over the mean of probabilities is a major fault. The core problem is assuming independence over the predictions at each stage. The right move would have been to aggregate the total P(doom) of each forecaster using geo mean of odds (not that I think that asking random people and aggregating their beliefs like this is particularly strong evidence).
The intuition pump that if someone assigns a zero percent chance then the geomean aggregate breaks is flawed:
There is an equally compelling pump the other way around: the arithmetic mean of probabilities defers unduly to people assigning a high chance. A single dissenter between 10 experts can bound the lower bound of the probability to their preferred up to a factor of 10.
And surely if anyone is assigning a zero percent chance to something, you can safely assume they are not taking the situation seriously and ignore them.
In ultimate instance, we can theorize all we want, but as a matter of fact the best performance when predicting complex events is achieved when taking the geometric mean of odds, both in terms of logloss and brier scores. Without more compelling evidence or a very clear theoretical reason that distinguishes between the contexts, it seems weird to argue that we should treat AI risk differently.
And if you are still worried about dissenters skewing the predictions, one common strategy is to winsorize, by clipping the predictions among the 5% and 95% percentile for example.
Yeah in hindisght that is probably about right.
It’d be interesting to look at some of these high profile journalists, and see if they are well supported to do impactful journalism or if they have to spend a lot of time on chasing trends to afford working on IJ pieces.
TL;DR
Sentience appear in many animals, indicating it might have a fundamental purpose for cognition. Advanced AI, specially if trained on data and environments similar to humans, will then likely be conscious
Restrictions to advanced AI would likely delay technological progress and potentially require a state of surveillance. A moratorium might also shift society towards a culture that is more cautious towards expanding life.
I think what is missing for this argument to go through is arguing that the costs in 2 are higher than the cost of mistreated Artificial Sentience.