Superforecaster, former philosophy PhD, Giving What We Can member since 2012. Currently trying to get into AI governance.
David Mathersđ¸
Karma: 6,065
InÂcreasÂing Model Weight SeÂcuÂrity Might InÂcenÂtivize Racing
One reason to think we might not find anything morally valuable that distinct from what we already know about is that our concept of morality is made to fit with the stuff we already know about.
I have a much more positive feelings about EAs than rationalists, and I think this is quite normal for people who came to EA from outside rationalism. I mean, I actually liked the vast majority of rationalists Iâve met a lot-when I worked in a rationalist office in Prague it had a lovely culture-but I think only about .5 of rationalists like EA as an idea, and my suspicion is that âdislikes EAâ amongst rationalists correlates fairly heavily with âhas political views that make me uncomfortableâ.
One thing it might be useful for people to look at here when reflecting on the causes of the failure was how much experience the HR team had of working outside of EA organizations. If the answer is âvery littleâ then maybe bringing in more experienced non-EA pros would help, but if the answer is âa decent amountâ itâs less likely that will prevent future errors on its own.
Might they just be lying about the numbers?
Iâm not sure democracy arguments work that well for military stuff. The people who military actions are going to be deployed against are extremely obvious stakeholders, but they get no input into any feasible âdemocraticâ process that determines what the US military does, and procedural democracy is compatible with the US doing literally anything to non-citizens to advance US interests. Given that, attempting to restrain the US military in ways that are legal and non-deceptive doesnât seem that procedurally dubious to me.
There is absolutely no necessity for anyone to be âmonstersâ in order for the picture painted in Frances post to be accurate and her anger justified though.
All your contributions in this thread seem to be marked by the idea that people would have to be unusually bad in some way to fail to deal well with sexual harassment. But I donât see why we should be particularly confident that the base rate of orgs dealing badly with sexual misconduct, in ways that-*correctly*-look really bad to outsiders when the facts come out is all that low. There have been numerous cases where schools, universities, religious institutions etc. have covered up far worse conduct than Rileyâs: i.e. sexual abuse scandals in the Catholic church as one example among many. Now, of course, we typically donât hear about the cases where sexual harassment or abuse was promptly identified and the perpetrator fired/âimprisoned, unless the perpetrator was independently famous. And there are a lot of people and institutions in the world, so maybe those cases are way more common. But I donât see any particular reason to be confident of that.
More importantly. this isnât even EAâs first scandal where a major org themselves admitted they dealt badly with sexual misconduct: https://ââforum.effectivealtruism.org/ââposts/ââ4CBoJ5jgmGfdMFnAE/ââev-investigation-into-owen-and-community-health In the face of that, I donât see why we should find it particularly implausible that in another case, an EA org dealt badly with sexual misconduct.
But also, people are generally very good at avoiding believing things that are extremely inconvenient for them, without really noticing they are doing it, and without conscious malice. Most of us are much better at doing this in my view than we think we are. When HR originally received Rileyâs doc, they had a choice between 1) trying to start sexual harassment proceedings against him, and 2) overlooking the fact that what he had written about Frances was wholly inappropriate, and possibly malicious, and concentrating on his own complaints. Doing 2) was no doubt made easier by the fact that the doc was apparently not âaboutâ Frances specifically. Now, if they took choice 1), that was a lot of hassle and unpleasantness for them:
A) Potentially having to directly punish Riley. People donât generally like significantly deliberately harming people they know, and have no personal beef with, and will avoid it if they can. This is maybe especially true of the kind of friendly people personish person who wants to work in HR, although that is only a guess on my part.
B) Having to deal with Riley potentially portraying any investigation of him as retaliatory for his complaints. Even a completely baseless accusation of that is likely to be a huge stress for the people on the receiving end of it. This is likely to have been particularly unpleasant with the kind of person who writes a massive doc giving his detailed personal opinions on multiple colleagues, including for at least one person their mental health, sexual victimization and his own romantic feelings for them, because anyone who does something that inappropriate and obsessive canât be trust to act normally during a complaints process. This is true even if Riley is in fact a vulnerable, naive person with no malicious intentions, let alone if he is actually vengeful.
C) Having to either upset Frances by informing her about the doc, and having an excruciatingly awkward conversation with her about it, or alternatively NOT inform her even while disciplining Riley, and admit to themselves that they are concealing from Frances that one of her colleagues has sexually harassed her.
On the other hand, if they take choice 2) and just fail to recognize how inappropriate the stuff Riley said about Frances was, then they can realistically hope that Frances never finds out, and they donât have to upset Riley or Frances, or having any awkward conversations about rape and sexualization of rape victims with anyone or face the consequences of being accused or retaliating against a complainant. They could just hope that Frances never found out about the doc, something that may well have happened if Riley himself had not shown it beyond HR. (Unless the person who informed Frances was themselves in HR anyway). Then they could avoid any stress or hurt feelings. Once the CEO and the COO found out, they also had the same incentives to get this wrong, but with the additional incentive that they could decide to defer to HRâs judgments, since they are the orgs experts on well, HR issues.
I say all this not to excuse anyone at CEA. If this is the psychological dynamics of what happened-which I obviously donât know-thatâs not all that exculpatory. Not dealing with sexual harassment properly because you want to avoid awkwardness/âfuss/âpunishment and youâre in HR or management, is like not running into a burning building as a firefighter because you donât want to get burnt: itâs your job to get past this stuff! But claims that Francesâ account requires an implausible level of malice or incompetence on the part of multiple people seem wrong to me..
Itâs implausible if this is what happened that Riley would have provided a long detailed description of the rape rather than just mentioning itâs occurrence. If Frances was mischaracterizing the document so badly it didnât contain such a description then Zachary could have said that and he hasnât.
Anthropic arenât objecting to killbots as a matter of principle though, they are just saying the tech isnât reliable yet. The stand on surveillance seems principled and I absolutely admire Amodei for risking his business to do the right thing, but letâs avoid deceiving ourselves about what his stance actually is.
Less clearly, sure. Iâm mostly warning about complacency about liberals being safe from error just because you can use liberal ideas to criticize bad things liberals have done, rather than defending communism. Certainly lots of communists have, for example, attacked Stalinism in communist terms.
I donât really understand why liberalism is getting the prefix âclassicalâ here though. The distinction between âclassicalâ and other forms of liberalism, like social liberalism, is more about levels of government support for the poor through the welfare state and just how strong a presumption we should have in favour of market solutions vs government ones, with agreement on secularism, individual human rights, free speech, pluralism, a non-zero sum conception of markets and trade etc. I also think that insofar as âliberalsâ have an unusually good record, this doesnât distinguish âliberalsâ in the narrow sense from other pro-democratic traditions that accept pluralism: i.e. European social democracy on the left, and European Christian democracy, and Anglosphere mainstream conservatism 1965-2015 on the right. If anything classical liberals might have a worse record than many of these groups, because I think classical liberal ideas were used in the 19th century by the British Empire to justify not doing anything about major famines. Of course there is a broad sense of liberal in which all these people are âliberalsâ too, and they may well have been influenced by classical liberalism. But they arenât necessarily on the same side as classical liberals in typical policy debates.
I think there is something to this, but the US didnât just âprop upâ Suharto in the sense of had normal relations of trade and mutual favours even though he did bad things. (That indeed may well be the right attitude to many bad governments, and ones that many lefitsts might demand the US to take to bad left-wing governments, yes.) They helped install him, a process which was incredibly bloody and violent, even apart from the long-term effects of his rule: https://ââen.wikipedia.org/ââwiki/ââIndonesian_mass_killings_of_1965%E2%80%9366
Remember also that the same people are not necessarily making all of these arguments. Relatively few radical leftists saying the first two things are also making a huge moral deal about the US failing to help Ukraine, I think. Even if they are strongly against the Russian invasion. Itâs mostly liberals who are saying the 3rd one.
Communism is a âreason-basedâ ideology, at least originally, in that it sees itself as secular and scientific and dispassionate and based on hard economics, rather than tradition or God. I mean, yes, Marxists tend to be more keen on evoking sociological explanations for peopleâs beliefs than liberals are, but even Marxists usually believe social science is possible and even liberals admit peopleâs beliefs are distorted by bias all the time, so the difference is one of emphasis rather than fundamental commitment I think.
This isnât a defence of communism particularly. The mere fact that people claim that something is the output of reason and science doesnât mean it actually is. That goes for liberalism too.
âClassical liberalism provides the intellectual resources to condemn the Jakarta killings. â
Communism probably also provides intellectual resources that would enable you to condemn most of the many very bad things communists have done, but that doesnât mean that those outcomes arenât relevant to assessing how good an idea communism is in practice.
Not that you said otherwise, and I am a liberal, not a communist. But I do think sometimes liberals can be a bit too quick to conclude that all crimes of liberal regimes having nothing distinctive to do with liberalism, while presuming that communist and fascist and theocratic crimes are inherent products of communism/âfascism/âtheocracy. (I have less than zero time for fascism or theocracy, to be clear.)
The report has many authors, some of whom maybe much less concerned or think the whole thing is silly. I never claimed that Bengio and Hintonâs views were a consensus, and in any case, I was citing their views as evidence for taking the idea that AGI may arrive soon seriously, not their views on how risky AI is. Iâm pretty sure Iâve seen them give relatively short time-lines when speaking individually, but I guess I could be misremembering. For what itâs worth Yann LeCunn seems to think 10 years is about right, and Gary Marcus seems to think a guess of 10-20 years is reasonable: https://ââhelentoner.substack.com/ââp/ââlong-timelines-to-advanced-ai-have
I guess Iâm just slightly confused about what economists actually think here since Iâd always thought they took the idea that markets and investors were mostly quite efficient most of the time fairly seriously.
I donât know if/âhow much EA money should go to AI safety either. EAs are trying to find the single best thing, and itâs very hard to know what that is, and many worthwhile things will fail that bar. Maybe David Thorstad is right, and small X-risk reductions have relatively low value because another X-risk will get us in the next few centuries anyway*. What I do think is that society as a whole spending some resources caring about the risk of AGI arriving in the next ten years is likely optimal, and that itâs not more silly to do so than to do many other obviously good things. I donât actually give to AI safety myself, and I only work on AI-related stuff-forecasting etc., Iâm not a techy person-because itâs what people are prepared to pay more for, and people being prepared to pay me to work on near-termist causes is less common, though it does happen. I myself give to animal welfare, not AI safety.
If you really believe that everyone putting money into Open AI etc. will only see returns if they achieve AGI that seems to me to be a point in favour of âthere is a non-negligible risk of AGI in the next 10 yearsâ. I donât believe that, but if I did I that alone would significantly raise the chance I give to AGI within the next 10 years. But yes, they have some incentive to lie here, or to lie to themselves, obviously. Nonetheless, I donât think that means their opinion should get zero weight. For it to actually have been some amazing strategy for them to talk up the chances of AGI, *because it attracted cash* youâd have to believe they can fool outsiders with serious money on the line, and that this will be profitable for them in the long term, rather than crashing and burning when AGI does not arrive. I donât think that is wildly unlikely or anything, indeed, I think it is somewhat plausible-though my guess is Anthropic in particular believe their own hype. But it does require a fairly high amount of foolishness on behalf of other quite serious actors. Iâm much more sure of âraising large amounts of money for stuff that obviously wonât work is relatively hardâ than I am of any argument about how far we are from AGI that looks at the direct evidence, since the latter sort of arguments are very hard to evaluate. Iâd feel very differently here if we were arguing about 50% chance of AI in ten years, or even 10% chance. Itâs common for people to invest in things that probably wonât work but have a high pay-off if they do. But what your saying is that Richard is wrong for thinking there is a non-negligible risk, because the chance is significantly under 1%. I doubt there are many takers for like â1 in 1000â chance of a big pay-off.
It is of course not THAT unlikely that they are fooling the serious money: serious investors make mistakes and even the stock market does. Nonetheless, being able to attract serious investment that is genuinely only investing because they think youâll achieve X, whilst simultaneously being under huge media attention and scrutiny is a credible signal that youâll eventually achieve X.
I donât think the argument Iâve just given is all that definitive, because they have other incentives to hype, like attracting top researchers (who I think it is probably eaiser to fool, because if they are fooled about AGI working at a big lab was probably good for them anyway; quite different from what happens to people funding the labs who are fooled who just lose money.) So itâs possible that the people pouring serious money in donât take any of the AGI stuff seriously. Nonetheless, I trust âserious organisations with technical prowess seem to be trying to do thisâ as a signal to take something minimally seriously, even if they have some incentive to lie.
Similarly, if you really think Microsoft and Google have taken decisions that will crash their stock if AGI doesnât arrive, I think a similar argument applies: Are you really sure youâre better at evaluating whether there is a non-negligible chance that a tech will be achieved by the tech industry than Microsoft and Google? Eventually, if AGI is not arriving from the huge training runs that are being planned in the near future, people will notice, and Microsoft and Google donât want to lose money in 5 years from now either. Again, itâs not THAT implausible that they are mistaken, mistakes happen. But you arenât arguing that there probably wonât be AGI in ten years-a claim I actually strongly agree with!-but rather that Richard was way out in saying that itâs a tail risk we should take seriously given how important it would be.
Slower progress on one thing than another does not mean no progress on the slower thing.
âdespite those benchmarks not really being related to AGI in any way.â This is your judgment, but clearly it is not the judgment of some of the worldâs leading scientific experts in the area. (Though there may well be other experts who agree with you.)
*Actually Thorstadâs opinion is more complicated than that, he says that this is true conditional on X-risk currently being non-negligible, but he doesnât himself endorse the view that it is currently non-negligible as far as I can tell.
METR has an official internal view on what time horizons correspond to âtakeover not ruled outâ?
Yeah, I am inclined to agree-for what my opinion is worth which on this topic is probably not that much-that there will be many things AIs canât do even once they have a METR 80% time-horizon of say 2 days. But I am less sure of that than I am of the meta-level point about this being an important crux.
Sure, but I I wasnât really thinking of people on LessWrong, but rather of the fact that at least some relevant experts outside of the LW milieu seem worried and/âor think that AGI is not THAT far. I.e. Hinton, Bengio, Stuart Russell (for danger) and even people often quoted as skeptical experts* like Gary Marcus or Yann LeCunn often give back of the envelope timelines of 20 years, which is not actually THAT long. Furthermore I actually do think the predictions of relatively near term AGI by Anthropic and the fact that DeepMind and OpenAI have building AGI as a goal to carry some weight here. Please donât misunderstand me, I am not saying that these orgs are trustworthy exactly: I expect them to lie in their own interests to some degree, including about how fast their models will get better, and also to genuinely overestimate how fast they will make progress. Nonetheless they are somewhat credible in the sense that a) they are at the absolute cutting edge of the science here and have made some striking advancements, and b) they have some incentive not to overpromise so much that no one ever believes anything they say ever again, and c) they are convincing enough to outsiders with money that they keep throwing large sums of money at them, which suggests those outsiders at least expect reasonably rapid advancement, whether or not they expect AGI itself, and which is also evidence that these are serious organizations.
Iâd also say that near-term AGI is somewhat disanalogous to Hinduism, ESP, Christianity, crystal healing etc. in that all these things are actually in conflict with a basic scientific worldview fairly directly, in that they describe things that would plausibly violate known laws of physics, or are clearly supernatural in a fairly straightforward sense. Thatâs not true of near-term AGI.
Having said that I certainly agree that it is not completely obvious that there is enough real expertise behind predictions of near-term AGI to treat them with deference. My personal judgment is that there is, but once we get away from obvious edge cases like textbook hard science on the one hand and âexpertsâ in plainly supernatural things on the other, it gets hard to tell how much deference people deserve.
Thereâs also an issue of priors here of course: I donât think âAGI will be developed in the next 100 yearsâ is an âextraordinaryâ claim in the same sense as supernatural claims, or even just something unlikely but possible like âScotland will win the next football world cupâ. We know it is possible in principle, and that technology can advance quickly over a timespan of decades-just compare where flight was in 1900 to 1960-and that trillions of dollars are going to be spent advancing AI in the near term, and while Mooreâs law is breaking down, we havenât actually hit theoretical in principle limits on how good chips can be, and that more money and labour is currently going into making advanced AI than ever before. If we say thereâs a 25% chance of it being developed in the next hundred years, an even divide per decade of that would say 2.5% chance of it arriving next decade. Even if we cut that 5x for the next decade, that would give a 0.5% chance which I think is worth worrying about given how dramatic the consequence of AGI would be. (Of course, you personally have lots of arguments against it being near, but I am just talking about what itâs reasonable to expect from broad features of the current situation before we get into the details.) But of course, forecasting technology 100 years out is extraordinarily hard. In general because forecasting is hard beyond the next few years, so I guess maybe 25% is way too high (although the uncertainty cuts both ways.)
I get that the worry here is that people can just say any possible technology might happen soon, so if it was very consequential we should worry about it now. But my response is just that if itâs a technology that several of the worldâs largest or, fastest growing or most innovative companies claim or hint to be building it, and a Nobel winning scientist in the area in question agree that they very well might be, probably this is right, whereas if no one serious is working towards a technology, a higher degree of skepticism is probably warranted. (Presumption could be overcome if almost all scientists with relevant expertise think that Bengio and Hinton are complete cranks on this, but I havenât seen strong evidence of that.)
*In fairness, my guess is that they are actually more bullish on AGI than many people in machine learning, but that is only a guess.
What about the risk we spread wild animal suffering to other planets?