âwidely (and imo falsely) believed that the openai coup was for EA reasonsâ
False why?
David Mathersđ¸
Karma: 4,924
Everything you say is correct I think, but I think in more normal circles, pointing out the inconsistency between someoneâs wedding page and their corporate PR bullshit would seem a bit weird and obsessive and mean. I donât find it so, but I think ordinary people would get a bad vibe from it.
Because you are so strongly pushing a particular political perspective on twitter-tech right=good roughly, I worry that your bounties are mostly just you paying people to say things you already believe about those topics. Insofar as you mean to persuade people on the left/âcentre of the community to change their views on these topics, maybe it would be better to do something like make the bounties conditional on people who disagree with your takes finding the investigations move their views in your direction.
I also find the use of the phrase âsuch controversial criminal justice policiesâ a bit rhetorical dark artsy and mildly incompatible with your calls for high intellectual integrity. It implies that a strong reason to be suspicious of Open Philâs actions has been given. But you donât really think the mere fact that a political intervention on an emotive, polarized topic is controversial is actually particularly informative about it. Everything on that sort of topic is controversial, including the negation of the Open Phil view on the US incarceration rate. The phrase would be ok if you were taking a very general view that we should be agnostic all political issues where smart, informed people disagree. But youâre not doing that, you take lots of political stances in the piece: de-regulatory libertarianism, the claim that environmentalism has been net negative and Dominic Cummings can all accurately be described as âhighly controversialâ.
Maybe I am making a mountain out of a molehill here. But I feel like rationalists themselves often catastrophise fairly minor slips into dark arts like this as strong evidence that someone lacks integrity. (I wouldnât say anything as strong as that myself; everyone does this kind of thing sometimes.) And I feel like if the NYT referred to AI safety as âtied to the controversial rationalist communityâ or to âhighly controversial blogger Scott Alexanderâ you and other rationalists would be fairly unimpressed.
More substantively (maybe I should have started with this as it is a more important point), I think it is extremely easy to imagine the left/âDemocrat wing of AI safety becoming concerned with AI concentrating power, if it hasnât already. The entire techlash anti âsurveillanceâ capitalism, âthe algorithms push extremismâ thing from left-leaning tech critics is ostensibly at least about the fact that a very small number of very big companies have acquired massive amounts of unaccountable power to shape political and economic outcomes. More generally, the American left has, I keep reading, been on a big anti-trust kick recently. The explicit point of anti-trust is to break up concentrations of power. (Regardless of whether you think it actually does that, that is how its proponents perceive it. They also tend to see it as âpro-marketâ; remember that Warren used to be a libertarian Republican before she was on the left.) In fact, Lina Khanâs desire to do anti-trust stuff to big tech firms was probably one cause of Silicon Valleyâs rightward shift.
It is true that most people with these sort of views are currently very hostile to even the left-wing of AI safety, but lack of concern about X-risk from AI isnât the same thing as lack of concern about AI concentrating power. And eventually the power of AI will be so obvious that even these people have to concede that it is not just fancy autocorrect.
It is not true that all people with these sort of concerns only care private power and not the state either. Dislike of Palantirâs nat sec ties is a big theme for a lot of these people, and many of them donât like the nat sec-y bits of the state very much either. Also a relatively prominent part of the left-wing critique of DOGE is the idea that itâs the beginning of an attempt by Elon to seize personal effective control of large parts of the US federal bureaucracy, by seizing the boring bits of the bureaucracy that actually move money around. In my view people are correct to be skeptical that Musk will ultimately choose decentralising power over accumulating it for himself.
Now strictly speaking none of this is inconsistent with your claim that the left-wing of AI safety lacks concern about concentration of power, since virtually none of these anti-tech people are safetyists. But I think it still matters for predicting how much the left wing of safety will actually concentrate power, because future co-operation between them and the safetyists against the tech right and the big AI companies is a distinct possibility.
Section 4 is completely over my head I have to confess.
Edit: But the abstract gives me what I wanted to know :) : âTo quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rateâ
I donât know of anything better right now.
Itâs actually the majority view amongst academics who directly study the issue. (Iâm probably an anti-realist though). https://ââsurvey2020.philpeople.org/ââsurvey/ââresults/ââ486
Thanks, that is reassuring.
I donât quite get what that means. Do they really take exactly the same amount of time on all tasks for which they have the same success rate? Sorry, maybe I am being annoying here and this is all well-explained in the linked post. But I am trying to figure out how much this is creating the illusion that progress on it means a model will be able to handle all tasks that it takes normal human workers about that amount of time to do, when it really means something quite different.
âI donât think that, for a given person, existing can be better or worse than not existing. â
Presumably even given this, you wouldnât create a person who would spending their entire life in terrible agony, begging for death. If that can be a bad thing to do even though existing canât be worse than not existing, then why canât it be a good thing to create happy people, even though existing canât be better than not existing?
Is the point when models hit a length of time on the x-axis of the graph meant to represent the point where models can do all tasks of that length that a normal knowledge worker could perform on a computer? The vast majority of knowledge worker tasks of that length? At least one task of that length? Some particular important subset of tasks of that length?
Morally, I am impressed that you are doing an in many ways socially awkward and uncomfortable thing because you think it is right.
BUT
I strongly object to you citing the Metaculus AGI question as significant evidence of AGI by 2030. I do not think that when people forecast that question, they are necessarily forecasting when AGI, as commonly understood or in the sense thatâs directly relevant to X-risk will arrive. Yes the title of the question mentions AGI. But if you look at the resolution criteria, all an AI model has to in order to resolve the question âyesâ is pass a couple of benchmarks involving coding and general knowledge, put together a complicated model car, and imitate. None of that constitutes being AGI in the sense of âcan replace any human knowledge worker in any jobâ. For one thing, it doesnât involve any task that is carried out over a time span of days or weeks, but we know that memory and coherence over long time scales is something current models seem to be relatively bad at, compared to passing exam-style benchmarks. It also doesnât include any component that tests the ability of models to learn new tasks at human-like speed, which again, seems to be an issue with current models. Now, maybe despite all this, itâs actually the case that any model that can pass the benchmark will in fact be AGI in the sense of âcan permanently replace almost any human knowledge workerâ, or at least will obviously only be a 1-2 years of normal research progress away from that. But that is a highly substantive assumption in my view.
I know this is only one piece of evidence you cite, and maybe it isnât actually a significant driver of your timelines, but I still think it should have been left out.
Yes. (Though Iâm not saying this will happen, just that it could, and that is more significant than a short delay.)
The more task lengths the 80% threshold has to run through before it gets to task length weâd regard as AGI complete though, the more different the tasks at the end of the sequence are from the beginning, and therefore the more likely it is that the doubling trend will break down somewhere along the length of the sequence. That seems to me like the main significance of titotalâs point, not the time gained if we just assume the current 80% doubling trend will continue right to the end of the line. Plausibly 30 seconds to minute long tasks are more different from weeks long tasks than 15 minute tasks are.
The total view is not the only view on which future good lives starting has moral value. You can also think that if you believe in (amongst other things):
-Maximizing average utility across all people who ever live, in which case future people coming into existence is good if their level of well-being is above the mean level of well-being of the people before them.
-A view on which adding happy lives gets less and less valuable the more happy people have lived, but never reaches zero. (Possibly helpful with avoiding the repugnant conclusion.)
-A view like the previous one on which both the total amount of utility and how fairly it is distributed matter, so that more utility is always in itself better, and so adding happy people is always intrinsically good in itself, but a population with less total utility but a fairer distribution of utility can sometimes be better than a population with more utility, less fairly distributed.
This isnât just nitpicking: the total view is extreme in various ways that the mere idea that happy people coming into existence is good is not.
Also, even if you reject the view that creating happy people is intrinsically valuable, you might want to ensure there are happy people in the future just to satisfy the preferences of current people, most of whom probably have at least some desire for happy descendants of at least one of their family/âculture/âhumanity as whole, although it is true that this wonât get you the view that preventing extinction is astronomically valuable.
I genuinely have no idea.
One thing to worry about here is deception. All things being equal, itâs general a reason against doing something that is deceives people, and trying to ease people in gently can be a special case of that because itâs deceiving them about the beliefs you hold. It also might stop you yourself getting useful information, since if you only introduce your more and unusual and radical commitments to people whoâve already been convinced by your more mainstream ones, you are missing out on criticism of the radical commitments from the people most opposed to them.
This sort of thing has been an issue with EA historically: people have accused EA leaders (fairly or not) of leading with their beliefs about global poverty to give the impressiont that that is what they (the leader) and EA are really all about, when actually what the leader really cares about is a bunch of much more controversial things: AI safety, longtermism or niche animal welfare stuff like shrimp welfare.
Iâm not saying that this means no one should ever introduce people to radical ideas gently, I think it can be reasonable, just that this is worth keeping in mind.
What makes it leftist? If anything my immediate reaction is that abundance is in some sense right-coded in that itâs about unleashing markets. Maybe more British right and pre-Trump American right than current right.
(Mostly a nitpick, as I donât want Open Phil doing centrist or libertarian or centre-right coded things rather than whatâs most effective either, and I think strongly upvoted your comment.)
Why use automation as your reference class for AI and not various other (human) groups of intelligent agents though? And if you use the later, co-operation is common historically but so are war and imperialism.
I donât think the issue here is actually about whether all science grants should go only to actual scientific work. Suppose that a small amount of the grant had been spent on getting children interested in science in a completely non-woke way that had nothing to do with race or gender. I highly doubt that either the administration or you regard that as automatically and obviously horrendously inappropriate. The objection is to stuff targeted at women and minorities in particular, not to a non-zero amount of science spending being used to get kids interested in science. Describing it as just being about spending science grants only on science is just a disingenuous way of making the adminâs position sound more commonsense and apolitical than it actually is.
No, I donât move in corporate circles.