It is most apparent in this piece of the review:
He also points out that Tanzanian natives using their traditional farming practices were more productive than European colonists using scientific farming. I’ve had to listen to so many people talk about how “we must respect native people’s different ways of knowing” and “native agriculturalists have a profound respect for the earth that goes beyond logocentric Western ideals” and nobody had ever bothered to tell me before that they actually produced more crops per acre, at least some of the time. That would have put all of the other stuff in a pretty different light.
He remains focused on the expected crops per acre, even though every case study in the book illustrates that such a single variable doesn’t encompass the multitude of uses that the acre in question has. I don’t think I could describe it better than Reddit user u/TheHiveMindSpeaketh does:
The point of the book is not to point and laugh at the technocrats who failed to squeeze the most X out of Y because they didn’t listen to the noble savages. The point is that ‘how do we squeeze the most X out of Y’ is a bad way to position yourself in relation to your surroundings. The point is that technocrats often succeed in squeezing more X out of Y over a relevant period of time via their techniques, but that treating a forest like a timber-maximizer is already missing the [..] point because a forest is also a home for woodland creatures, and a source for medicinal herbs and fruits and berries, and a nice place to take a hike and stare at the stars. The point is that the mistake was not made at the level of what was implemented, the mistake was made at the level of what was valued, and the implementation mistake was an inevitable downstream consequence of that. The point is that even if traditional Tanzanian farming methods didn’t produce more crops per acre, they might still be preferable, because they are more sustainable or less time-intensive or etc, but that these benefits become unintelligible to the technocrat who has already committed to a value system where land is only judged by its yield per acre.
I personally think this is an important question for EA’s to grapple with: can we reason abstractly about doing good without this abstraction causing mistakes at the level of what to value. Scott’s technocrats surely did not think they were making that mistake, but they were. If we believe that we are somehow different, that is kind of arrogant.
For a different take on the consequences of being “rational”, I would highly recommend James C. Scott’s book Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed. The book summary of SSC is pretty good, but when he gives his opinion on the book he seems to have missed the point of the book entirely.
Thank you for your response.
Yes, that is what I meant. If you could convince me that AGI Safety were solvable with increased funding, and only solvable with increased funding, that would go a long way in convincing me of it being an effective cause.
In response to your question of giving up: If AGI were a long way off from being built, then helping others now is still a useful thing to do, no matter if either of the scenarios you describe were to happen. Sure, extinction would be bad, but at least from some person-affecting viewpoints I’d say extinction is not worse than existing animal agriculture.
Let me try to rephrase this part, as I consider it to be the main part of my argument and it doesn’t look like I managed to convey what I intended to:
AI Safety would be a worthy cause if a superintelligence were powerful and dangerous enough to be an issue but not so powerful and dangerous as to be uncontrollable.
The most popular cause evaluation framework within EA seems to be Importance/Neglectedness/Tractability. AI Safety enthusiasts tell a convincing story on importance and neglectedness being good and make an effort at arguing that tractability is as well.
But here is the thing: all arguments given in favour of AI being risky (to establish importance) can be rephrased as arguments against tractability. Similarly for neglectedness.
I’ll illustrate this with a caricature, but it takes little effort to transfer this line of thought to the real arguments being made. Let’s say the pro-AIS argument is “AGI will become infinitely smart, so it can out-think all humans and avoid all our security measures. Hence AGI is likely to escape any restrictions we put on it, so it will be able to tile the universe with paperclips if it wants to”. Obviously, if it can out smart any security measure, then no sufficient security exists, AI Safety research will never lead to anything and the problem is intractable.
AI Safety is only effective if you can simultaneously argue for each of importance/neglectedness/tractability without detracting from the others. Moreover, your arguments have to address the exact same scenarios. It is not enough for AIS to be important with 50% probability and tractable with 50% probability, these two properties have be likely to hold simultaneously. A coin flip has 50% probability of heads and 50% probability of tails, but they will never happen at the same time.
AI Safety can only be an effective cause (on the margin) if solving it is possible (tractability) but not trivial (importance/neglectedness). I think this is a narrow window to hit, and current arguments are all way off-target.
Same for the unconference, should be this link.
Thank you for this nice summary of the argument in favour of AI Safety as a cause. I am not convinced, but I appreciate your write-up. As you asked for counterarguments, I’ll try to describe some of my gripes with the AI Safety field. Some have to do with how there seems to be little awareness of results in adjacent fields, making me doubt if any of it would stand up to scrutiny from people more knowledgeable in those areas. There are also a number of issues I have with the argument itself.
Where’s does it end? Well, eventually, at the theoretical limits of computation. These theoretical limits are very, very high—without even getting close to the limit, a 10kg computer could do more computation every hour than 10 billion human brains could do in a million years. (And a superintelligence wouldn’t be limited to just 10kg). At that point, we are talking about something that can essentially do anything that is allowed by the laws of physics—something so incredibly smart it’s comparable to a civilisation millions of years ahead of us.
The theoretical limits of computation are lower bounds, we don’t know if it is possible to achieve them for any kind of computation, let alone for general computation. Moreover, having a lot of computational power probably doesn’t mean that you can calculate everything. A lot of real-world problems are hard to approximate in a way that adding more computational power doesn’t meaningfully help you. For example, computing approximate Nash-equilibria or finding good lay-outs for microchip design. It is not clear that having a lot of computing power translates into relevant superior capabilities.
We don’t yet know how to program any high-level human concept like morality, love, or happiness—the difficulty is in nailing down the concept to the kind of mathematical language a computer can understand before it becomes superintelligent.
There is a growing literature on making algorithms fair, accountable and transparent. This is a collaborative effort between researchers in computer science, law and many other fields. There are so many similarities between this and the professed goals of the AI Safety community that it is strange that no cross-fertilization is happening.
The problem is Instrumental Convergence.
You can’t just ask the AI to “be good”, because the whole problem is getting the AI to do what you mean instead of what you ask. But what if you asked the AI to “make itself smart”? On the one hand, instrumental convergence implies that the AI should make itself smart. On the other hand, the AI will misunderstand what you mean, hence not making itself smart. Can you point the way out of this seeming contradiction?
So a superintelligence could be super powerful and super dangerous if and when we are able build it.
AI Safety would be a worthy cause if a superintelligence were powerful and dangerous enough to be an issue but not so powerful and dangerous as to be uncontrollable. A solution has to be necessary, but it also has to exist. Thus, there is a tension between scale and tractability here. Both Bostrom and Yudkowsky only ever address one thing at a time, never acknowledging this tension.
If it takes off slow enough, we’ll have time to figure out how to make it safe after we create the first superintelligence, which would be very handy indeed. Unfortunately, it turns out nobody agrees on that either.
Most estimates on take-off speed start counting from the point that the AI is superintelligent. Why wait until then? A computer can be reset, so if you had a primitive AGI specimen you’d have unlimited tries to spot problems and make it behave.
I’d say that a 0.0001% chance of a superintelligence catastrophe is a huge over-estimate. Hence, AI Safety would be an ineffective cause area if you hold a person-affecting view. If you don’t, then at least this opens the way for the kind of counterarguments used against Pascal’s Mugging.
ahead of their time, in the sense that if they hadn’t been made by their particular discoverer, they wouldn’t have been found for a long time afterwards?
This definition is surprisingly weak, and in fact includes some scientific results that were way past their time. One striking example is Morley’s trisector theorem, which is an elegant fact in Euclidean 2d geometry which had been overlooked for 2000 years. If not for Morley, this fact might have remained unknown for millennia longer.
1. The mechanics of cryptographic attack and defense are more complicated that you might imagine. This is because (a) there is a huge difference between the attack capabilities of nations versus those of other maligne actors. Even if the NSA, with its highly-skilled staff and big budget, is able to crack your everyday TLS traffic, doesn’t mean that your bank transactions aren’t safe against petty internet criminals. And (b) state secrets typically need to be safe against computers of 20+ years in the future, as you don’t want enemy states to capture your traffic now and decrypt it as soon as slightly better hardware is available.
2. NIST is running a project at this moment to standardize a post-quantum cryptographical protocol. Cryptographers from many countries in the world are collaborating on this. The tentative timeline lists the completion of the draft standards in 2022-2024.
Hence, experts worldwide estimate that strong quantum computers will not be deployed even by intelligence agencies until well into the 2030s (e: 40′s). Consumer targets will stay safe for even longer than that.
I remember EA-aligned vegan Youtuber Unnatural Vegan making a video about this argument last week in response to a recent Vox article. She argues that the meat industry is very elastic, but I don’t think she cites any specific sources. As she normally does tend to do that, I suspect those numbers are hard to come by.
3b justifies 3a, as well as that I have a much easier time paying attention to the talk. In video, there is too much temptation to play at 1.5x speed and aim for an approximate understanding. Though I guess watching the video together with other people also helps.
As for 3b, in my experience asking questions adds a lot of value, both for yourself as well as for other audience members. The fact that you have a question is a strong indication that the question is good and that other people are wondering the same thing.
I like your list. Here is my conference advice, contradicting some of yours, based mostly on my experience with academic conferences:
1. Focus on making friends. Of course it would be good to have productive discussions and make useful connections, but it is most important to know some friendly faces and feel comfortable. For me it works best to talk about unrelated things like hobbies, not about work or EA or anything like that.
2. Listening to talks is exhausting, so don’t force yourself to attend too many of them. It is fine to pick just the 2-3 most interesting talks on a day and skip everything else.
3a. Attending a talk in person is widely preferable over watching the video.
3b. Ask questions at talks. If you ask less than one question over the course of a multi-day conference, you are doing something wrong.
The issue is that FLOPS cannot accurately represent computing power across different computing architectures, in particular between single CPUs versus computing clusters. As an example, let’s compare 1 computer of 100 MFLOPS with a cluster of 1000 computers of 1 MFLOPS each. The latter option has 10 times as many FLOPS, but there is a wide variety of computational problems in which the former will always be much faster. This means that FLOPS don’t meaningfully tell you which option is better, it will always depend on how well the problem you want to solve maps onto your hardware.
In large-scale computing, the bottleneck is often the communication speed in the network. If the calculations you have to do don’t neatly fall apart into roughly separate tasks, the different computers have to communicate a lot, which slows everything down. Adding more FLOPS (computers) won’t prevent that in the slightest.
You can not extrapolate FLOPS estimates without justifying why the communication overhead doesn’t make the estimated quantity meaningless on parallel hardware.
I don’t think that 11% figure is correct. It depends on how long you would stay at the company if you would get the job, and on the time you would be unemployed for if the offer were rescinded.
Without commenting on your wider message, I want to pick on two specific factual claims that you are making.
AlphaZero went from a bundle of blank learning algorithms to stronger than the best human chess players in history...in less than two hours.
Training time of the final program is a deeply misleading metric, as these programs have been through endless reruns and tests to get the setup right. I think it is most honest to count total engineering time.
I know people are wary of Kurzweil, but he does seem to be on fairly solid ground here.
Extrapolating FLOPS is inherently fraught, as is the very idea of FLOPS being a useful unit. The problem is best illustrated by the following CS proverb: “A supercomputer is a device for turning computational complexity into communication complexity.” In particular, estimates for the complexity of imitating a small, mostly separate, part of a brain don’t linearly scale to estimates of imitating the much more interconnected whole.
The EA forum doesn’t seem like an obvious best choice. Just because it is related to EA does not make it effective, especially considering the existence of discussion software like Reddit, Discourse, and phpBB.
I’d say it mostly depends on what kind of skills and career capital you are aiming for. There are a number of important (scientific) software packages with either zero or one maintainers, which could be useful to work on either upstream or downstream.
Personally, I am presently just doing (easy) fixes for bugs that I run into myself. But I am considering to either start officially maintaining a driver that I keep patching for my own use anyway or to contribute to some decentralized web project.
It might not be super relevant for you specifically, but I do want to plug Google Summer of Code for all university students of 18 years and older as a wonderful opportunity. (application deadline April 9th)
I used to think pretty much exactly the argument you’re describing, so I don’t think I will change my mind by discussing this with you in detail.
On the other hand, the last sentence of your comment makes me feel that you’re equating my not agreeing with you with my not understanding probability. (I’m talking about my own feelings here, irrespective of what you intended to say.) So, I don’t think I will change your mind by discussing this with you in detail.
I don’t feel motivated to go back and forth on this thread, because I think we will both end up feeling like it was a waste of time. I want to make it clear that I do not say this because I think badly of you.
I will try to clear up the bits you pointed out to be confusing. In the Language section, I am referring to MIRI’s writing, as well as Bostrom’s Superintelligence, as well as most IRL conversations and forum talk I’ve seen. “bits” are an abstraction akin to “log-odds”, I made them up because not every statement in that post is a probabilistic claim in a rigorous sense and the blog post was mostly written for myself. I really do estimate that there is less than 2−170 chance of AI being risky in a way that would lead to extinction, whose risk can be prevented, and moreover that it is possible to make meaningful progress on such prevention within the next 20 years, along with some more qualifiers that I believe to be necessary to support the cause right now.
Thank you for your response and helpful feedback.
I’m not making any predictions about future cars in the language section. “Self-driving cars” and “pre-driven cars” are the exact same things. I think I’m grasping at a point closer to Clarke’s third law, which also doesn’t give any obvious falsifiable predictions. My only prediction is that thinking about “self-driving cars” leads to more wrong predictions than thinking about “pre-driven cars”.
I changed the sentence you mention to “If you want to understand present-day algorithms, the “pre-driven car” model of thinking works a lot better than the “self-driving car” model of thinking. The present and past are the only tools we have to think about the future, so I expect the “pre-driven car” model to make more accurate predictions.” I hope this is clearer.
Your remark on “English that’s precise enough to translate into code” is close, but not exactly what I meant. I think that it is a hopeless endeavour to aim for such precise language in these discussions at this point in time, because I estimate that it would take a ludicrous amount of additional intellectual labour to reach that level of rigour. It’s too high of a target. I think the correct target is summarised in the first sentence: “All sentences are wrong, but some are useful.”
I think that I literally disagree with every sentence in your last paragraph on multiple levels. I’ve read both pages you linked a couple months ago and I didn’t find them at all convincing. I’m sorry to give such a useless response to this part of your message. Mounting a proper answer would take more time and effort than I have to spare in the foreseeable future. I might post some scraps of arguments on my blog soonish, but those posts won’t be well-written and I don’t expect anyone to really read those.
My troubles with this method are two-fold.
1. SHA256 is a hashing-algorithm. Its security is well-vetted for certain kinds of applications and certain kinds of attacks, but “randomly distribute the first 10 hex-digits” is not one of those applications. The post does not include so much as a graph of the distribution of what the past drawing results would have been with this method, so CEA hasn’t really justified why the result would be uniformly distributed.
2. The least-significant digits in the IRIS data are probably fungible by adversaries. It is hard to check them, and IRIS has no reason to secure their data pipeline against attacks that might cost tens of thousands of dollars, because there are normally no stakes whatsoever attached to those bits.
Random.org is exactly in the business that we’re looking for, so they’d be a good option for their own institutional guarantee. Otherwise, any big lottery in any country will work as a source of randomness: the prizes there are bigger, which means that, even if these lotteries could be corrupted, nobody would waste that ability on rigging the donor lottery.
I’d like to see some justification for using this approach over the myriad of more responsible ways of generating random draws.