SamuelKnoche
On Tesla: I don’t think training a special model for expensive test cars makes sense. They’re not investing in a method that’s not going to be scalable. The relevant update will come when AI5 ships (end of this year reportedly), with ~9x the memory. I’d be surprised if they don’t solve it on that hardware.
On the broader point about predictions failing: I think these were mostly failures of economic reasoning more than failures of AI progress. AI has made enormous progress on both translation and radiology imaging. What Hinton and others got wrong wasn’t the capability prediction, it was assuming the job consisted entirely of the task AI was getting good at. Turns out radiologists do more than read images, translators do more than translate sentences, and AI ends up complementary rather than substitutive. Maybe some Jevons paradox at play too. Cheaper translation means more content gets translated, which means more demand for people who can review AI output, catch cultural nuance, and put their name on the final result.
The benchmarks aren’t perfect but they consistently point to rapid progress, from METR time horizons, SWE-bench, GDPval...
On the 90% prediction: my somewhat conservative view is that AI could write 90%+ of production code this year and will next year. But I don’t think this will mean immediate mass unemployment for programmers. The job will initially just shift toward review, specification, and technical direction of AI systems. I think most SWE jobs will look more like the one of a technical PM next year.
Edit:
Customer support chat is another area with some applicability, but results are mixed
That study uses a fine-tuned GPT-3 model.
Tesla has shown that scale doesn’t confer the benefits that many (including me) had hoped.
This misses a hardware bottleneck: HW4/AI4 has 16GB RAM, capping model size regardless of training data. Tesla made real progress with v12/v13. We won’t get a clean test of the scaling hypothesis for self-driving until they ship better inference hardware.
We have already started to cross into the timeframe where some of the megaphones of AGI fever are proving to be false prophets. Dario Amodei, the CEO of Anthropic, predicted in the first quarter of this year that AI would write 90% of software code by the last quarter of this year. Nothing close to that happened.
I think you’re reading too much into self-driving as evidence about AI progress generally. Anyone who’s used Opus 4.5 in Claude Code over the past month will tell you Amodei wasn’t far off on his coding prediction. The tool’s creator reports 100% of his PRs last month were written by Claude. Amodei may have been off by a few months, and maybe is too optimistic about speed of adoption, but I still don’t think it was such a bad prediction.
My reading of this post is that it attempts to gesture at the valley of bad rationality.
I give the MWI a probability of greater than 0.5 of being correct, but as far as I can tell, there isn’t any way to generate more value out of it. There isn’t any way to create more branches. You only can choose to be intentional and explicit about creating new identifiable branches, but that doesn’t mean that you’ve created more branches. The branching happens regardless of human action.
Someone with a better understanding of this please weigh in.
I believe Sam Harris is working on an NFT project for people having taken the GWWC pledge, so that would be one example.
Academia seems like the highest leverage place one could focus on. Universities are to a large extent social status factories, and so aligning the status conferred by academic learning and research with EA objectives (for example, by creating an ‘EA University’) could be very high impact. Also relates to the point about ‘institutions.’
″...cryptocurrencies makes stopping the funding of terrorists basically impossible.”
No. Really, really, no. I could talk a lot more about this, but if you think terrorist groups can manage infosec well enough to overcome concerted attacks by the NSA, or Mossad, or FSB, etc., you’re fooling yourself.
“Impossible” might be an exaggeration, but it does seem to make it much easier. That’s also what the article you link to suggests. Edit: Are you skeptical because of the on/off ramps, the security of terrorist’s computer infrastructure or something else?
Other than the misunderstanding and conflating nodes for hash power, this is also not true. Has power is concentrated, so you’d need to somehow convince the biggest mining groups that they don’t care about countries keeping their operations legal, and as we’ve seen, they do. That means they will continue to run to embrace KYC/AMF regulation, and will do whatever else makes their investments go well—including cooperating with nation-states in almost any way you can imagine.
So far, the only serious KYC/AMF happens at the level of centralized exchanges. Nation-states cannot enforce KYC/AMF at the level of decentralized exchanges. They can also use chain-analysis and put pressure on mining groups within their countries to do KYC/AMF or to create address “black lists” but so far there hasn’t been much political will for this, and probably would lead to a big backlash from the crypto community. And this becomes impossible for privacy coins such as Zcash and Monero.
I feel like a number of these maybe could be fitted under a single very large organization. Namely:
Max-Planck Society (MPG) for EA research
EA university
Forecasting Organization
EA forecasting tournament
ML labs
Large think tank
Basically, a big EA research University with a forecasting, policy research and ML/AI safety department.
I’d also add non-profit and for-profit startup incubator. I think Universities would be much better if they made it possible to try something entrepreneurial without having to fully drop-out.
In my experience, EAs tend to be pretty dissatisfied with the higher education system, but I interpreted the muted/mixed response to my post on the topic as a sign that my experience might have been biased, or that despite the dissatisfaction, there wasn’t any real hunger for change. Or maybe a sense that change was too intractable.
Though I might also have done a poor job at making the case.
My speculative, cynical, maybe unfair take is that most senior EAs are so enmeshed in the higher education system, and sunk so much time succeeding in it, that they’re incentivized against doing anything too disruptive that might jeopardize their standing within current institutions. And why change how undergrad education is done if you’ve already gone through it?
The very quick summary: Japan used to be closed off from the rest of the world, until 1853 when the US forced them to open up. This triggered major reforms. The Shogun was overthrown and replaced with the emperor, and in less than a century, Japan went from an essentially medieval economic and societal structure, to a modern industrial economy.
I don’t know of any books exclusively focused on it, but it’s analyzed in Why Nations Fail and Political Order and Political Decay.
I have argued for a more “mutiny” (edit: maybe “exit” is a better word for it) style theory of change in higher education so I really like the idea of an EA university where learning would be more guided by a genuine sense of purpose, curiosity and ambition to improve the world rather than a zero-sum competition for prestige and a need to check boxes in order to get a piece of paper. Though I realize that many EAs probably don’t share my antipathy towards the current higher education system.
One downside of EA universities I can think of is that it might slow movement growth since EAs will be spending less time with people unfamiliar with the movement / fewer people at normal universities will come across EA.
Though if it becomes really successful and prestigious, it could also raise the profile of EA.
Another example that comes to mind is Japan’s Meiji Restoration. I don’t think it fits neatly in any of the categories. It’s a combination of mutiny, steering and rowing. But just like the American revolution, I think it illustrates that very rapid and disruptive change in political and economic systems can be undertaken successfully.
The ability to maintain, or improve steering and/or rowing seem to be two important preconditions for a successful mutiny.
Also, the various revolutions that swept Eastern Europe and led to the end of the Soviet Union also seem to be successful mutinies. Of course, the reason these countries ended up under Soviet communism and needed to rise up was because of the Bolshevik mutiny, but still.
I feel like people in EA are mostly anti-mutiny because the only people advocating for it seem to be far left, anti-capitalist types who don’t seem to have a realistic plan for how to go about it, or a coherent plan for what could replace it. But I don’t think EA should be closed to the idea of mutiny in principle. It’s just that any mutiny proposal has to pass a really high bar.
Thanks for clarifying. I did somewhat misinterpret the intention of your comment.
I agree that the US revolution was unusual and in many ways more conservative than other revolutions.
I guess you could think of the US revolution as being a bit like a mutiny that then kept largely the same course as the previous captain anyway.
I feel like this is really underselling what happened, though I guess it might be subjective. Sure, they didn’t try to reinvent government, culture and the economy completely from scratch, but it was still the move from a monarchy to the first modern liberal constitutional republic.
If something dangerous occurs when driving, slamming on the brakes is often a pretty good heuristic, regardless of the specific nature of the danger.
What if you’re being chased by a dragon?
I think we can make a similar analogy for Anchoring, because some the same reasons that make Steering more attractive now than in the past also apply for Anchoring. If there are an unusually large number of icebergs up ahead, or you are afraid the Mutineers will steer us towards them, or you are attempting to moor up alongside a larger vessel, reducing speed could be a generally prudent move—and this is the case even if full speed ahead was the optimal strategy in the past when you were on the open seas.
What if you think that the people currently Steering are the ones blindly heading towards the icebergs? Wouldn’t Mutiny be an option worth considering? What if the ship is taking on water and people in the lower decks are drowning? Wouldn’t you want to Speed up and get to land as fast as possible?
This metaphor doesn’t seem too informative until we’ve made sense of what world we actually live in.
I agree with this. I was just pushing back against the “somewhere between never-before-done and impossible” characterization. Mutiny definitely goes wrong more often than not, and just blindly smashing things without understanding how they work, and with no real plan for how to replace them is a recipe for disaster.
Certainly, but I still think that it counts as an example of a successful “mutiny.” If overthrowing the government and starting a new country isn’t mutiny, I don’t know what is. And I don’t think anyone sympathetic to the mutiny theory of change wants to restart from the state of nature and reinvent all of civilization completely from scratch.
I would suggest that the feasibility of managing it once you smashed all the working pieces is somewhere between never-before-done and impossible.
How does the American Revolution fit into this? Wasn’t the US basically created from scratch, and now is arguably the most successful country in the world?
Ideally, people would get the opportunity to get up to speed, “bridge the inferential gap” and get to start thinking about how to have an impact full time during their undergraduate studies. The way most university programs are set up right now, people spend years on often irrelevant content and wasteful busywork. I was thus pleased to see Ben Todd and Will MacAskill mention the idea of creating some kind of EA University during their EAG appearances.
See also my own “case for education.”
If the Everett interpretation is true, then all experiences are already amplified exponentially. Unless I’m missing something, a QC doesn’t deserve any special consideration. It all adds up to normality.
Re Tesla: My best guess is that they still need 5-20x reliability to match human level, and I don’t entirely rule out that they’ll manage it with AI4. Hard to get good data on this though. It sounds like they were still finalizing the AI5 chip design until quite recently, and I’m not sure it makes sense to spend training budget on a model that can only run on a handful of cars while they’re still hoping to squeeze more out of AI4. There’s likely an inference overhang here. They’ve spent years scaling training data and compute while model size stayed fixed, way, way past the usual theoretical optimal tradeoff point. Lifting the size constraint will probably yield disproportionate gains.
Re the middle stuff: I think we just disagree on how to weigh various evidence from failed predictions (based on narrow models, older models...), various firsthand reports and more recent benchmark results.
Re the 90% prediction: by “conservative” I meant this is like my 5-10th percentile slowest timeline. I’ve heard from a number of SWEs that they’re already basically not writing code, just instructing and reviewing. I’m also uncertain about adoption speed. I’d put it at >50% chance that among SWEs actually using the latest LLMs and tools, AI writes 90%+ of their code in the first half of this year.