This other Ryan Greenblatt is my old account[1]. Here is my LW account.
- ^
Account lost to the mists of time and expired university email addresses.
This other Ryan Greenblatt is my old account[1]. Here is my LW account.
Account lost to the mists of time and expired university email addresses.
EAs are especially rational people and not eating animals is obviously the more rational choice for 90%+ people reading this
I’m about 99% bivalve vegan (occasionally I eat fish for cognitive reasons). However, I think it doesn’t make sense for strongly longtermist individuals in terms of the direct straightforward benefits of veganism. The direct animal suffering is negligable relative to the future. I’m strongly longtermist, but I stay vegan for a combination of less direct reasons like signaling to myself and generally being cooperative (for reasons like acausal decision theory and directly being cooperative with current people).
First, the recent surveys of the general public’s attitudes towards AI risk suggest that a strongly enforced global pause would actually get quite a bit of support. It’s not outside the public’s Overton Window. It might be considered an ‘extreme solution’ by AI industry insiders and e/acc cultists. But the public seems to understand that it’s just fundamentally dangerous to invent Artificial General Intelligence that’s as smart as smart humans (and much, much faster), or to invent Artificial Superintelligence. AI experts might patronize the public by claiming they’re just reacting to sensationalized Hollywood depictions of AI risk. But I don’t care. If the public understands the potential risks, through whatever media they’ve been exposed to, and if it leads them to support a pause, we might as well capitalize on public sentiment.
I think the public might support a pause on scaling, but I’m much more skeptical about the sort of hardware-inclusive pause that Holden discusses here:
global regulation-backed pause on all investment in and work on (a) general3 enhancement of AI capabilities beyond the current state of the art, including by scaling up large language models; (b) building more of the hardware (or parts of the pipeline most useful for more hardware) most useful for large-scale training runs (e.g., H100’s); (c) algorithmic innovations that could significantly contribute to (a)
A hardware-inclusive pause which is sufficient for pausing for >10 years would probably effectively dismantle companies like nvidia and would be at least a serious dent in TSMC. This would involve huge job loss and a large hit to the stock market. I expect people would not support such a pause which effectively requires dismantling a powerful industry.
It’s possible I’m overestimating the extent to which hardware needs to be stopped for such a ban to be robust and an improvement on the status quo.
I’m in a relatively similar position to Neel. I think technical AI safety grant makers typically know way more than me about what is promising to fund. There is a bunch of non-technical info which is very informative for knowing whether a grant is good (what do current marginal grants look like, what are the downside risks, is there private info on the situation which makes things seem sketchier, etc.) and grant makers are generally in a better position than I am to evaluate this stuff.
The limiting factor [in technical ai safety funding] is in having enough technical grant makers, not in having enough organizational diversity among grantmakers (at least at current margins).
If OpenPhil felt more saturated on technical AI grant makers, then I would feel like starting new orgs pursing different funding strategies for technical AI safety could look considerably better than just having more people work at grant making at OpenPhil.
That said, note that I tend to agree to reasonable extent with the technical takes at OpenPhil on AI safety. If I heavily disagreed, I might think starting new orgs looks pretty good.
OP doesn’t have the capacity to evaluate everything, so there are things they don’t fund that are still quite good.
Also OP seems to prefer to evaluate things that have a track record, so taking bets on people to be able to get more of a track record to then apply to OP would be pretty helpful.
IMO, these both seem like reasons for more people to work at OP on technical grant making more than reasons for Neel to work part time on grant making with his money.
The underlying claim is that many people with technical expertise should do part time grant making?
This seems possible to me, but a bit unlikely.
One problem I have with these discussions, including past discussions about why national EA orgs should have fundraising platform, is the reductionist and zero-sum thinking given in response.
Wait, but it might actually have opportunity cost? Like those poeple could be doing something other than trying to get more medium sized donors? There is a cost to trying to push on this versus something else. (If you want to push on it, then great, this doesn’t impose any cost on others, but that seems different from a claim that this is among the most promising things to be working on at the margin.)
I identified above, how an argument stating less donors results in more efficiency, would never be made in the for profit world. Similarly, a lot of the things we care about (talent, networks, entrepreneurship) become stronger the more small/medium donors we have. For the same reason that eating 3 meals in a day makes it easier to be productive—despite it taking more time compared to not eating at all—having more of a giving small-donor ecosystem will make it easier to achieve other things we need.
Your argument here is “getting more donors has benefits beyond just the money” (I think). But, we can also go for those benefits directly without necessarily getting more donors. Like maybe trying to recruit more medium sized donors is the best way to community build, but this seems like sort of a specific claim which seems a priori unlikely (it could be true ofc) unless having more small donors is itself a substantial fraction of the value and that’s why it’s better than other options.
So, recruiting donors is perhaps subsidized by causing the other effects you noted, but if it’s subsidized by some huge factor (e.g. more like 10x than 1.5x) than directly pursuing the effects seems like probably a better strategy.
Thanks for the report.
If I were to add one thing to this report, it would probably be a comparison of increasing the likelihood of space settlement vs increasing the likelihood of extremely resilient and self-sustaining disaster shelters (e.g. shelters that could be self-sustaining for decades or possibly centuries). You note the similarities in “Design of disaster shelters”, but don’t compare these as possible interventions (as far as I can tell).
My naive (mostly uninformed) guess would have been that very good disaster shelters are wildly cheaper and easier (prior to radical technology change like superhuman AI or nanotech) while offering most of the same benefits.
(I put a low probability on commercially viable and self-sustaining space colonies prior to some other radical change in the technical landscape, but perhaps I’m missing some story for economic viability. Like I think the probability of these sorts of space colonies in the next 60 years is low (without some other radical technical advancement like AI or nanotech happening prior in which case the value add is more complex).)
This seems like it might be a good price discrimination strategy though I’m not sure if that’s the intent.
From a “current generations” perspective, reducing GCRs is probably not more cost-effective than directly improving the welfare of people / animals alive today
I think reducing GCRs seems pretty likely to wildly outcompete other traditional approaches[1] if we use a slightly broad notion of current generation (e.g. currently existing people) due to the potential for a techno utopian world which making the lives of currently existing people >1,000x better (which heavily depends on diminishing returns and other considerations). E.g., immortality, making them wildly smarter, able to run many copies in parallel, experience insanely good experiences, etc. I don’t think BOTECs will be a crux for this unless we ignore start discounting things rather sharply.
If GCRs actually are more cost-effective under a “current generations” worldview, then I question why EAs would donate to global health / animal charities (since this is no longer a question of “worldview diversification”, just raw cost-effectiveness)
IMO, the main axis of variation for EA related cause prio is “how far down the crazy train do we go” not “person affecting (current generations) vs otherwise” (though views like person affecting ethics might be downstream of crazy train stops).
Mildly against the Longtermism --> GCR shift
Idk what I think about Longtermism --> GCR, but I do think that we shouldn’t lose “the future might be totally insane” and “this might be the most important century in some longer view”. And I could imagine focus on GCR killing a broader view of history.
That said, if we literally just care about experiences which are somewhat continuous with current experiences, it’s plausible that speeding up AI outcompetes reducing GCRs/AI risk. And it’s plausible that there are more crazy sounding interventions which look even better (e.g. extremely low cost cryonics). Minimally the overall situation gets dominated by “have people survive until techno utopia and ensure that techno utopia happens”. And the relative tradeoffs between having people survive until techno utopia and ensuring that techno utopia happen seem unclear and will depend on some more complicated moral view. Minimally, animal suffering looks relatively worse to focus on.
I believe prior work showed large effects from short periods of supplementation. (Edit: Note that this work seems to debunk prior work, but this should explain the study design)
Thanks for the comment.
I’d like to stress that it seems at least plausible to me that encouraging space colonization could be a worthwhile cause area. (I’m uncertain what is sufficiently neglected here, but it seems plausible that there are some key neglected areas.) And I’d like to stress that I haven’t thought about the area very much.
Accordingly, feel free to not engage with the rest of my comment.
A sufficient crux for me would be thinking that it’s doable to substantially effect the probability of commercially viable long-term space colonization[1] within the next 40 years. My main concern here would be that commercially viable long-term space colonization within 40 years is quite unlikely by default and thus hard to boost the absolute probability by much (it’s way down on the logistic success curve). My second biggest concern would be that this doesn’t really seem very neglected. (Though perhaps resources are sufficiently inefficiently allocated that there is a neglected subfield?)
To emphasize, this is a sufficient crux, but not the only sufficient crux.
I have >25% on AI taking longer than 30 years, so this isn’t that much of discount on this view (but the potential for working on AI being better might be a substantial discount.)
That is, this space colonization happening prior to some other radical technological change. As, in encouraging space colonization after transformative AI or nanotech doesn’t seem that important for various reasons.
Are you aware of any projects currently pursuing this?
People have certainly talked about this on the forum, but maybe people currently think it seems somewhat more cost effective to work on other projects given the current x-risk reduction funding?
Finally, to get on my favourite soapbox, dunking on the Metaculus ‘Weakly General AGI’ forecast:
I think the forecast seems broadly reasonable, but the question and title seem quite poor. As in, the operationalization seems like a very poor definition for “weakly general AGI” and the tasks being forecast don’t seem very important or interesting.
I think GPT-4V likely already achieves 2 (winograd) and 3 (SAT) while 4 (montezuma’s revenge) seems plausible for GPT-4V, though unclear. Beyond this, 1 (turing test) seems to be extremely dependent on the extent to which the judge is competently adversarial and whether or not anyone actually finetunes a powerful model to perform well on this task. This makes me think that this could plausibly resolve without any more powerful models, but might not happen because no one bothers running a turing test seriously.
Number 3 requires SAT tests (or, i guess, tests with overlapping Questions and Answers) not be in the training data. The current paradigm relies on scooping up everything, and I don’t know how much fidelity the model makers have in filtering data out. Also, it’s unlikely they’d ever show you the data they trained on as these models aren’t proprietary. So there’s know way of knowing if a model can meet point 3!
Can’t we just use an SAT test created after the data cutoff? Also, my guess is that the SAT results discussed in the GPT-4 blog post (which are >75th percentile) aren’t particularly data contaminated (aside from the fact that different SAT exams are quite similar which is the same for human students). You can see the technical report for more discussion on data contamination (though account for bias accordingly etc.)
Previously some AI risk people confidently thought that Gemini would be substantially superior to GPT-4.
I think this slightly misrepresents the corresponding article and the state of the forecasts. The quote from the linked article is:
By all reports, and as one would expect, Google’s Gemini looks to be substantially superior to GPT-4. We now have more details on that, and also word that Google plans to deploy it in December, Manifold gives it 82% to happen this year and similar probability of being superior to GPT-4 on release.
This doesn’t seem to exhibit that much confidence in “gemini being substantially superior”? I expect that if Zvi gave specific probabilites, they would be pretty reasonable.
ETA: I retract my claim about Zvi, on further examination, he seems pretty wrong here. That said, manifold doesn’t seem to have done too badly.
Further, manifold doesn’t seem that wrong here on GPT4 vs gemini? See for instance, this market:
The forecast has updated from 80% to about 60%, which doesn’t seem like much of an update.
I agree that we should update down on google competence and near term AGI, but it just doesn’t seem like that big of an update yet?
[Unimportant]
here are two recent papers from Melanie Mitchell, [...] and another actually empirically performing the Turing Test with LLMs (the results will shock you)
This doesn’t seem to be by Melanie Mitchell FYI. At least she isn’t an author.
I think this announcement should make people think near term AGI, and thus AIXR, is less likely. To me this is what a relatively continuous takeoff world looks like, if there’s a take off at all. If Google had announced and proved a massive leap forward, then people would have shrunk their timelines even further. So why, given this was a PR-fueled disappointment, should we not update in the opposite direction?
[...]
Gemini release is disappointing. Below many people’s expectations of its performance. Should downgrade future expectations. Near term AGI takeoff v unlikely. Update downwards on AI risk (YMMV).
I think the update here should be pretty small. I’m unsure if you disagree. I would also think the update should be pretty small if gemini is notably better than GPT4, but not wildly better. It seems plausible to me that people would (incorrectly) have a large update toward shorter timelines if gemini was merely substantially better than GPT4, but we don’t have to make the same mistake in the other direction.
It’s worth noting there is some asymmetry in the likely updates with a high probability of a mild negative update on near term AI and a low probability of a large positive update toward powerful near term AI. E.g., even if google were to explode and never release a better LLM than gemini, this would be a relatively smaller update than if they were to release transformatively powerful AI.
Some previous discussion:
Thanks for the response!
A few quick responses:
it says ‘less than 10 SAT exams’ in the training data in black and white
Good to know! That certainly changes my view of whether or not this will happen soon, but also makes me think the resolution criteria is poor.
I think funding, supporting, and popularising research into what ‘good’ benchmarks would be and creating a new test would be high impact work for the AI field—I’d love to see orgs look into this!
You might be interested in the recent OpenPhil RFP on benchmarks and forecasting.
Perhaps the median community/AI-Safety researcher response was more measured.
People around me seemed to have a reasonably measured response.
I think we’ll probably get a pretty big update about the power of LLM scaling in the next 1-2 years with the release of GPT5. Like, in the same way that each of GPT3 and GPT4 were quite informative even for the relatively savvy.
But Buck wasn’t saying you shouldn’t be scared? He was just saying that high burner count isn’t much evidence for this.
Precisely, I think he was claiming that p(lots of burners | hiding identity is important) and p(lots of burners | hiding identity isn’t important) are pretty close.
I interpreted this as a pretty decoupled claim. (I do think a disclaimer might have been good.)
Now, this second comment (which is the root comment here) does try to argue you shouldn’t be worried, at least from Holden and somewhat from buck.