Glad to hear about this!
I have a recommendation for the structure of it. I’d recommend that anonymous reviewers review submissions and share their reviews with the authors (perhaps privately) before a rebuttal phase (also perhaps private). And then reviewers can revise their reviews, and then chairs can make judgments about which submissions to publish.
Is your claim that AI policy is currently talent-constrained, and having Yang as president would lead to more people working on it, thereby making it money-constrained?
No—just that there’s perhaps a unique opportunity for cash to make a difference. Otherwise, it seems like orgs are struggling to spend money to make progress in AI policy. But that’s just what I hear.
Can you elaborate on this?
First pass: power is good. Second pass: get practice doing things like autonomous weapons bans, build a consensus around getting countries to agree to international monitoring of software, get practice doing that monitoring, negotiate minimally invasive ways of doing this that protect intellectual property, devote resources to AI safety research (and normalize the field), and ten more things I haven’t thought of.
The stake of the public good in any given election is much larger than the stake of any given entity, so the correct amount for altruists to invest in an election should be much larger than for a self-interested corporation or person.
not that he single-handedly caused Trump’s victory.
Didn’t claim this.
This is naive.
Not sure what this adds.
I’m someone who has read your work (this paper and FGOIL, the latter of which I have included in a syllabus), and who would like to see more work in similar vein, as well as more formalism in AI safety. I say this to establish my bona fides, the way you established your AI safety bona fides.
Thanks! I should have clarified it has received some interest from some people.
you don’t show that “when a certain parameter of a certain agent is set sufficiently high, the agent will not aim to kill everyone”, you show something more like “when you can design and implement an agent that acts and updates its beliefs in a certain way and can restrict the initial beliefs to a set containing the desired ones and incorporate a human into the process who has access to the ground truth of the universe, then you can set a parameter high enough that the agent will not aim to kill everyone”
“When you can design and implement an agent that acts and updates its beliefs in a certain way and can restrict the initial beliefs to a set containing the desired ones”. That is the “certain agent” I am talking about. “Restrict” is an odd word choice, since the set can be as large as you like as long as it contains the truth. “and incorporate a human into the process who has access to the ground truth of the universe.” This is incorrect; can I ask you to edit your comment? Absolutely nothing is assumed about the human mentor, certainly no access to the ground truth of the universe; it could be a two-year-old or a corpse! That would just make the Mentor-Level Performance Corollary less impressive.
I don’t deny that certain choices about the agent’s design make it intractable. This is why my main criticism was “People don’t seem to bother investigating or discussing whether their concerns with the proposal are surmountable.” Algorithm design for improved tractability is the bread and butter of computer science.
I constructed an agent where you can literally prove that if you set a parameter high enough, it won’t try to kill everyone, while still eventually at least matching human-level intelligence. Sure it uses a realizability assumption, sure it’s intractable in its current form, sure it might require an enormously long training period, but these are computer science problems, not philosophy problems, and they clearly suggest paths forward. The underlying concept is sound. It struck me as undignified to say this in the past, but maybe dignity rightly construed should compel me to: it absolutely boggles me that ~no one in the EA community talks about this. It’s not in this blog post; it’s not in Richard’s curriculum; it wasn’t in Evan’s list of promising AGI safety ideas.
I agree with your perspective on all of these approaches, except my initial reaction is to be more pessimistic about natural abstractions. It seems to me that a good understanding of natural abstractions is not good enough for putting a handle on a part of an agent’s mind. We’d also need to understand “natural types”, the type signatures that agents’ brains use to represent those abstractions. And I think that there is a long, long list of types, in which each is as natural as the rest.
There’s an interpretability benchmark that occurred to me recently, which I may as well mention here, because I agree approximately none of the interpretability research I see strikes me as progress toward strategically relevant interpretation of AGI. Try to understand what corvids are saying to each other.
I am in general more trusting, so I appreciate this perspective. I know he’s a huge fan of Sam Harris and has historically listened to his podcast, so I imagine he’s head Sam’s thoughts (and maybe Stuart Russell’s thoughts) on AGI.
This is very high-quality. No disputes just clarifications.
I don’t just mean meta-orgs.
I think working for a well-financed grantmaking organization is not outrageously unconventional, although I suspect most lean on part-time work from well-respected academics more than OpenPhil does.
And I think 80k may just be an exception (a minor one, to some extent), borne out of an unusually clear gap in the market. I think some of their work should be done in academia instead (basically whatever work it’s possible to do), but some of the very specific stuff like the jobs board wouldn’t fit there.
Also, if we imagine an Area Dad from an Onion Local News article, I don’t think he’s skepticism would be quite as pronounced for 80k as for other orgs like, e.g., an AI Safety camp.
What “major life goals should include (emphasis added)” is not a sociological question. It is not a topic that a sociology department would study. See my comment that I agree “conventional wisdom is wrong” in dismissing the philosophy of effective altruism (including the work of Peter Singer). And my remark immediately thereafter: “Yes, these are philosophical positions, not sociological ones, so it is not so outrageous to have a group of philosophers and philosophically-minded college students outperform conventional wisdom by doing first-principles reasoning”.
I am not citing Amazon as an example of an actor using evidence and reason to do as much good as possible. I am citing it as an example of an organization that is effective at what it aims to do.
Upvoted this.
You generally shouldn’t take Forum posts as seriously as peer-reviewed papers in top journals
I suspect I would advise taking them less seriously than you would advise, but I’m not sure.
It could also imply that EA should have fewer and larger orgs, but that’s a question too complicated for this comment to cover
I think there might be a weak conventional consensus in that direction, yes. By looking at the conventional wisdom on this point, we don’t have deal with the complicatedness of the question—that’s kind of my whole point. But even more importantly, perhaps fewer EA orgs that are not any larger; perhaps only two EA orgs (I’m thinking of 80k and OpenPhil; I’m not counting CHAI as an EA org). There is not some fixed quantity of people that need to be employed in EA orgs! Conventional wisdom would suggest, I think, that EAs should mostly be working at normal, high-quality organizations/universities, getting experience under the mentorship of highly qualified (probably non-EA) people.
Among many things I agree with, the part I agree the most with:
EAs give high credence to non-expert investigations written by their peers, they rarely publish in peer-review journals and become increasingly dismissive of academia
I think a fair amount of the discussion of intelligence loses its bite if “intelligence” is replaced with what I take to be its definition: “the ability to succeed a randomly sampled task” (for some reasonable distribution over tasks). But maybe you’d say that perceptions of intelligence in the EA community are only loosely correlated with intelligence in this sense?
As for cached beliefs that people accept on faith from the writings of perceived-intelligent central figures, I can’t identify any beliefs that I have that I couldn’t defend myself (with the exception that I think many mainstream cultural norms are hard to improve on, so for a particular one, like “universities are the best institutions for producing new ideas”, I can’t necessarily defend this on the object level). But I’m pretty sure there aren’t any beliefs I hold just because a high-status EA holds them. Of course, some high-status EAs have convinced me of some positions, most notably Peter Singer. But that mechanism for belief transmission within EA, i.e. object-level persuasion, doesn’t run afoul of your concerns about echochamberism, I don’t think.
But maybe you’ve had a number of conversations with people who appeal to “authority” in defending certain positions, which I agree would be a little dicey.
If you’re super focused on that issue, then it will definitely be better to spend your money on actual AI research, or on some kind of direct effort to push the government to consider the issue (if such an effort exists).
I am, and that’s what I’m wondering. The “definitely” isn’t so obvious to me. Another $20 million to MIRI vs. an increase in the probability of Yang’s presidency by, let’s say, 5%--I don’t think it’s clear cut. (And I think MIRI is the best place to fund research).
Well, there are >100 million people who have to join some constituency (i.e. pick a candidate), whereas potential EA recruits aren’t otherwise picking between a small set of cults philosophical movements. Also, AI PhD-ready people are in much shorter supply than, e.g. Iowans, and they’d be giving up much much much more than someone just casting a vote for Andrew Yang.
we’ve had two presidents now who actively tried to counteract mainstream views on climate-change, and they haven’t budged climate scientists at all.
I have updated in your direction.
Of course, AI alignment is substantially more scientifically accepted and defensible than climate skepticism.
Yep.
You only mean this as a possibility in the future, if there is any point where AGI is believed to be imminent, right?
No I meant starting today. My impression is that coalition-building in Washington is tedious work. Scientists agreed to avoid gene editing in humans well before it was possible (I think). In part, that might have made it easier since the distantness of it meant fewer people were researching it to begin with. If AGI is a larger part of an established field, it seems much harder to build a consensus to stop doing it.
MIRI’s current size seems to me to be approximately right for this purpose, and as far as I know MIRI staff don’t think MIRI is too small to continue making steady progress.
My guess is that this intuition is relatively inelastic to MIRI’s size. It might be worth trying to generate the counterfactual intuition here if MIRI were half its size or double its size. If that process outputs a similar intuition, it might be worth attempting to forget how many people MIRI employs in this area, and ask how many people should be working on a topic that by your estimation has a 10% chance of being instrumental to an existential win. Though my number is higher than 10%, I think even if I had that estimate, my answer to the number of people that should be working on that topic would be “as many as are available.”
The other thing is that in 20 years, we might want the president on the phone with very specific proposals. What are the odds they’ll spend a weekend discussing AGI with Andrew Yang if Yang used to be president vs. if he didn’t?
But as for what a president could actually do: create a treaty for countries to sign that ban research into AGI. Very few researchers are aiming for AGI anyway. Probably the best starting point would be to get the AI community on board with such a thing. It seems impossible today that consensus could be built about such a thing, but the presidency is a large pulpit. I’m not talking about making public speeches on topic; I mean inviting the most important AI researchers to the White House to chat with Stuart Russell and some other folks. There are so many details to work out that we could go back and forth on, but that’s one possibility for something that would be a big deal if it could be made to work.
Thank you for the edit, and thank you again for your interest. I’m still not sure what you mean by a person “having access to the ground truth of the universe”. There’s just no sense I can think of where it is true that this a requirement for the mentor.
“The system is only safe if the mentor knows what is safe.” It’s true that if the mentor kills everyone, then the combined mentor-agent system would kill everyone, but surely that fact doesn’t weight against this proposal at all. In any case, more importantly a) the agent will not aim to kill everyone regardless of whether the mentor would (Corollary 14), which I think refutes your comment. And b) for no theorem in the paper does the mentor need to know what is safe; for Theorem 11 to be interesting, he just needs to act safely (an important difference for a concept so tricky to articulate!). But I decided these details were beside the point for this post, which is why I only cited Corollary 14 in the OP, not Theorem 11.
That is plausible. But “definitely” definitely wouldn’t be called for when comparing Yang with Grow EA. How many EA people who could be sold on an AI PhD do you think could recruited with $20 million?
I’d love to chat with you about directions here, if you’re interested. I don’t know anyone with a bigger value of p(survival | West Wing levels of competence in major governments) - p(survival | leave it to OpenAI and DeepMind leadership). I’ve published technical AI existential safety research at top ML conferences/journals, and I’ve gotten two MPs in the UK onside this week. You can see my work at michael-k-cohen.com, and you can reach me at michael.cohen@eng.ox.ac.uk.