Forum? I’m against ’em!
utilistrutil
Karma: 601
Tarbell intentionally omitted?
Agree that headlines are biased to sound stronger than what you read in the piece, but I think the effect is pretty small.
Yes, sometimes things change after an article is published. Seems to me you would have to have some extra knowledge to think that AISI would be safe ex-ante. (If AISI is in fact safe now I would love to know what happened.)
Bloomberg had to add that information after publishing. See their correction:
(Updates with further details about scope of terminations in second paragraph.)
I’m not sure what the Axios piece said because of paywall rip
Overall, I think you’d be clearly better off reading the Axios piece and knowing that AISI could be in jeopardy because of pending cuts to probationary employees vs not reading it at all.
I’m not advocating to get your whole worldview from the news! And especially not your priorities. Just object level knowledge about things that are happening.
Most people in AI safety I know don’t read books either fwiw because their reading time goes to papers and blog posts.
Thanks for the links, Richard!
See my response to Scott—I think “obligatory” might have been a distracting word choice. I’m not trying to make any claims about blame/praiseworthiness, including toward oneself for (not) acting.
The post is aimed at someone who sits down to do some moral reasoning, arrives at a conclusion that’s not demanding (eg make a small donation), and feels the pull of taking that action. But when they reach a demanding conclusion (eg make a large donation), they don’t think they should feel the same pull.
For what it’s worth I didn’t have your tweets in mind when I wrote this, but it’s possible I saw them a couple weeks ago when the Discourse was happening.
Thanks for linking to the post! It satisfies most of my complaint about people not providing reasoning.
I still have some objections to it, but now I’m arguing for “there are no good reasons for certain actions to be supererogatory,” which is a layer down from “I wish people would try to give reasons.”
The post mostly gives a description of people’s attitudes toward different actions, but not so much a justification for those attitudes. (Reminds me of this paper on moral attitudes toward tradeoffs.) I agree that no one thinks it’s blameworthy to stop short of donating your last dollar.
To the extent the post make a normative case, I also agree that morality has to account for the pragmatic considerations you name (uncertainty, coordination, cognitive limitations, etc), rather than naively trying to fashion the world after your favored axiology. But I think the post makes the right response: you can just factor in those considerations, and the resulting morality might still allow offsets / be demanding. “People are only going to do a certain amount and then get tired” but maybe we’re obligated to be tired a lot, etc.
Side note: I think it’s interesting that your argument against demandingness came out of an argument against offsets because another argument against offsets is related to demandingness: whenever you’re considering an offset, one option in your choice set is to buy the offset but not do the bad thing, so you should always do that.
On obligatory: maybe using this word was a mistake, I used it because it’s what everyone uses. If it means “blameworthy not to do,” then I don’t have a position. Finding the optimal schedule of blame and praise for acts of varying levels of demandingess is an empirical problem.
I meant obligatory in the sense that moral reasoning typically obligates you to take actions. When you do a bit of moral reasoning that leads you to believe that some action would be good to take, you should feel equally bound by the moral force of that reasoning, whether it implies you should donate your first dollar or your last.
Do you agree with something like “trying to apply your axiology in the real world is probably demanding”?
Eliminating the malice or recklessness requirement and allowing punitive damages calculations to account for unrealized uninsurable risk are both big asks to make of common law courts.
Would it make a difference if the risks were insured?
My main objection is that people working in government need to be able to get away with a mild level of lying and scheming to do their jobs (eg broker compromises, meet with constituents). AI could upset this equilibrium in a couple ways, making it harder to govern.
If the AI is just naive, it might do things like call out a politician for telling a harmless white lie, jeopardizing eg an international agreement that was about to be signed.
One response is that human overseers will discipline these naive mistakes, but the more human oversight is required, the more you run into the typical problems of human oversight you outlined above. “These evaluators can do so while not seeing critical private information” is not always true. (Eg if the AI realizes that Biden is telling a minor lie to Xi based on classified information, revealing the existence of the lie to the overseer would necessarily reveal classified information).
Even if the AI is not naive, and can distinguish white lies from outright misinformation, say, I still worry that it undermines the current equilibrium. The public would call for stricter and stricter oversight standards, while government workers will struggle to fight back because
That’s a bad look, and
The benefits of a small level of deception are hard to identify and articulate.
TLDR: Government needs some humans in the loop making decisions and working together. To work together, humans need some latitude to behave in ways that would become difficult with greater AI integration.
Thanks, Agustín! This is great.
Please submit more concrete ones! I added “poetic” and “super abstract” as an advantage and disadvantage for fire.
If the organization chooses to directly support the new researcher, then the net value depends on how much better their project is than the next-most-valuable project.
This is nit-picky, but if the new researcher proposes, say, the best project the org could support, it does not necessarily mean the org cannot support the second-best project (the “next-most-valuable project”), but it might mean that the sixth-best project becomes the seventh-best project, which the org then cannot support.
In general, adding a new project to the pool of projects does not trade off with the next-best project, it pushes out the nth-best project, which would have received support but now does not meet the funding bar. So the marginal value of adding projects that receive support depends on the quality of the projects around the funding bar.
Another way you could think about this is that the net value of the researcher depends on how much better this bundle of projects is than the next-most-valuable bundle.Essentially, this is the marginal value of new projects in AI safety research, which may be high or low depending on your view of the field.
So I still agree with this next sentence if marginal = the funding margin, i.e., the marginal project is one that is right on the funding bar. Not if marginal = producing a new researcher, who might be way above the funding bar.
These are beautiful!! Made my day :))
Update: We have finalized our selection of mentors.
I’ll be looking forward to hearing more about your work on whistleblowing! I’ve heard some promising takes about this direction. Strikes me as broadly good and currently neglected.
This is so well-written!
I’m cringing so hard already fr
Thanks for such a thorough response! I am also curious to hear Oscar’s answer :)
When applicants requested feedback, did they do that in the application or by reaching out after receiving a rejection?
Is that lognormal distribution responsible for
the cost-effectiveness is non-linearly related to speed-up time.
If yes, what’s the intuition behind this distribution? If not, why is cost-effectiveness non-linear in speed-up time?
Ah yes, unfortunately journalists are very cynical in general (more on this in the upcoming newsroom epistemics post.) And rationalists are very suspicious of journalists. Together, that means my points probably sound pretty cynical/defensive. I agree with you that in practice some sources are motivated by altruistic, pro-social reasons as well.