I wonder if the disclosures could be non-text by default—e.g. colour-coded with an optional footnote for details.
The thing I’m not liking as a reader is having words to process on this stuff at the start (for me this isn’t just cases where people aren’t following policy; I’ve felt it some about a case where the words were one of the suggested wordings from the policy). Non-text ways to signal could potentially get best-of-both-worlds in terms of reader attention.
Ok so I can kind of tune into what you’re saying here, but I also feel kind of uneasy about it. I guess I’d be curious what you make of the following potential arguments:
Ingredients are important because we can’t directly discern what’s in food. But with writing we can see exactly what’s there and judge that directly without needing to judge the process. (This perspective would endorse reviews being posted warning people not to read low-quality stuff.)
Requiring disclosure is an inappropriate form of thought policing—people should have the right to use whatever cognitive processes and augmentation methods they like, and take responsibility for the words they then share. If this produces LLM garbage it’s not on them to label that up front, but this should have the natural consequence that people stop listening to them.
Ok, so one place the predictions of these theories might come apart is that my theory suggests a norm against impersonating medics, whereas I think yours doesn’t (although maybe I’m just not seeing it; I don’t think I would have said that avoiding torture of prisoners was part of protecting the mechanisms of ending war, although I do kind of see what you mean). I haven’t looked into it at all, but if that norm has emerged independently multiple times that would be suggestive in favour of the broader theory; whereas if it has just emerged once it looks perhaps more potentially-idiosyncratic, which would be suggestive in favour of the narrower theory.
I agree that the model I proposed is imprecise; I think this counts against its usefulness but not its validity.
I’m not suggesting this as a thing to advocate for; merely as a descriptive pattern of what the category of war crimes is doing. I think the things which make ending war harder are an important class of really destructive thing, but it seems clarity-obscuring to me to claim that this is definitionally what war crimes are? Rather than giving your thing a new label and then getting to discuss what fraction of war crimes are in that category, and whether there are things in that category which aren’t war crimes (e.g. if torturing POWs counts under your categorization, then why doesn’t conscription count—after all, it damages the “one side runs out of soldiers” mechanism for ending war).
I like the puzzle. But I wonder if you can make your answer even simpler:
Actions taken in war have some benefit to the perpetrator, and some costs to the larger system of permitting them
When the ratio between these things gets too extreme, it’s regarded as a war crime
I think this explains the category that you outline (undermining trust in the kind of institutions that could stop the war is super destructive!), but also explains some other cases, e.g. abuse of prisoners, not impersonating medical staff, etc.
Hmm, I’ve used LLMs to varying degrees in writing articles. Usually not to the point of writing significant amounts of text, but a case where I think it clearly helped to improve the output is this story: https://strangecities.substack.com/p/some-days-soon
Functionally, I wrote a complete draft, then got Claude to redraft, then I went through and stitched the best bits of the two drafts together (or wrote new versions where that seemed best). (If you thought the original draft was better I’d be interested to hear that: https://docs.google.com/document/d/1icY2wpcgvKszfzHFButKcOwV8B9xMypTAk48kjnOGz0/edit?usp=drivesdk )
(I notice that I’m more likely to find LLMs helpful in drafting things when writing fiction. I think it’s least likely to help when it’s important to convey my precise epistemic status towards the things I’m saying.)
Requiring disclosures to be at the top of the post (rather than e.g. allowing them to be at the bottom) does feel like it’s sending some implicit “this is kind of bad so people need to be warned about it” message, even if it’s in a “recommended uses” section.
Like I think people might reasonably worry about others pre-judging posts with this disclaimer, and hence (perhaps, sometimes) prefer workflows where they don’t need to include the disclaimer, even if this makes their posts worse.
I don’t think there’s an easy answer here—like, presumably the point of the policy is to allow this kind of pre-judging and let people make differently-informed choices about what they engage with. But I think the post kind of papers over this tension.
I think we’re still primarily assessing what was said. And you can make your system flexible enough that it doesn’t just entrench existing actors.
I think this pushes towards filtering happening via means other than just assessing the argument—e.g. something like persistent reputations (with some occasional sampling to allow new entrants to get out of the zone of being ignored). cf. discussion of how AI could help us keep tabs on the reliability of different actors.
I imagine that there’s a relatively cheap filter to work out which arguments to even engage with, and then most effort can still go into engaging with those arguments.
I haven’t read the whole post, but tripped up on this:
When the cost of generating arguments approaches zero, their value as traces of reasoning falls, while their value as proxy signals — indirect markers of reliability more amenable to social confirmation — rises.
This seems backwards to me? They’re no longer a costly signal, but they can still be assessed for legitimacy.
You can have a smart system make inferences from camera visible information.
But yeah, the main use case we had in mind for the monitoring layer was not about these very tricky-to-observe states, but expanding the space of things you can make agreements about (potentially including some high-stakes cases, as I write about at the end of this story: https://strangecities.substack.com/p/some-days-soon).
This is basically the reason I regard this as the most technically challenging of the things we’re presenting here. You eventually want a system which is not just a passive consumer of data, but can actively explore. You may need to give it access to robots with cameras and internet so that it can verify some of the basics of its setup. It might still fear that the entire thing is being spoofed, but I think it’s vastly harder to generate a plausible world that’s robust to the agent exploring and running consistency probes.
I’m kind of unsure which of the sketches you’re talking about with this question. Could you ask it of whichever one you feel it’s cleanest for?
Plausible, yes. For one thing you can run versions of the coordination tech in parallel with old cheap models, and flag and dig into discrepancies. This could make it harder for misalignment to strongly bite.
Of course if there are big misalignment issues and we’re not seriously tracking that there could be big misalignment issues, that’s gonna be a problem.
I feel like you’re baking a lot into this clause:
With AI delegates, they would presumably be verifiable and would be programmed to tell the truth and keep to deals
I think that aiming for an equilibrium where that’s true would be good, but I’m not certain that’s the starting point (and if it were otherwise going to scupper getting this off the ground, it probably shouldn’t be the starting point).
So if one person adopts the AI delegate and another doesn’t, then the human can overexaggerate their preferences, withhold information, and even defect on the deal (without blatantly lying), but a verifiable AI delegate presumably wouldn’t be able to do that?
I see no reason why an AI delegate shouldn’t be able to withhold information. I agree that people might want delegates that could do the other things too, but I think that it might be better for the human principal if it couldn’t—it can develop a reputation as trustworthy (in a way that’s hard for an individual human to develop enough of a reputation for because others don’t get enough track record).
I agree that there are significant concerns here! FWIW I’m more concerned about the adversarially-manipulated layer (at least at something needing attention now). I think that a lot of these applications could work with systems that aren’t much stronger than what we have today; but that getting effective misaligned scheming would require a significant step up in capabilities. (You might have weaker forms of misalignment, but I think that those are pretty similar to “the systems just aren’t really good enough yet”.)
We didn’t; although two of us were involved in running the AI for Human Reasoning fellowship, and some of the fellows on that did.
I think the reasons we didn’t go deeper on this are basically a mix of:
Eh, I’m not sure how much signal you get from the simple prototypes. Like for sure you can get some, but mostly what you’re testing is “Are the LLMs already good enough that they can be quite useful here even with minimal scaffolding?”
A lot of the research was done 6-9 months ago (when Claude Code was significantly weaker)
Questions of comparative advantage—it being unclear that we’re the best people to be exploring this (although I agree that Claude Code makes this more plausible than it would have been in the past)
We didn’t want to let the perfect be the enemy of the good—indeed in many ways Claude Code improvements make it more attractive to get out, since it’s more plausible that someone else will casually run with and develop one of these ideas
Yeah I think that would be an improvement over the current behaviour. I’d still probably prefer something very short (“LLM usage: zero/minimal/moderate/major”) which can be expanded if people want more texture.