Do I view every problem in my life and my community as analogous or bearing surprising similarity to the alignment problem
This made me laugh.
But also, as I said at the top of the post, I actually do think the alignment problem does bear surprising similarities to other things, but this is mainly because of general ideas about complex systems pertain to both.
Welcome to the forum!
I’ve done research in reinforcement learning, and I can say this sort of behavior is very common and expected. I was working on a project once and incorrectly programmed the reward function, leading the agent to kill itself rather than explore the environment so that it could avoid an even greater negative reward from sticking around. I didn’t consider this very notable, because when I thought about the reward function, it was obvious this would happen.
Here is a spreadsheet with a really long list of examples of this kind of specification gaming. I suspect the reason your grant was rejected is because if the agent did as you suspect it would, this wouldn’t provide much original insight beyond what people have already found. I do think many of the examples are in “toy” environments and it might be interesting to observe more behavior like this in more complex enviroments.
It might still be useful for your own learning to implement this yourself!
The article now says the FAQ was “put together by the effective altruism forum.” This is not true either, nor does it make sense. If they would like to copy and paste:
“A user on the effective altruism forum posted an FAQ.”
This makes it sound a lot less official and less like something you’d want to quote in a publication. But that’s because it is in fact not official at all.
I interpreted this post as the author saying that they thought general AI capabilities would be barely advanced by this kind of thing, if they were advanced by it at all. The author doesn’t seem to suggest building an AGI startup, but rather some kind of AI application startup.
I’m curious if you think your reasoning applies to anything with a small chance of accelerating timelines by a small amount, or if you instead disagree with the object-level claim that such a startup would only have a small chance of accelerating timelines by a small amount.
The Centre for Effective Altruism has had to deal with a lot of questions about Bankman-Fried since FTX’s collapse. Here’s an FAQ it put together.
This is not true. The FAQ was not put together by CEA, and so far as I know Hamish has no affiliation with CEA (nor did he ever claim to).
This is an interesting post, and I’m glad you wrote it.
People are driven by incentives, which can be created with cash in a variety of ways
I agree with these ways, but I think it’s quite hard to manage incentives properly. You mention DARPA, but DARPA is a major bureacracy comprised of people who are aligned to their own incentive structures, and ultimately the most powerful organization in the world (the US government). Such a thing does not exist in AI safety—not even close. Money would certainly help with this, but it certainly can’t just be straightforwardly turned into good research.
Where is that money going to come from? How are they going to fine-tune the latest models when OpenAI is spending billions?
It’s unclear to me that having EA people starting an AI startup is more tractable than convincing other people that the work is worth funding. It certainly works faster to convince people who already have money now, vs. creating people who might make money later. I don’t really have a strong opinion, but this doesn’t seem wholly justified.
It’s frustratingly difficult to predict what will actually be useful for AI Safety...But money is flexible. It’s hard to imagine a world where another billion doesn’t come in handy.
I don’t see how the flexibility of money makes any difference? Isn’t it frustratingly difficult to predict which uses of money will actually be useful for AI safety? In that case, you still have the same problem.
The blog post says ChatGPT is trained with proximal policy optimization. This documentation says text-davinci-003 was trained with PPO, but not text-davinci-002.
However, it is interesting what you’re saying about the request payloads, because this seems to be contradictory. So I’m not quite sure anymore. It’s possible that ChatGPT was trained with PPO on top of the non-PPO text-davinci-002.
text-davinci-003 (which is effectively ChatGPT) was a bit better than text-davinci-002 anecdotally and when I benchmarked it on TriviaQA. It was only released about a week before ChatGPT so it’s not necessarily unreasonable to lump them together. If you do, then the interface isn’t the only change one might associate with ChatGPT.
This is an important consideration, thanks for bringing it up! I pretty much agree with all of it.
Thank you for writing this. I’m sure it was very difficult to do, and so I really appreciate it.
Effective altruism has an emotions problem.
I strongly agree with this. Have you seen Michel’s post on emotional altruism? It doesn’t get to your points specifically, but is similarly the need for more open emotion in the movement.
I also to add something that, in my experience, cannot really be ignored when speaking about the expression of emotion in particular in EA. EA has a lot of people who are on the autism spectrum who may relate to emotion differently, particularly in the way they speak about it. There are others who aren’t on the spectrum but similarly have a natural inclination to be less or differently publicly emotional. EA/rationality can feel rather welcoming (like “my people”) to people like this (which is good—welcomingness of people whose brains work differently is good) and this may produce a feedback loop. This is not at all to deny your recommendations on this particular area. Rather, it is to acknowledge that some proportion (far from all) of what you call the “emotions problem” is probably just people being themselves in a way we should find acceptable, which means that I am a bit more confused about how to best address it.
I sympathize with this. My AI safety career was started when I turned down a tech company return offer for the chance of maybe later getting an AI safety research internship. The tech company paid more than triple (seeing as AI safety internship was in academia). I ended up getting the AI safety internship, but I almost didn’t!
Not everyone wants to take that risk. And not everyone can afford to, either.
It’s not obvious to me that January is correct. For instance, at Yale, more than two thirds of CS majors received their full time job offers in December or earlier (most of these are likely in the tech industry). You often don’t have very long to reply to these offers so I wouldn’t be surprised if many had to accept prior to January.
I agree that April is later than nearly anywhere else; but I’m also not convinced that January would be better than November. I basically agree with this post but I think people should do their own research on what they’re competing with rather than just deciding on January on the basis of this post.
I’m not sure why you think 10k people signing it matters (as opposed to which people), or why you assume the factor is 10^(-6). I read this post and I’m not sure the estimate is anywhere close. By that I mean that it could be tens of orders of magnitude off, or even that the sign of the estimate could be wrong and it would be bad for this to get many signatures.
I think it probably would have been more useful to people if you’d discussed more generally why you think this letter might have a good effect rather than trying to quantify existential risk reductions from individual EA forum users’ signatures (which I take to be extremely intractable).
Ah makes sense, thanks!
You can see the amount of matching remaining in real time at the top of this post.
Am I missing it somehow, or is this amount not actually at the top of the post? Does anyone know how to find out how much is left?
What is the EA Charter? I haven’t heard of it before. Is there a link to a description of it?
And I think Neel was asking who is organizing the program’s day to day operations, not just who is funding it.
(I’m sorry if this comment is not accessible to many people. I also might have missed something in reading the paper, please do let me know if that’s the case!)
I do think many EAs who haven’t studied philosophy of mind probably (implicitly) believe functionalism a bit too credulously. It doesn’t matter very much right now, but maybe it will later.
But I’m not really convinced by this paper. I skimmed it and here are some initial thoughts:
A TV screen flickers between frames. If you watch an old or broken TV, you’ll notice this, but once the frame rate is high enough, you experience it continuously. Of course, it’s not actually continuous. A similar phenomenon occurs with pixels: they are discrete, but we experience them continuously.
You say that light is continuous, not discrete, but light (and every other elementary particle) behaves as discrete packets (photons) as well as waves. This makes me wonder if there is a real difference between analog and digital at extremely high levels of fidelity.
You give the example of mechanical watches, but I’m pretty sure the ones that seem continuous (i.e. have “sweeping hands” rather than ticking) are actually just higher frequency, and still move discretely rather than continuously. See here. Again, we experience them continously.
You mention hue, saturation, and brightness. We represent these in a computer just fine with only 24 (8 for each) bits most of the time, and we still get most of the same experiences. There are higher-fidelity color schemes but we barely notice this.
You argue in the paper that neurons are analog rather than digital. I agree that neurons are not simply on/off. But again, does this show anything about whether you can properly represent a neuron if you just allocate more than one bit to it? Something I find particularly problematic is that the evidence that neurons are analog presumably came from measuring action potentials and analyzing them. But aren’t the action potential measurements not represented digitally in a computer? How could that show evidence of analog behavior?
In the paper you have this caveat (and further explanation), which seems to dismantle many of my objections above:
On the Lewis-Maley view we adopt, to be analog does not require continuity, but only monotonic covariation in magnitude between the representation and what is represented. That increase or decrease can happen in steps, or it can happen smoothly (i.e., discretely or continuously). For example, consider the fuel gauges found in cars. Older cars often have a physical dial that more-or-less continuously moves from ‘F’ to ‘E’ as fuel is consumed; this dial is an analog representation of the amount of fuel in the car, because as fuel decreases, so does the literal angle of the dial. Newer cars often have a different way of displaying fuel. Instead of a physical dial, fuel is displayed as a bar graph on an LED or LCD display. Importantly, that bar graph is composed of discrete segments. Nevertheless, this is still an analog representation of the amount of fuel: as fuel decreases, the number of segments decreases. In contrast, we can imagine a fuel gauge that simply displays the number of gallons (or liters) of fuel in the tank (e.g., ‘6.5’ for six and a half gallons). Here, as fuel decreases, the digits displayed do not increase or decrease (the way they would if we changed the font in a paper): they simply change.
The suggestion here is that if we encode something in unary (number of on bits = magnitude) it is qualitatively different from being encoded in binary or decimal. This is not a straightforward claim. In your paper, it relies on panpsychism with microexperience: roughly speaking, the idea that consciousness arises from microconsciousness of fundamental particles. I think you’ve done a pretty decent job of arguing the conclusion given that premise, but I find the premise pretty unconvincing myself, and anyone else who does is similarly unlikely to be convinced by your argument.
TLDR: Yes, the magnitude of discrete representations of bits is completely contingent and arbitrary. But to say that two functionally identical organisms that are also isomorphic (usually more than we could ever ask for) are different in terms of the consciousness they produce seems to require some kind of microphenomenal laws. If you don’t believe in such laws, you shouldn’t believe this argument.
Unfortunately, people (and this includes AI researchers) tend to hear what they want to hear, and not what they don’t want to hear. What to call this field is extremely dependent on the nature of those misinterpretations. And the biggest misinterpretation right now does not appear to be “oh so I guess we need to build impotent systems because they’ll be safe”.
“Alignment” is already broken, in my view. You allude to this, but I want to underscore it. Instruct GPT was billed as “alignment”. Maybe it is, but it doesn’t seem to do any good for reducing x risk.
“Safety”, too, lends itself to misinterpretation. Sometimes of the form “ok, so let’s make the self-driving cars not crash”. So you’re not starting from an ideal place. But at least you’re starting from a place of AI systems behaving badly in ways you didn’t intend and causing harm. From there, it’s easier to explain existential safety as simply an extreme safety hazard, and one that’s not even unlikely.
If you tell people “produce long term near optimal outcomes” and they are EAs or rationalists, they probably understand what you mean. If they are random AI researchers, this is so vague as to be completely meaningless. They will fill it in with whatever they want. The ones who think this means full steam ahead toward techno utopia will think that. The ones who think this means making AI systems not misclassify images in racist ways will think that. The ones who think it means making AI systems output fake explanations for their reasoning will think that.
Everyone wants to make AI produce good outcomes. And you do not need to convince the vast majority of researchers to work on AI capabilities. They just do it anyway. Many of them don’t even do it for ideological reasons, they do it because it’s cool!
The differential thing we need to be pushing on is AI not creating an existential catastrophe. In public messaging (and what is a name except public messaging?) we do not need to distract with other considerations at this present moment. And right now, I don’t think we have a better term than safety that points in that direction.
[I’m a contest organizer but I’m recusing myself for this because I personally know Andrew.]
Thanks for writing! A few minor points (may leave more substantive points later).
In 2014, one survey asked the 100 most cited living AI scientists by what year they saw a 10%, 50%, and 90% chance that HLMI would exist
There is updated research on this here (survey conducted 2019) and here (2022; though it’s not a paper yet, so might not be palatable for some people).
Only 17% of respondents said they were never at least 90% confident HLMI would exist.
I think this is a typo.
Considering all of these scenarios together, 80,000 Hours’ team of AI experts estimates that “the risk of a severe, even existential catastrophe caused by machine intelligence within the next 100 years is something like 10%.”
I don’t think I would cite 80,000 hours, as that particular article is older. There is a newer one recently, but it still seems better for ethos to cite something that looks like a paper. You could possibly cite Carlsmith or the survey above, which I think says the median researcher assigns 5% chance of extinction-level catastrophe.
[This comment isn’t meant to signal any opinion about the rest of your post.]
Carlsmith’s report in particular is highly interdisciplinary and draws on technical AI, economics, and also philosophy. It doesn’t have much in the way of technical AI or economics claims. It’s not really clear who would be most qualified to write this, but in general a philosopher doesn’t seem like such a bad choice. In fact, I’d think the average philosopher with strong quantitative skills would be better at this than the average economist or certainly AI researcher.
Whether a more experienced philosopher should have done it is another question, but I’d imagine that even with money Open Phil cannot summon very experienced experts to write reports for them at the drop of a hat.