You can see the amount of matching remaining in real time at the top of this post.
Am I missing it somehow, or is this amount not actually at the top of the post? Does anyone know how to find out how much is left?
TW123
Karma: 3,714
ML Safety Scholars Summer 2022 Retrospective
What is the EA Charter? I haven’t heard of it before. Is there a link to a description of it?
And I think Neel was asking who is organizing the program’s day to day operations, not just who is funding it.
(I’m sorry if this comment is not accessible to many people. I also might have missed something in reading the paper, please do let me know if that’s the case!)
I do think many EAs who haven’t studied philosophy of mind probably (implicitly) believe functionalism a bit too credulously. It doesn’t matter very much right now, but maybe it will later.
But I’m not really convinced by this paper. I skimmed it and here are some initial thoughts:
A TV screen flickers between frames. If you watch an old or broken TV, you’ll notice this, but once the frame rate is high enough, you experience it continuously. Of course, it’s not actually continuous. A similar phenomenon occurs with pixels: they are discrete, but we experience them continuously.
You say that light is continuous, not discrete, but light (and every other elementary particle) behaves as discrete packets (photons) as well as waves. This makes me wonder if there is a real difference between analog and digital at extremely high levels of fidelity.
You give the example of mechanical watches, but I’m pretty sure the ones that seem continuous (i.e. have “sweeping hands” rather than ticking) are actually just higher frequency, and still move discretely rather than continuously. See here. Again, we experience them continously.
You mention hue, saturation, and brightness. We represent these in a computer just fine with only 24 (8 for each) bits most of the time, and we still get most of the same experiences. There are higher-fidelity color schemes but we barely notice this.
You argue in the paper that neurons are analog rather than digital. I agree that neurons are not simply on/off. But again, does this show anything about whether you can properly represent a neuron if you just allocate more than one bit to it? Something I find particularly problematic is that the evidence that neurons are analog presumably came from measuring action potentials and analyzing them. But aren’t the action potential measurements not represented digitally in a computer? How could that show evidence of analog behavior?
In the paper you have this caveat (and further explanation), which seems to dismantle many of my objections above:
On the Lewis-Maley view we adopt, to be analog does not require continuity, but only monotonic covariation in magnitude between the representation and what is represented. That increase or decrease can happen in steps, or it can happen smoothly (i.e., discretely or continuously). For example, consider the fuel gauges found in cars. Older cars often have a physical dial that more-or-less continuously moves from ‘F’ to ‘E’ as fuel is consumed; this dial is an analog representation of the amount of fuel in the car, because as fuel decreases, so does the literal angle of the dial. Newer cars often have a different way of displaying fuel. Instead of a physical dial, fuel is displayed as a bar graph on an LED or LCD display. Importantly, that bar graph is composed of discrete segments. Nevertheless, this is still an analog representation of the amount of fuel: as fuel decreases, the number of segments decreases. In contrast, we can imagine a fuel gauge that simply displays the number of gallons (or liters) of fuel in the tank (e.g., ‘6.5’ for six and a half gallons). Here, as fuel decreases, the digits displayed do not increase or decrease (the way they would if we changed the font in a paper): they simply change.
The suggestion here is that if we encode something in unary (number of on bits = magnitude) it is qualitatively different from being encoded in binary or decimal. This is not a straightforward claim. In your paper, it relies on panpsychism with microexperience: roughly speaking, the idea that consciousness arises from microconsciousness of fundamental particles. I think you’ve done a pretty decent job of arguing the conclusion given that premise, but I find the premise pretty unconvincing myself, and anyone else who does is similarly unlikely to be convinced by your argument.
TLDR: Yes, the magnitude of discrete representations of bits is completely contingent and arbitrary. But to say that two functionally identical organisms that are also isomorphic (usually more than we could ever ask for) are different in terms of the consciousness they produce seems to require some kind of microphenomenal laws. If you don’t believe in such laws, you shouldn’t believe this argument.
Unfortunately, people (and this includes AI researchers) tend to hear what they want to hear, and not what they don’t want to hear. What to call this field is extremely dependent on the nature of those misinterpretations. And the biggest misinterpretation right now does not appear to be “oh so I guess we need to build impotent systems because they’ll be safe”.
“Alignment” is already broken, in my view. You allude to this, but I want to underscore it. Instruct GPT was billed as “alignment”. Maybe it is, but it doesn’t seem to do any good for reducing x risk.
“Safety”, too, lends itself to misinterpretation. Sometimes of the form “ok, so let’s make the self-driving cars not crash”. So you’re not starting from an ideal place. But at least you’re starting from a place of AI systems behaving badly in ways you didn’t intend and causing harm. From there, it’s easier to explain existential safety as simply an extreme safety hazard, and one that’s not even unlikely.
If you tell people “produce long term near optimal outcomes” and they are EAs or rationalists, they probably understand what you mean. If they are random AI researchers, this is so vague as to be completely meaningless. They will fill it in with whatever they want. The ones who think this means full steam ahead toward techno utopia will think that. The ones who think this means making AI systems not misclassify images in racist ways will think that. The ones who think it means making AI systems output fake explanations for their reasoning will think that.
Everyone wants to make AI produce good outcomes. And you do not need to convince the vast majority of researchers to work on AI capabilities. They just do it anyway. Many of them don’t even do it for ideological reasons, they do it because it’s cool!
The differential thing we need to be pushing on is AI not creating an existential catastrophe. In public messaging (and what is a name except public messaging?) we do not need to distract with other considerations at this present moment. And right now, I don’t think we have a better term than safety that points in that direction.
[I’m a contest organizer but I’m recusing myself for this because I personally know Andrew.]
Thanks for writing! A few minor points (may leave more substantive points later).
In 2014, one survey asked the 100 most cited living AI scientists by what year they saw a 10%, 50%, and 90% chance that HLMI would exist
There is updated research on this here (survey conducted 2019) and here (2022; though it’s not a paper yet, so might not be palatable for some people).
Only 17% of respondents said they were never at least 90% confident HLMI would exist.
I think this is a typo.
Considering all of these scenarios together, 80,000 Hours’ team of AI experts estimates that “the risk of a severe, even existential catastrophe caused by machine intelligence within the next 100 years is something like 10%.”
I don’t think I would cite 80,000 hours, as that particular article is older. There is a newer one recently, but it still seems better for ethos to cite something that looks like a paper. You could possibly cite Carlsmith or the survey above, which I think says the median researcher assigns 5% chance of extinction-level catastrophe.
I have posted on the forum quite a few times. It has sometimes been part of my job to write posts for the forum. I think that I am generally a strong writer, and I rarely feel imposter syndrome.
But even I find it intimidating to post on the forum. Some of what is written here is just really, really well written and thought out. It’s intimidating to think you are posting alongside those posts! I think in my case though, it’s a good thing I’m intimidated. If I wasn’t at all intimidated, I would innundate the forum with random ramblings, because my general bar for sharing thoughts in other contexts (e.g. with my friends in informal settings) is very low. I was the kind of person who used to write emails to administrators in middle school asking them for changes to the school’s caterpillar abatement policy (yes, actually). I am the kind of person who writes 50% of the texts in the group chat. And so on. You do not want somebody like me constantly posting on the EA forum!
I have several full length posts I’ve written that in the past year that I haven’t put up. I could post them, and maybe somebody could get value from them, but they aren’t finished according to my standards. And I won’t post them until they are, if they ever are.
I suspect there are many others who are the opposite of me. Their thoughts are just as good, or better, but their general bar for posting things is way too high and they should work on becoming less intimidated. That’s why it’s really hard to give generalized advice. If you say, “please, post, don’t worry about it!” people like me will post way too much. If you say, “the bar is so high, really think about it” people who aren’t like me will post way too little. That’s why the most helpful thing in my view is to just ask some friends for personalized advice. They probably have a pretty good idea of which side of this spectrum you might fall on.
First just want to flag that I don’t have extremely high confidence in the last section in general, it wasn’t nearly as researched as the rest.
I agree there are a number of disanalogies, most specifically that it does seem like biological weapons are straightforwardly less useful than lethal autonomous weapons. In this sense maybe LAWS are more like chemical weapons, which were at least claimed to be useful (though probably still not as useful), but were also eventually banned.
I’m not sure I agree about the creep factor. I think it’s possible to make LAWS “creepy;” at least, watching the Slaughterbots documentary felt creepy to me. I think it’s true they could be “cooler” though; I can’t imagine a biological weapon being cool.
I don’t believe you explicitly defined “CBW.”
Thanks, fixed. It stands for “chemical and biological weapons.”
“A Creepy Feeling”: Nixon’s Decision to Disavow Biological Weapons
Yeah I wasn’t sure which grants you were referring to (haven’t looked through them all), but indeed that doesn’t seem to be explained by what I said.
I agree that EA already selects for high SES people and that offering funding for them to organize a group doesn’t negate this problem. Other steps are also needed. However, I know quite a few anecdotal cases of group organizers being able to organize more than they otherwise would have because they were being paid, and so this policy does concretely make some difference.
I’m not involved with EA funds, but some university group organizers have taken semesters of leave in the past to do group organizing full time for a semester. If you assume their term is 14 weeks, then that’s 14*40=560 hours of work. At $20/hr, that’s more than $10,000. And I think it is pretty reasonable to request more than $20/hr (various funding sources have previously offered something like $30/hr).
In general, nowadays, many group organizers are not volunteers and are paid for their part time work (if they are not full time, this shouldn’t amount to five figures for one semester though). I think this is a good thing, since many university students simply cannot afford to take a volunteer job with a commitment of 10+ hours per week, and I wouldn’t want EA groups only run by people who are rich enough that that’s feasible.
Ironically, I don’t think the comment I made that Thomas included above rises to my usual standard for my posts. You can see in the thread there are some issues with it, and if I were going to make a post about it I’d probably collect some data. I tend to have a lower standard for making comments.
I think you’re right and it’s worth thinking about these cases. That being said, I think tail impact is going to come from people who have good judgement including on how to develop their careers, open new opportunities, and select future jobs across their career. It’s unclear to me which group the 80k job board should be catering to, but plausibly those most extremely self-motivated people don’t need a job board to show them their options.
Thanks for posting, I think you have some great thoughts! I generally agree with the spirit of this post.
I do think it’s worth noting that what people view as “excessive” or “not frugal” is not always in line with reality. For instance, many people find it “excessive” to order a $10 uber to save 30 minutes, but don’t find it excessive to wait until the very last minute to book a train ticket such that the price has increased from $20 to $120. In my view the latter is more excessive. This is just to say that the actual numbers matter, rather than purely the vibes of the spending. But as you say, the vibes matter too for community culture.
If you haven’t already seen them, you might be interested in some other posts on similar issues:
I think this is putting too much on 80k. They have hundreds of jobs in many different areas listed on their website, and it’s a very daunting task to evaluate each one of them when the evaluators are generalists who often have to rely on (conflicting) opinions from people in the community. On top of that, what is career capital for one person could plausibly be direct impact for some other person, so it doesn’t really seem one size fits all.
If somebody can’t evaluate jobs on the job board for themselves, I’m not that confident that they’ll take a good path regardless. People have a tendency to try to offload thinking to places like 80k, and I actually think it could be bad if 80k made it easier to do that on extremely fine grained topics like individual jobs.
I do like the idea of having comments on particular jobs. And it also would be good for 80k to be more clear they don’t expect all these jobs to necessarily have direct impact.
I don’t know whether it’s the case that many people on the internet are looking at the job board and deciding which jobs to apply to when they don’t have a strong engagement with EA ideas, and that these sorts of people are the types who would actually get the jobs. If that’s the case (80k would know better than me), then I think it maybe does make sense to restrict to jobs that aren’t going to be bad if such a person gets them. That seems like an empirical question.
I just think anguish is more likely than physical pain. I suppose there could be physical pain in a distributed system as a result of certain nodes going down.
It’s actually not obvious to me that simulations of humans could have physical pain. Seems possible, but maybe only other orders of pain like anguish and frustration are possible.
Students have by far the most flexibility in their careers. It’s not uncommon for university students (in the US—maybe some other age in other countries) to do things like switch their major from biology to economics; except in very rare circumstances, 40 year old biologists do not become economists. If you suppose that certain high-impact career paths require very special skills not common in the population, then you might need people to develop those skills early rather than try to find people who already have them. There are some areas of EA that probably do have this property, though the popular perception of it is maybe overblown.
I do think it would be good if there could be more experienced older people in EA, since I think there are probably many people out there with highly relevant and useful experience who haven’t heard of EA but would be receptive.
In philosophy of mind the theory of functionalism defines mental states as causal structures. So for example, pain is the thing that usually causes withdrawal, avoidance, yelping, etc. and is often caused by e.g. tissue damage. If you see pain as the “tissue damage signaling” causal structure, then you could imagine insects also having this as well, even if there is no isomorphism. It’s hard to imagine AI systems having this, but you could more easily imagine AI systems having frustration, if you define it as “inability to attain goals and realization that such goals are not attained”. The idea of an isomorphism is required by the theory of machine functionalism, which essentially states that two feelings are the same if they are basically the same Turing machine running. But humans could be said to be running many Turing machines, and besides no two humans are running the same Turing machine, and comparing states across two Turing machines doesn’t really make sense. So I’m not very interested in this idea of strict isomorphism.
But I’m not fully onboard with functionalism of the more fuzzy/”squishy” kind either. I suppose something could have the same causal structures but not really “feel” anything. Maybe there is something to mind body materialism: for instance pain is merely a certain kind of neuron firing. In that case, we should have reason to doubt that insects suffer if they don’t have those neurons. I certainly am one to doubt that insects suffer, but on the more functionalist flavor of thinking I don’t. So I’m pretty agnostic. I’d imagine I might be similarly agnostic towards AI, and as such wouldn’t be in favor of handing over the future to them and away from humans, just as I’m not in favor of handing over the future to insects.
To answer the second question, I think of this in a functionalist way, so if something performs the same causal effects as positive mental states in humans, that’s a good reason to think it’s positive.
For more I recommend Amanda Askell’s blog post or Jaegwon Kim’s Philosophy of Mind textbook.
Since animals share many similar biological structures with us and evolved similarly, it’s relatively possible to make claims about their sentience by analogy to our own. Claims about AI sentience are far harder to verify. One could imagine the possibility of an AI that behaves as if sentient but isn’t really sentient. This gives significantly more reason to be wary of just handing everything over to AI systems, even if you are a total hedonistic utilitarian.
I also agree with others that building a sentient AI with a positive inner life doesn’t seem remotely easy.
Ah makes sense, thanks!