Mechatronics Engineer, recently pivoted from work at a medical robotics startup (semi-autonomous eye surgery robot).
Ariel G.
Karma: 163
(have only skimmed the post, but sending this link in case it is relevant) - https://cltc.berkeley.edu/seeking-input-and-feedback-ai-risk-management-standards-profile-for-increasingly-multi-purpose-or-general-purpose-ai/
I am part of a team collaborating with him
It is still the EU, so those things aren’t going anywhere :) And in the discussion on real-time Biometric monitoring, “protection of democracy” was a main in talking point, making references to social scoring etc in China
EU AI Act passed vote, and x-risk was a main topic
Are there examples of standards in other industries where people were quite confused about what “safety” would require?
Yes, medical robotics is one I was involved in. Though there, the answer is often just wait for the first product to hit the market (there is nothing quite there yet, doing full autonomous surgery), and then copy their approach. As is, the medical standards don’t cover much ML, and so the companies have to come up with the reasoning themselves for convincing the FDA in the audit. Which in practice means many companies just don’t risk it, and do something robotic, but surgeon controled, or use classical algorithms instead of deep learning.
I’m not sure I agree with the conclusion but I like the overall analysis, I think it is very useful.
I’m confused, we make this caution compromise all the time—for example, medicine trial ethics. Can we go faster? Sure, but the risks are higher. Yes, that can mean that some people will not get a treatment that is developed a few years too late.
Another closer example is gain of function research. The point is, we could do a lot, but we chose not to—AI should be no different.
Seems to me that this post is a little detached from real world caution considerations, even if it isn’t making an incorrect point.
Might be better to look for a Prof doing interesting/relevant research, rather than specifically a PhD program
Ah right, I had that thought but wasn’t sure, makes sense!
From my playing with it, ChatGPT uses complex language even when told not to. In notion, there’s a AI assistant (GPT3 based) and it has a “simplify writing” feature. The outputs were still pretty verbose and had overly long sentences. Soon though, sure!
Well said, though I think your comment could use that advice :) Specific phrases/words I noticed: reign in, tendancy, bearing in mind, inhibit, subtlety, IQ-signal (?).
I’m non-native and I do know these words, but I’m mostly native level at this point (spent half my life in an English speaking country) I think many non-native speakers won’t be as familiar
I came across this some time ago through lesswrong - https://putanumonit.com/2016/02/03/015-dating_1/
might be what your looking for :)
As a (semi humorous) devils advocate, if we applied existential risk/longtermist ideas to non human animals, couldn’t the animal lives on Mars still be net positive, as they help their respective species flourish for billions of years, and reduce their risk of going extinct if they were only on earth?
I’m not sure I take this seriously yet, but it’s interesting to think about.
This was really well written! I appreciate the concise and to the point writing style, as well as a summary at the top.
Regarding the arguments, I think they make sense to me. Although this is where the whole discussion of longtermism does tend to stay pretty abstract, if we can’t actually put real numbers on it.
For ex, in the spirit of your example—does working on AI safety at MIRI prevent extinction, while assuming a sufficiently great future compared to, say, working on AI capabilities at OpenAI? (That is, maybe a misaligned AI can cause a greater future?)
I don’t think it’s actually possible to do a real calculations in this case, and so we make the (reasonable) base assumption that a future with alligned AI is better than a future with a misaligned AI, and go from there.
Maybe I am overly biased against longtermism either way, but in this example it seems to me like the problem you mention isnt really a real-world worry, but only really a theoretically possible pascals mugging.
Having said that I still think it is a good argument against strong longtermism
While this is important (clarifying of misinformation), I want to mention that I don’t think this takes away from the main message of the post. I think it’s important to remember that even with a culture of rationality, there are times when we won’t have enough information to say what happened (unlike in Scotts case), and for that reason Mayas post is very relevant and I am glad it was shared.
It also doesn’t seem appropriate to mention this post as “calling out”. While it’s legitimate to fear reputations being damaged with unsubstantiated claims, this post doesn’t strike me as doing such.
Great post, well explained!
I like the idea of having a more relatable message than “do the most good”, but I am not sure how much more relatable “do alot of good” is. To me it seems that there might not be that much of a difference between the two, at least in how they are used in day to day discussion (that is, applying a filter of “practicality” to the maximization problem).
For example, I thought it was common EA Knowledge that there are “top recommended cause areas” (on 80K), where some are higher on the list but with a big * of uncertainty. Theres also enough people to work on all of them, so there’s no need for a final judgement of a top 3, let alone 1 most important cause. In a way this could be a “macro” EA perspective—asking not what is the most good an individual could do, but what is the most good a group/society can do, with appropriate allocation between cause areas of high ITN.
I think EA can come across as a bit elitist to others, especially to people voulenteering in non-EA charities or trying to do good in “traditional” ways (doctors, Med-tech, activism, etc). Perhaps the “do alot of good” can help with that—but I still think it would come to similar conclusions in some cases. I have a friend who is volenteering in “Make a Wish” for the past 10 years, and I felt a little uneasy telling him about EA without offending him—though I was able to, and while he was intrigued I don’t think he was convinced.
I had a thought a while ago, that perhaps the world would be much better if there were alot of people committed to doing “at least a little good” , rather than a (relatively) small group of highly ambitious people doing “the most good”. However, perhaps there is room for that as a separate movement from EA. Plus, someone for sure needs to work on the “big things” too, which seems like a good niche for EA.
I really loved reading the Sequences, but I don’t think they are actually a practical rationality guide. (And EY also acknowledges this as far as I know).
I try to keep some cognitive biases in Anki as flashcards, and review some personal reflections and writings, some of which have some valuable rationality reminders, or articulate ideas in more relatable ways to my own life experience.
The site “Effectiviology” has some great practical cognitive bias/rationality practical tips. The biggest one I try to implement is externalizing my thought process—both by talking to others (from casual chat to formal design reviews), and by metacognition—keeping a written log of my thoughts as I work, especially when I get stuck.
There’s also the CFAR handbook, though I have yet to go though it.
Isn’t it possible that we already have 2 (or more) separate “Qualias”, i.e the left and right brain? But, being highly integrated, don’t result in any noticable experimental consequences in the way that Split brain patients do (it could also be that only one side is conscious, and the other side only seems) I’m thinking a bit in the way it’s described in the short story “Learning to be Me” by Greg Egan.
I also like the thought experiment in Shelly Kagans great YouTube lecture series (philosophy of death) - if we assumed the soul existed (and was “you” in the sense of conscious experience/qualia), you wouldn’t be able to tell if you are the same soul you were yesterday, or even 1 millisecond ago. Someone could in theory replace your “soul”, while keeping all of the memories intact, and the result would be subjectively the same.
To me this is a nice reminder that a single continuous self (or consciousness) in the way I like to think of it might not make much sense, or perhaps that other possibilities exist that fit the data.
One could imagine me being the right brain for one moment, the left brain for another, or being both at the same time—eliminating the contradiction from the Split brain experiment.
This was a great read! I liked the structure as well, which allowed me to jump across sections before deciding to read the whole thing.
I got into meditation a few years ago and recently did a Vipassana 10 day retreat. I guess it’s hard to quantify systematic change but it is interesting to imagine what would happen if meditation was taught in schools—though I prefer it in non-spiritual form, just pure practice.
I’m still mostly a beginner in EA, so I am not sure if I’m qualified yet to judge your views on longtermism but it seems to make sense to me.
Regarding the book—perhaps it wouldn’t be a bad thing that it is more geared towards a “popular audience”. It is too easy to aim too high, and that perhaps makes much of EA writing somewhat impenetrable to many people who aren’t yet onboard.
I did a 10 day Vipassana meditation retreat, after meditating casually 10min daily (I had streaks of several months where I wouldn’t miss a day, then months without) for ~4 years. I used the Headspace and Waking up apps
The 10min sessions were helpful especially if I was consistent, in feeling less anxious/irritated and more happy. I ended up liking the Waking Up app more since he explains the theory as well.
But, those basically disappear when compared to the 10 day retreat, where we would do basically a full day of meditation (split into 1 and 2h segments), and have nothing to distract—no notebook, no phone/computer, book, or even talking to others. This level of immersion was very much necessary for getting the hang of the practice, being alone with your thoughts and experience. I wish I could get this level of immersion into other projects!
The first 3 days were just noticing the breath (at the nostrils) gradually narrowing the zone of focus each day. And then the rest of the days made use of this increased sensitivity to body sensations. It got to the point where I could feel a vibrating, flowing sensation through the entire body—perhaps the bloodflow, perhaps the muscular electrical signals...
I finished the course completely changed, though this faded somewhat afterwards due to lack of practice.
But whenever I do meditate now (ideally for a full hour but even 20min is ok) I am still able to get into a deep state of focus, and even once every 1-2 weeks definitely improves my mental state. Though I’d imagine doing 1h daily is much better. On the few days after the retreat I could focus super easily without headphones. It was like getting effortlessly into a flow state, where usually it would only happen once every few days.
Overall would highly recommend! I did through the Dhamma organization, which is donation based (i.e pay what you wish) and completely voulenteer run.
This makes sense to me, good writeup!