Volunteer at EA Finland / professional data scientist / confused about AI safety / interested in communications
Ada-Maaria Hyvärinen
Don’t be sorry! Feedback on language and grammar is very useful to me, since I usually write in Finnish. (This is probably the first time since middle school that I’ve written a piece of fiction in English.)
Apparently the punctuation slightly depends on whether you are using British or American English and whether the work is fiction or non-fiction (https://en.wikipedia.org/wiki/Quotation_marks_in_English#Order_of_punctuation ). Since this is fiction, you are in any case totally right about the commas going inside the quotes, and I will edit accordingly. Thanks for pointing this out!
I tried out a couple of different ones and iterated based on feedback.
One ending I considered would have been just leaving out the last paragraph and linking to GiveWell like this:“Besides,” his best friend said. “If you actually want to save a life for 5000 dollars, you can do it in a way where you can verify how they are doing it and what they need your money for.”
“What do you mean?” he asked, now more confused than ever.
I also considered embedding the link explicitly in the story like this:
“Besides,” his best friend said. “If you actually want to save a life for 500 dollars, you can do it in a way where you can verify how they are doing it and what they need your money for.”
“What do you mean?” he asked, now more confused than ever.
”I’ll send you a link”, she said.
And the link she send him was this: https://www.givewell.org/However, some of my testers found that this also broke the flow and that moving the link “outside” the story gave a less advertisement-like feeling.
And I also tried an ending that would wrap up the story more nicely (at this point the whole story was around 40% longer and not that well-edited in general):
“You know there are organizations that save people from dying of preventable illness and poverty,” she said. “The best ones can actually save a life for around that much, maybe even less.”
“But how do I know what those organizations are and how much they actually need to save somebody from dying?” he asked. “That sounds even more complicated than coming up with questions about side effects to the fey.”
“You don’t have to do that all by yourself,” she said. “There are people who are working on this stuff. You can see if you agree with their reasoning and conclusions, and then make your own decisions.”
This could be something. For a split second, he wished she wouldn’t have told him that. If what she said was true, he would have to make a choice, again. But it was better to know than to not know. And suddenly the thought of actually being able to save a person who would otherwise die was so overwhelming he had a hard time wrapping his head around it.
“I guess I don’t really like making decisions,” he said.
“I feel you,” she said. “But if you don’t make a choice, that’s actually a decision, too. It just means you chose to do nothing.”
“Yeah,” he said. “I’ve noticed.”
The fey were still in the woods, and would be in the woods, maybe forever. It didn’t matter. Anyway, he had to choose. But at least he could find out what his options were.
This longer ending was most liked by readers that were already quite familiar with EA, so I decided to not go for it, since I wanted to write for people who have not thought and discussed about EA that much yet. But of course, my pool of proof-readers was not that big and everyone was at least somewhat familiar with EA, even if not involved in the movement. It would be interesting to get feedback from total newbies.
This one is nice as well!
Personally I like the method of embedding the link in the story, but since a many in my test audience considered it off-putting and too advertisement-like I thought it it better to trust their feedback, since I obviously personally already agree with the thought I’m trying to convey with my text. But like I said I’m not certain what the best solution is, probably there is no perfect one.
glad to hear you like it! :)
With EA career stories I think it is important to to keep in mind that new members might not read them the same way as more engaged EAs who already know what organization is considered cool and effective within EA. When I started attending local EA meetups I met a person who worked at OpenPhil (maybe as a contractor? I can’t remember the details), but I did not find it particularly impressive because I did not know what OpenPhilanthropy was and assumed the “phil” stood for “philosophy”.
thanks Aayush! Edited the sentence to be hopefully more clear now :)
Thanks for the nice comment! Yes, I am quite uncomfortable with uncertainty and trying to work on that. Also, I feel like by now I am pretty involved in EA and ultimately feel welcome enough to be able to post a story like this in here (or I feel like EA apprechiates different views enough despite also feeling this pressure to conform at the same time).
Interesting to hear your personal opinion on the persuasiveness of Superintelligence and Human Compatible! And thanks for designing the AGISF course, it was useful.
I think you understood me in the same way than my friend did in the second part of the prolog, so I apparently give this impression. But to clarify, I am not certain that AI safety is impossible (I think it is hard, though), and the implications of that depend a lot on how much power the AI systems will be given at the end, and what part of the damage they might cause is due to them being unsafe and what for example misuse, like you said.
To clarify, my friends (even if they are very smart) did not come up with all AI safety arguments by themselves, but started to engage with AI safety material because they had already been looking at the world and thinking “hmm, looks like AI is a big thing and could influence a lot of stuff in the future, hope it changes things for the good”. So they got quickly on board after hearing that there are people seriously working on the topic, and it made them want to read more.
Glad it may have invoked some ideas for any discussions you might be having at Israel :) For us in Finland, I feel like I at least personally need to get some more clarity on how to balance EA movement building efforts and possible cause priorization related differences between movement builders. I think this is non-trivial because forming a consensus seems hard enough.
Curious to read any object-level response if you feel like writing one! If I end up writing any “Intro to AI Safety” thing it will be in Finnish so I’m not sure if you will understand it (it would be nice to have at least one coherent Finnish text about it that is not written by an astronomer or a paleontologist but by some technical person).
Generally, I find links a lot less frustrating if they are written by the person who sends me the link :) But now I have read the link you gave and don’t know what I am supposed to do next, which is another reason I sometimes find linksharing a difficult means of communication. Like, do I comment on specific parts on your post, or describe how reading it influenced me, or how does the conversation continue? (If you find my reaction interesting: I was mostly unmoved by the post, I think I had seen most of the numbers and examples before, there were some sentences and extrapolations that were quite off-putting for me but I think “minimalistic” style was nice.)
It would be nice to call and discuss if you are interested.
Thanks for giving me permission, I guess can use this if I need ever the opinion of “the EA community” ;)
However, I don’t think I’m ready to give up on trying to figure out my stance on AI risk just yet, since I still estimate it is my best shot in forming a more detailed understanding on any x-risk, and understanding x-risks better would be useful for establishing better opinions on other cause priorization issues.
That’s right, thanks again for answering my question back then!
Maybe I formulated my question wrong but I understood from your answer that you got first interested in AI safety, and only then on DS/ML (you mentioned you had had a CS background before but not your academic AI experience). This is why I did not include you in this sample of 3 persons—I wanted to narrow the search to people who had more AI specific background before getting into AI safety (not just CS). It is true that you did not mention Superintelligence either, but interesting to hear you also had a good opinion on it! If I would have known both your academic AI experience and that you liked Superintelligence I could have made the number to 4 (unless you think Superintelligence did not really influence you, then it would be 3 out of 4).
You were the only person who answered my PM but stated they got into AI safety before getting to DS/ML. One person did not answer, and the other 3 that answered stated they got into DS/ML before AI safety. I guess there are more than 6 people with some DS/ML background on the course channel but also know not everyone introduced themselves, so the sample size is very anecdotal anyway.
I also used the Slack to ask for recommendations of blog posts or similar stories on how people with DS/ML backgrounds got into AI safety. Aside of recommendations on who to talk on the Slack, I got pointers to Stuart Russell’s interview on Sam Harris’ podcast and a Yudkowsky post.
I’m still quite uncertain on my beliefs but I don’t think you got them quite right. Maybe a better summary is that I am generally pessimistic about both humans being ever able to create AGI and especially about humans being able to create safe AGI (it is a special case so it should probably be harder than any AGI). I also think that relying a lot on strong unsafe systems (AI powered or not) can be an x-risk. This is why it is easier to me to understand why AI governance is a way to try to reduce x-risk (at least if actors in the world want to rely on unsafe systems, I don’t know how much this happens but I would not find it very surprising).
I wish I had a better understanding on how x-risk probabilities are estimated (as I said I will try to look into that) but I don’t directly understand why x-risk from AI would be a lot more probable than, say, biorisk (that I don’t understand in detail at all).
Like I said it is based on my gut feeling, but fairly sure.
Is it your experience that adding more complexity and concatenating different ML models results to better quality and generality and if so, in what domains? I would have the opposite intuition especially in NLP.
Also, do you happen to know why “prosaic” practices are called “prosaic”? I have never understood the connection to the dictionary definition of “prosaic”.
I feel like everyone I have ever talked about AI safety with would agree on the importance of thinking critically and staying skeptical, and this includes my facilitator and cohort members from the AGISF programme.
I think the 1.5h discussion session between 5 people who have read 5 texts does not allow really going deep into any topics, since it is just ~3 minutes per participant per text on average. I think these kind of programs are great for meeting new people, clearing misconceptions and providing structure/accountability on actually reading the material, but they by nature are not that good for having in-depth debates. I think that’s ok, but just to clarify why I think it is normal I probably did not mention most of the things I described on this post during the discussion sessions.
But there is an additional reason that I think is more important to me, which is differentiating between performing skepticism and actually voicing true opinions. It is not possible for my facilitator to notice which one I am doing because they don’t know me, and performing skepticism (in order to conform to the perceived standard of “you have to think about all of this critically and by your own, and you will probably arrive to similar conclusions than others in this field”) looks the same as actually raising the confusions you have. This is why I thought I can convey this failure mode to others by comparing to inner misalignment :)
When I was a Math freshman my professor told us he always encourages people to ask questions during lectures. Often, it had happened that he’d explained a concept and nobody would ask anything. He’d check what the students understood, and it would turn out they did not grasp the concept. When asking why nobody asked anything, the students would say that they did not understand enough to ask a good question. To avoid this dynamic, he told us that “I did not understand anything” counts as a valid question on his lectures. It helped somewhat but at least I still often stayed silent instead of raising my hand and saying “I did not understand anything”.
I feel like the same dynamic can easily happen when discussing AI safety (or any difficult EA concept, really). If people are encouraged to raise questions and concerns they might only raise the “good” ones, and stay silent if they feel like they just did not understand the concepts well enough (like I did in my avoidance strategy 1).
Hi Otto! Thanks, it was nice talking to you on EAG. (I did not include any interactions/information I got from this weekend’s EAG in the post because I had written it before the conference, felt like it should not be any longer than it already was, but wanted to wait until my friends who are described as “my friends” in the post had read it before publishing it.)
I am not that convinced AGI is necessarily the most important component to x-risk from AI – I feel like there could be significant risks from powerful non-generally intelligent systems, but of course it is important to avoid all x-risk, so x-risk from AGI specifically is also worth talking about.
I don’t enjoy putting numbers to estimates but I understand why it can be a good idea so I will try. At least then I can later see if I have changed my mind and by how much. I would give quite low probability to 1), perhaps 1%? (I know this is lower than average estimates by AI researchers.) I think 2) on the other hand is very likely, maybe 99%, by the assumption that there can be enough differences between implement AGIs to make a team of AGIs surpass a team of humans by for example more efficient communication (basically what Russell says in Human Compatible on this seems credible to me). Note that even if this would be superhuman intelligence it could still be more stupid than some superintelligence scenarios. I would give a much lower probability to superintelligence like Bostrom describes it. 3) is hard to estimate without knowing much about the type of superintelligence, but I would spontanously say something high, like 80%? So because of the low probability on 1) my concatenated estimate is still significantly lower than yours.I definitely would love to read more research on this as well.
Hi Caleb! Very nice to read your reflection on what might make you think what you think. I related to many things you mentioned, such as wondering how much I think intelligence matters because of having wanted to be smart as a kid.
You understood correctly that intuitively, I think AI is less of a big deal than some people feel. This probably has a lot to do with my job, because it includes making estimates on if problems can be solved with current technology given certain constraints, and it is better to err to the side of caution. Previously, one of my tasks was also to explain people why AI is not a silver bullet and that modern ML solutions require things like training data and interfaces in order to be created and integrated to systems. Obviously, if the task is to find out all things that can future AI systems might be able to do at some point, you should take a quite different attitude than when trying to estimate what you yourself can implement right now. This is why I try to take a less conservative approach than would come naturally to me, but I think it still comes across as pretty conservative compared to many AI safety folks.
I also find GPT-3 fascinating but I think the feeling I get from it is not “wow, this thing seems actually intelligent” but rather “wow, statistics can really encompass so many different properties of language”. I love language so it makes me happy. But to me, it seems that GPT-3 is ultimately a cool showcase of the current data-centered ML approaches (“take a model that is based on a relatively non-complex idea[1], pour a huge amount of data into it, use model”). I don’t see it as a direct stepping stone to science-automating AI, because it is my intuition that “doing science well” is not that well encompassed in the available training data. (I should probably reflect more on what the concrete difference is.)
Importantly, this does not mean I believe there can be no risks (or benefits!) from large language models, and models that will be developed in the near future.
I think it is very hard to be aware of your intuitions, incorporate new valid information to your world view and communicate with others at the same time. But I agree that for everyone it is better if we create better opportunities to do that, because otherwise we will lose information.
- ^
not to say non-complexity would make the model somehow insignificant, quite the opposite, it is fascinating what attention mechanisms accomplish not only in NLP but on other domains as well
- ^
Thanks for the feedback! Deciding how to end the story was definitely the hardest part in writing this. Pulling the reader out of the fantasy was a deliberate choice, but that does not mean it was necessarily the best one – I did some A/B testing on my proof reading audience but I have to admit my sample size was not that big. Glad you liked it in general anyway :)