Thanks for the reply, and for trying to attach numbers to your thoughts!
So our main disagreement lies in (1). I think this is a common source of disagreement, so it’s important to look into it further.
Would you say that the chance to ever build AGI is similarly tiny? Or is it just the next hundred years? In other words, is this a possibility or a timeline discussion?
Hmm, with a non-zero probability in the next 100 years the likelihood for a longer time frame should be bigger given that there is nothing that makes developing AGI more difficult the more time passes, and I would imagine it is more likely to get easier than harder (unless something catastrophic happens). In other words, I don’t think it is certainly impossible to build AGI, but I am very pessimistic about anything like current ML methods leading to AGI. A lot of people in the AI safety community seem to disagree with me on that, and I have not completely understood why.
So although we seem to be relatively close in terms of compute, we don’t have the right algorithms yet for AGI, and no one knows if and when they will be found. If no one knows, I’d say a certainty of 99% that they won’t be found in hundred years, with thousands of people trying, is overconfident.
Yeah, I understand why you’d say that. However it seems to me that there are other limitations to AGI than finding the right algorithms. As a data scientist I am biased to think about available training data. Of course there is probably going to be progress on this as well in the future.
Could you explain a bit more about the kind of data you think will be needed to train an AGI, and why you think this will not be available in the next hundred years? I’m genuinely interested, actually I’d love to be convinced about the opposite… We can also DM if you prefer.
This intuition turned out harder to explain than I thought and got me thinking a lot about how to define “generality” and “intelligence” (like all talk about AGI does). But say, for example, that you want to build an automatic doctor that is able examine a patient and diagnose what illness they most likely have. This is not very general in the sense that you can imagine this system as a function of “read all kinds of input about the person, output diagnosis”, but I still think it provides an example of the difficulty of collecting data.
There are some data that can be collected quite easily by the user, because the user can for example take pictures of themselves, measure their temperature etc. And then there are some things the user might not be able to collect data about, such as “is this joint moving normally”. I think it is not so likely we will be able to gather meaningful data about things like “how does a persons joint move if they are healthy” unless doctors start wearing gloves that track the position of their hand while doing the examination and all this data is stored somewhere with the doctor’s interpretation.
To me it currently seems that we are collecting a lot of data about various things but there are still many things where there are no methods for collecting the relevant data, and the methods do not seem like they would start getting collected as a by-product of something (like in the case where you track what people by from online stores). Also, a lot of data is unorganized and missing labels and it can be hard to label after it has been collected.
I’m not sure if all of this was relevant or if I got side-tracked too much when thinking about a concrete example I can imagine.
Regarding your example, I think it’s quite specific, as you notice too. That doesn’t mean I think it’s invalid, but it does get me thinking: how would a human learn this task? A human intelligence wasn’t trained on many specific tasks in order to be able to do them all. Rather, it first acquired general intelligence (apparently, somewhere), and was later able to apply this to an almost infinite amount of specific tasks with typically only a few examples needed. I would guess that an AGI would solve problems in a similar way. So, first learn general intelligence (somehow), then learn specific tasks quickly with little data needed.
For your example, if the AGI would really need to do this task, I’d say it could find ways itself to gather the data, just like a human would who would want to learn this skill, after first acquiring some form of general intelligence. A human doctor might watch the healthily moving joint, gathering visual data, and might hear the joint moving, gathering audio data, or might put her hand on the joint, gathering sensory data. The AGI could similarly film and record the healthy joint moving, with already available cameras and microphones, or use data already available online, or, worst case, send in a drone with a camera and a sound recorder. It could even send in a robot that could gather sensory data if needed.
Of course, current AI lacks certain skills that are necessary to solve such a general problem in such a general way, such as really understanding the meaning behind a question that is asked, being able to plan a solution (including acquiring drones and robots in the process), and probably others. These issues would need to be solved first, so there is still a long way to go. But with the manpower, investment, and time (e.g. 100 years) available, I think we should assign a probability of at least tens of percents that this type of general intelligence including planning and acting effectively in the real world, will eventually be found. I’d say it is still unsure whether it will be based on a neural network (large language model or otherwise) or not.
Perhaps the difference between longtermists and shorttermists is imagination, rather than intelligence? And I’m not saying which side is right: perhaps we have too much imagination, on the other hand, perhaps you have too little imagination. We will only really know when the time comes.
I agree that the example was not that great and that definitely lack of data sources can be countered with general intelligence, like you describe. So it could definitely be possible that a a generally intelligent agent could plan around to gather needed data. My gut feeling is still that it is impossible to develop such intelligence based on one data source (for example text, however large amounts), but of course there are already technologies that combine different data sources (such as self-driving cars), so this clearly is also not the limit. I’ll have to think more about where this intuition of lack of data being a limit comes from, since it still feels relevant to me. Of course 100 years is a lot of time to gather data.
I’m not sure if imagination is the difference either. Maybe it is the belief in somebody actually implementing things that can be imagined.
I agree that the difficult part is to get to general intelligence, also regarding data. Compute, algorithms, and data availability are all needed to get to this point. It seems really hard to know beforehand what kind and how much of algorithms and data one would need. I agree that basically only one source of data, text, could well be insufficient. There was a post I read on a forum somewhere (could have been here) from someone who let GPT3 solve questions including things like ‘let all odd rows of your answer be empty’. GPT3 failed at all these kind of assignments, showing a lack of comprehension. Still, the ‘we haven’t found the asymptote’ argument from OpenAI (intelligence does increase with model size and that increase doesn’t seem to stop, implying that we’ll hit AGI eventually), is not completely unconvincing either. It bothers me that no one can completely rule out that large language models might hit AGI just by scaling them up. It doesn’t seem likely to me, but from a risk management perspective, that’s not the point. An interesting perspective I’d never heard before from intelligent people is that AGI might actually need embodiment to gather the relevant data. (They also think it would need social skills first—also an interesting thought.)
While it’s hard to know how much (and what kind of) algorithmic improvement and data is needed, it seems doable to estimate the amount of compute needed, namely what’s in a brain plus or minus a few orders of magnitude. It seems hard for me to imagine that evolution can be beaten by more than a few orders of magnitude in algorithmic efficiency (the other way round is somewhat easier to imagine, but still unlikely in a hundred year timeframe). I think people have focused on compute because it’s most forecastable, not because it would be the only part that’s important.
Still, there is a large gap between what I think are essentially thought experiments (relevant ones though!) leading to concepts such as AGI and the singularity, and actual present AI. I’m definitely interested in ideas filling that gap. I think ‘AGI safety from first principles’ by Richard Ngo is a good try, I guess you’ve read that too since it’s part of the AGI Safety Fundamentals curriculum? What did you think about it? Do you know any similar or even better papers about the topic?
It could be that belief too, yes! I think I’m a bit exceptional in the sense that I have no problem imagining human beings achieving really complex stuff, but also no problem imagining human beings failing miserably at what appear to be really easy coordination issues. My first thought when I heard about AGI, recursive self-improvement, and human extinction was ‘ah yeah that sounds like typically the kind of thing engineers/scientists would do!’ I guess some people believe engineers/scientists could never make AGI (I disagree), while others think they could, but would not be stupid enough to screw up badly enough to actually cause human extinction (I disagree).
Hi Otto, I have been wanting to reply to you for a while but I feel like my opinions keep changing so writing coherent replies is hard (but having fluid opinions in my case seems like a good thing). For example, while I still think only a precollected set of text as a data source is unsufficient for any general intelligence, maybe training a model on text and having it then interact with humans could lead it to connecting words to references (real world objects), and maybe it would not necessarily need many reference points of the language model is rich enough? This then again seems to sound a bit like the concept of imagination and I am worried I am antropomorphising in a weird way.
Anyway, I still hold the intuition that generality is not necessarily the most important in thinking about future AI scenarios – this of course is an argument towards taking AI risk more seriously, because it should be more likely someone will build advanced narrow AI or advanced AGI than just advanced AGI.
I liked “AGI safety from first principles” but I would still be reluctant to discuss it with say, my colleagues from my day job, so I think I would need something even more grounded to current tech, but I do understand why people do not keep writing that kind of papers because it does probably not directly help solving alignment.
Thanks for the reply, and for trying to attach numbers to your thoughts!
So our main disagreement lies in (1). I think this is a common source of disagreement, so it’s important to look into it further.
Would you say that the chance to ever build AGI is similarly tiny? Or is it just the next hundred years? In other words, is this a possibility or a timeline discussion?
Hmm, with a non-zero probability in the next 100 years the likelihood for a longer time frame should be bigger given that there is nothing that makes developing AGI more difficult the more time passes, and I would imagine it is more likely to get easier than harder (unless something catastrophic happens). In other words, I don’t think it is certainly impossible to build AGI, but I am very pessimistic about anything like current ML methods leading to AGI. A lot of people in the AI safety community seem to disagree with me on that, and I have not completely understood why.
So although we seem to be relatively close in terms of compute, we don’t have the right algorithms yet for AGI, and no one knows if and when they will be found. If no one knows, I’d say a certainty of 99% that they won’t be found in hundred years, with thousands of people trying, is overconfident.
Yeah, I understand why you’d say that. However it seems to me that there are other limitations to AGI than finding the right algorithms. As a data scientist I am biased to think about available training data. Of course there is probably going to be progress on this as well in the future.
Could you explain a bit more about the kind of data you think will be needed to train an AGI, and why you think this will not be available in the next hundred years? I’m genuinely interested, actually I’d love to be convinced about the opposite… We can also DM if you prefer.
This intuition turned out harder to explain than I thought and got me thinking a lot about how to define “generality” and “intelligence” (like all talk about AGI does). But say, for example, that you want to build an automatic doctor that is able examine a patient and diagnose what illness they most likely have. This is not very general in the sense that you can imagine this system as a function of “read all kinds of input about the person, output diagnosis”, but I still think it provides an example of the difficulty of collecting data.
There are some data that can be collected quite easily by the user, because the user can for example take pictures of themselves, measure their temperature etc. And then there are some things the user might not be able to collect data about, such as “is this joint moving normally”. I think it is not so likely we will be able to gather meaningful data about things like “how does a persons joint move if they are healthy” unless doctors start wearing gloves that track the position of their hand while doing the examination and all this data is stored somewhere with the doctor’s interpretation.
To me it currently seems that we are collecting a lot of data about various things but there are still many things where there are no methods for collecting the relevant data, and the methods do not seem like they would start getting collected as a by-product of something (like in the case where you track what people by from online stores). Also, a lot of data is unorganized and missing labels and it can be hard to label after it has been collected.
I’m not sure if all of this was relevant or if I got side-tracked too much when thinking about a concrete example I can imagine.
Hi AM, thanks for your reply.
Regarding your example, I think it’s quite specific, as you notice too. That doesn’t mean I think it’s invalid, but it does get me thinking: how would a human learn this task? A human intelligence wasn’t trained on many specific tasks in order to be able to do them all. Rather, it first acquired general intelligence (apparently, somewhere), and was later able to apply this to an almost infinite amount of specific tasks with typically only a few examples needed. I would guess that an AGI would solve problems in a similar way. So, first learn general intelligence (somehow), then learn specific tasks quickly with little data needed.
For your example, if the AGI would really need to do this task, I’d say it could find ways itself to gather the data, just like a human would who would want to learn this skill, after first acquiring some form of general intelligence. A human doctor might watch the healthily moving joint, gathering visual data, and might hear the joint moving, gathering audio data, or might put her hand on the joint, gathering sensory data. The AGI could similarly film and record the healthy joint moving, with already available cameras and microphones, or use data already available online, or, worst case, send in a drone with a camera and a sound recorder. It could even send in a robot that could gather sensory data if needed.
Of course, current AI lacks certain skills that are necessary to solve such a general problem in such a general way, such as really understanding the meaning behind a question that is asked, being able to plan a solution (including acquiring drones and robots in the process), and probably others. These issues would need to be solved first, so there is still a long way to go. But with the manpower, investment, and time (e.g. 100 years) available, I think we should assign a probability of at least tens of percents that this type of general intelligence including planning and acting effectively in the real world, will eventually be found. I’d say it is still unsure whether it will be based on a neural network (large language model or otherwise) or not.
Perhaps the difference between longtermists and shorttermists is imagination, rather than intelligence? And I’m not saying which side is right: perhaps we have too much imagination, on the other hand, perhaps you have too little imagination. We will only really know when the time comes.
Hi Otto!
I agree that the example was not that great and that definitely lack of data sources can be countered with general intelligence, like you describe. So it could definitely be possible that a a generally intelligent agent could plan around to gather needed data. My gut feeling is still that it is impossible to develop such intelligence based on one data source (for example text, however large amounts), but of course there are already technologies that combine different data sources (such as self-driving cars), so this clearly is also not the limit. I’ll have to think more about where this intuition of lack of data being a limit comes from, since it still feels relevant to me. Of course 100 years is a lot of time to gather data.
I’m not sure if imagination is the difference either. Maybe it is the belief in somebody actually implementing things that can be imagined.
Hey I wasn’t saying it wasn’t that great :)
I agree that the difficult part is to get to general intelligence, also regarding data. Compute, algorithms, and data availability are all needed to get to this point. It seems really hard to know beforehand what kind and how much of algorithms and data one would need. I agree that basically only one source of data, text, could well be insufficient. There was a post I read on a forum somewhere (could have been here) from someone who let GPT3 solve questions including things like ‘let all odd rows of your answer be empty’. GPT3 failed at all these kind of assignments, showing a lack of comprehension. Still, the ‘we haven’t found the asymptote’ argument from OpenAI (intelligence does increase with model size and that increase doesn’t seem to stop, implying that we’ll hit AGI eventually), is not completely unconvincing either. It bothers me that no one can completely rule out that large language models might hit AGI just by scaling them up. It doesn’t seem likely to me, but from a risk management perspective, that’s not the point. An interesting perspective I’d never heard before from intelligent people is that AGI might actually need embodiment to gather the relevant data. (They also think it would need social skills first—also an interesting thought.)
While it’s hard to know how much (and what kind of) algorithmic improvement and data is needed, it seems doable to estimate the amount of compute needed, namely what’s in a brain plus or minus a few orders of magnitude. It seems hard for me to imagine that evolution can be beaten by more than a few orders of magnitude in algorithmic efficiency (the other way round is somewhat easier to imagine, but still unlikely in a hundred year timeframe). I think people have focused on compute because it’s most forecastable, not because it would be the only part that’s important.
Still, there is a large gap between what I think are essentially thought experiments (relevant ones though!) leading to concepts such as AGI and the singularity, and actual present AI. I’m definitely interested in ideas filling that gap. I think ‘AGI safety from first principles’ by Richard Ngo is a good try, I guess you’ve read that too since it’s part of the AGI Safety Fundamentals curriculum? What did you think about it? Do you know any similar or even better papers about the topic?
It could be that belief too, yes! I think I’m a bit exceptional in the sense that I have no problem imagining human beings achieving really complex stuff, but also no problem imagining human beings failing miserably at what appear to be really easy coordination issues. My first thought when I heard about AGI, recursive self-improvement, and human extinction was ‘ah yeah that sounds like typically the kind of thing engineers/scientists would do!’ I guess some people believe engineers/scientists could never make AGI (I disagree), while others think they could, but would not be stupid enough to screw up badly enough to actually cause human extinction (I disagree).
Hi Otto, I have been wanting to reply to you for a while but I feel like my opinions keep changing so writing coherent replies is hard (but having fluid opinions in my case seems like a good thing). For example, while I still think only a precollected set of text as a data source is unsufficient for any general intelligence, maybe training a model on text and having it then interact with humans could lead it to connecting words to references (real world objects), and maybe it would not necessarily need many reference points of the language model is rich enough? This then again seems to sound a bit like the concept of imagination and I am worried I am antropomorphising in a weird way.
Anyway, I still hold the intuition that generality is not necessarily the most important in thinking about future AI scenarios – this of course is an argument towards taking AI risk more seriously, because it should be more likely someone will build advanced narrow AI or advanced AGI than just advanced AGI.
I liked “AGI safety from first principles” but I would still be reluctant to discuss it with say, my colleagues from my day job, so I think I would need something even more grounded to current tech, but I do understand why people do not keep writing that kind of papers because it does probably not directly help solving alignment.