AGI will (likely) be quite different from current ML systems.
I’m afraid I disagree with this. For example, if this were true, interpretability from Chris Olah or the Anthropic team would be automatically doomed; Value Learning from CHAI would also be useless, our predictions about forecasting that we use to convince people of the importance of AI Safety equally so.
Wow, the “quite” wasn’t meant that strongly, though I agree that I should have expressed myself a bit clearer/differently. And the work of Chris Olah, etc. isn’t useless anyway, but yeah AGI won’t run on transformers and not a lot of what we found won’t be that useful, but we still get experience in how to figure out the principles, and some principles will likely transfer. And AGI forecasting is hard, but certainly not useless/impossible, but you do have high uncertainties.
Breakthroughs only happen when one understands the problem in detail, not when people float around vague ideas.
Breakthroughs happen when one understands the problem deeply. I think agree with the “not when people float around vague ideas” part, though I’m not sure what you mean with that. If you mean “academia of philosophy has a problem”, then I agree. If you mean “there is no way Einstein could derive special or general relativity mostly from thought experiments”, then I disagree, though you do indeed be skilled to use thought experiments. I don’t see any bad kind of “floating around with vague ideas” in the AI safety community, but I’m happy to hear concrete examples from you where you think academia methodology is better! (And I do btw. think that we need that Einstein-like reasoning, which is hard, but otherwise we basically have no chance of solving the problem in time.)
What academia does is to ask for well defined problems and concrete solutions. And that’s what we want if we want to progress.
I still don’t see why academia should be better at finding solutions. It can find solutions on easy problems. That’s why so many people in academia are goodharting all the time. Finding easy subproblems of which the solutions allow us to solve AI safety is (very likely) much harder than solving those subproblems.
Notice also that Shannon and many other people coming up with breakthroughs did so in academic ways.
Yes, in history there were some Einsteins in academia that could even solve hard problems, but those are very rare, and getting those brilliant not-goodharting people to work on AI safety is uncontroversially good I would say. But there might be better/easier/faster options than building the academic field of AI safety to find those people and make them work on AI safety.
Still, I’m not saying it’s a bad idea to promote AI safety in academia. I’m just saying it won’t nearly suffice to solve alignment, not by a longshot.
(I think the bottom of your comment isn’t as you intended it to be.)
Wow, the “quite” wasn’t meant that strongly, though I agree that I should have expressed myself a bit clearer/differently. And the work of Chris Olah, etc. isn’t useless anyway, but yeah AGI won’t run on transformers and not a lot of what we found won’t be that useful, but we still get experience in how to figure out the principles, and some principles will likely transfer. And AGI forecasting is hard, but certainly not useless/impossible, but you do have high uncertainties.
Breakthroughs happen when one understands the problem deeply. I think agree with the “not when people float around vague ideas” part, though I’m not sure what you mean with that. If you mean “academia of philosophy has a problem”, then I agree. If you mean “there is no way Einstein could derive special or general relativity mostly from thought experiments”, then I disagree, though you do indeed be skilled to use thought experiments. I don’t see any bad kind of “floating around with vague ideas” in the AI safety community, but I’m happy to hear concrete examples from you where you think academia methodology is better!
(And I do btw. think that we need that Einstein-like reasoning, which is hard, but otherwise we basically have no chance of solving the problem in time.)
I still don’t see why academia should be better at finding solutions. It can find solutions on easy problems. That’s why so many people in academia are goodharting all the time. Finding easy subproblems of which the solutions allow us to solve AI safety is (very likely) much harder than solving those subproblems.
Yes, in history there were some Einsteins in academia that could even solve hard problems, but those are very rare, and getting those brilliant not-goodharting people to work on AI safety is uncontroversially good I would say. But there might be better/easier/faster options than building the academic field of AI safety to find those people and make them work on AI safety.
Still, I’m not saying it’s a bad idea to promote AI safety in academia. I’m just saying it won’t nearly suffice to solve alignment, not by a longshot.
(I think the bottom of your comment isn’t as you intended it to be.)