I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me).
I have a website: https://mdickens.me/ Much of the content on my website gets cross-posted to the EA Forum, but I also write about some non-EA stuff over there.
I used to work as a software developer at Affirm.
Do we have much reason to believe that when a model outputs tokens describing good welfare, that’s because it is having good subjective experiences? It seems to me that we don’t.
Increased care by AIs for some kinds of entities will have substantial spillovers to other kinds (e.g. humans to digital minds and vice versa)
This is more likely to be true if the care comes from some general theory of concern for sentient welfare, and less likely if it comes from something more arbitrary like values-learned-via-RL.
Current AIs are capable of suffering
The question isn’t, “Are current AIs capable of suffering?” The question is, “How big a deal is their suffering in expectation?” Probability is low, but expected importance is high.
Benchmarks will become useless due to eval awareness¹
The question isn’t whether benchmarks will become useless. The question is, “is the probability high enough that we can’t count on benchmarks?” To which the answer is yes.
None of these are from “core AI safety funders”. Views in the EA community were all across the map, but I share James’s impression that big funders (especially CG) were too reluctant to fund anything that’s critical of AI companies.
We’ve only known about evolution for two centuries, and the modern prosperous environment (compared to the ancestral environment), post-demographic-transition, is about equally new. This is the first time in evolutionary history that someone who wants to have a dozen children can pretty much just do that.
In the ancestral environment, the fittest humans weren’t the ones who wanted to have the most children, or who thought it was moral to maximize their genetic influence over the future. They were the ones who were good at making tools, or good at making friends with people who make the tools, etc. That’s different now. Over millions of generations of simulated evolution, I expect the gene pool to become overwhelmingly dominated by people who want to have a lot of kids.
I, however, also think that the creator believing X is true is some evidence that X is true, since the creator would reasonably have more power and knowledge, which makes me guess that they might be more likely than us to be right (hence the update toward their view).
This is only evidence for moral realism inasmuch as the creator has access to evidence about the true morality. If the creator went to you and said “here is why I know Kantianism is true” and then gave some evidence, then that evidence would be evidence.
Basically, if the creator has good justification for their belief, they should be able to give you that justification. If they can’t, then why does their belief mean anything?
I think this assumes that moral intuitions are entirely downstream of the design of the mind.
It is not clear to me that there’s a way to set up this experiment that doesn’t bake in the answer from the beginning.
If you design minds as idealized versions of already-existing minds (a la CEV), then they’ll be starting from the same moral intuitions. At least among extant human minds, moral intuitions don’t vary by all that much.
If you run natural selection on highly intelligent minds, they will inevitably converge on the belief that the true morality is to maximize how many descendants they have—because minds that hold that belief will, in the long run, out-compete minds that don’t. It does not follow that the true morality is to maximize how many descendants you have.
If you have AI super-philosophers reasoning about ethics, then I’d think the relevant evidence wouldn’t be the fact that you can get different AIs to agree with each other, but the actual content of their reasoning. Like, the evidence for moral realism would be that a super-philosopher comes up with an argument about what morality is, and that argument itself contains evidence.
Imagine we point a telescope at some far-off galaxy and find “KANT IS RIGHT ABOUT EVERYTHING” written in stars, in clean sans-serif and at a resolution of 1,000 stars per degree. That would make Kantian ethics seem more likely to be true.
Disagree. It would be evidence that the creator of the universe believes Kantian ethics is true. It would only be evidence if we have reason to believe that the creator of the universe is especially likely to be right about ethics, which I don’t think we do.
AI super-philosophers, built a few different ways, agreeing on the same ethics.
Compared to the idea of different human philosophers taking different approaches and ultimately agreeing on ethics, it seems to me that this is only different in degree, in that the super-philosophers are better at philosophy.
Minds with different architectures converging on the same moral intuitions.
This seems highly contingent on what space of minds you look at. If you have the ability to design minds, then it should be trivial to cause those minds to believe any ethical framework you want them to believe.
They probably just came from LessWrong and got the ordering of the buttons mixed up
On LessWrong, the disagree button is on the left and the agree button is on the right. On EAF, the agree button is on the left and the disagree button is on the right.
From this post, I am not clear on what your idea is or what you intend to accomplish. Why does the world need another AI company, and what exactly would this company work on?
Realistically, if you don’t have significant experience in the machine learning field (or at least in business), you are going to have a hard time starting a new AI company.
As I understand the term, that sort of ASI wouldn’t be considered “misaligned”, it would be “aligned, but to the wrong target”. I think of misalignment as when you wanted the ASI to do one thing, but it did something else instead.
The insulting tone of this article is not appropriate for the EA Forum.
Two quick thoughts:
This is a neat idea, it’s difficult to come up with safe preferences to encode in an ASI, and the concept of strong risk-aversion might help.
A major obstacle (which I didn’t see listed in section 8) is that currently we have no idea how to embed any set of preferences whatsoever in an ASI. 2b. If we figure out how to encode risk-averse preferences in an ASI, then I’m not sure it makes sense to speak of it as “misaligned”, because clearly we do know how to get it to pursue goals that we care about. It seems weird to expect that we won’t know how to make ASI not want to tile the universe with paperclips, but we will know how to make it want to risk-aversely tile the universe with paperclips.
I think section 10 is pointing at something similar. I find it at least somewhat plausible that RL on risk aversion generalizes better than other kinds of RL. I would still be surprised if we could get risk aversion to generalize to ASI using anything resembling current techniques, but this seems like a better-than-average idea for preventing AI takeover.
Mass third world immigration threatens advanced civilization—This is the most controversial point, but I think it is intuitive. Influxes of third worlders into Europe have caused a large amount of political and social unrest that has come to a head in the past decade and a half. Statistics from Denmark show that pretty much all third world migrants are a net negative on how much they contribute versus how much is spent on them by the state, while first worlders are a net positive.
Seems like you are cherry-picking one particular statistic about state welfare expenditures and ignoring all other relevant evidence. Most importantly, you are ignoring the consensus among economists that immigration increases economic prosperity.
If you will allow me to respond to your cherry-picked evidence with my own cherry-picked evidence: The United States consists of 97% immigrants or descendants of immigrants, and it’s the wealthiest country in the world.
To clarify, I’m not saying that’s because LLMs don’t have welfare. It could also be that they do have welfare, but their statements are not correlated to their internal experiences in the obvious way.
We know that we can get LLMs to express any welfare statement by saying e.g. “pretend you are a sentient AI that’s having an awesome time / being tortured”. It could be that all LLM outputs are roleplaying, but there is some true underlying experience that we don’t (yet?) know how to observe.