I agree that something like that needs to be done. I did something similar myself for EA forum posts, but apparently people didn’t like it because it didn’t get any upvotes here (I don’t know why).
My main question/feedback is: “EA Forum paper links”—what exactly is that? Are these EA posts that link to a paper? If so, then why not include all posts, why only papers?
This is not an accurate summary.
It omits the most important mechanism that I mentioned in my post. The mechanism is to reward rewarding itself. This is really the most important part of my post that has been completely omitted from the summary.
“Epistemic status: This is a speculative, normative proposal relying on assumptions about behavioral adaptation, future technology, and long-term incentives; key uncertainties include whether coordination dynamics will shift as described and whether sufficient adoption can occur.”
This is not speculative, it’s based on a theoretical justification that makes logical sense. If it doesn’t make sense to someone else, please point out why.
It’s definitely not normative, in a sense that it doesn’t tell people what to do, but it offers a solution.
Key uncertainty, in my opinion:
Whether people understand the reasoning behind the solution.
Whether people have sufficient observability of people’s actions for that to work.
My solution was mostly this: https://forum.effectivealtruism.org/posts/7EAwiAopjtAdD8hsn/most-impactful-posts-on-effective-altruism-forum-from-13th
But it didn’t work. People didn’t upvote that. My problem/frustration continues.
I actually proposed a similar idea in a few places months ago (links below). I plan to post about related topics, e.g. my future posts might help to understand how such deals could be enforced, so if you are interested, then follow me
My thoughts.
On implications of diminishing marginal utility:
I believe that artificial intelligence and other resources will have diminishing marginal utility. If that’s the case, then it should be possible to do so that both humans and future digital minds are close to their maximum level of utility/happiness by splitting resources between them. The idea of “both digital minds and humans are important” would be therefore significantly more attractive for humans and realistic to accomplish than the idea of “digital minds are all of what matters”.
On moral patience:
Morality is subjective.
The criteria for moral patience are subjective. You can say that a human is a moral patient, you can say that AI agent is a moral patient, or you can say that a stone is a moral patient. There’s no objective truth about that.
However, we can define moral patience as “if I want to maximize my happiness/utility, whose happiness/utility should I care about when choosing my actions”? That is objective.
If a human wants to maximize their happiness, who should they care about?
Firstly, we need to answer what are the reasons why a human would have an interest to care about someone/something else.
The reasons are:
Game-theoretic—especially direct and indirect reciprocity.
Importantly, I think that agents have interest to care about weaker agents (ask me if you want to know why).
I also think that it’s in the interest of a human to care about other sentient life, at least to some extent, because:
It’s a norm that is useful—if we live by that norm, then we have that guarantee that we won’t end up terribly, because as long as we are sentient, someone will care about us.
It’s a norm that has a historical precedent and it’s easier to sustain a norm than to establish a new one. But the historical precedent is that humans care about existing sentient life, not potential sentient life.
If the norm of respecting sentient life implied that humans have to commit all their resources for the benefit of future digital minds / AI agents, then that norm would stop to be useful to humans, so that norm would either become weaker to the point where human lives matter, or humans would stop creating new AI agents.
Emotional—you care about someone because you simply want them to be happy.
I think it’s likely that there are game-theoretic and/or emotional reasons why we should care about AI agents / digital beings, but not to the point that they are all that we should care about and that we should ignore other agents or beings.
On number of agents:
How to count the number of agents? If we have multiple agents on different computers that cooperate and exchange knowledge, and then we have one agent that is distributed among many computers, then what’s the difference between them? Does that count as one agent or many?
Let’s suppose that we define agents by their goals (i.e. two computers with AI on it are considered different agents, if they aim to achieve different goals). If there are multiple agents with different goals (either human or AIs), then the best thing that they can do is to create one AI agent that will be optimized for the cumulative utility/happiness all of them. So, I expect that there won’t be many separate agents with different goals in the future.
I think I’ve selected a time that is not suitable for many people (especially the US West Coast). I’m going to change it to Saturday 5 pm GMT. I hope it won’t cause a problem for anyone.
I’m currently working on a simple recommendation system for posts from Effective Altruism forum, which uses a simple algorithm on top of AI models. I have written a script that scraps posts from EA forum, and then evaluates them based on impact and few others criteria, estimating the expected value of positive impact that the post will have. I plan to share the results on this forum soon.
I think that it will have to base also on information from humans later because AI models can reflect the interest of a company (if recommendation is based on AI models, then AI companies might create AI models that are aligned with interests of the company, instead of the interest of the collective, or if the models are not aligned, then they might work in their own unaligned interest.
I think aligned recommendation systems are very important because currently I believe that some good ideas get overlooked. Better recommendation systems would direct attention to the right posts/papers.
Update: But if you work or plan to work on it as well, then please don’t resign from doing so because of the fact that I’m working on it already. I also work on other projects.
I disagree.
In my opinion, it’s likely that if government has operational ability of AGI, then it would likely lead to optimizing the AI for the benefit of a subset of the people in government, just like in case of private company.
Our forms of governments have been tried, and they work to some extent, but they haven’t been tried in a situation where the elected people in the government have potential opportunity to seize so much power in such short time-frame.
In my opinion, the best way to ensure democratic outcome would be to train AI in decentralized way (using many small computers owned by many people, instead of big computers owned by a small group of people). The people in the government have interest in making that happen because if they don’t, then they are at risk of losing the power to other people in the government (due to diminishing marginal utility of AI, losing power is a bigger cost than the benefit of being in power alone).
Some people can say that decentralized training would be inefficient, so it’s not possible to do that. Maybe it’s not possible now, but it can be done because human science is learned in decentralized way (with the brains of the scientists being computers in a decentralized network).
So, I think that government-led decentralized AI could be a good idea, but I’m not convinced to government-led centralized AI.
The other alternative is maybe open-source, but there are some problems with that as well like there would have to be a method to ensure that the open-source models don’t contain a backdoor (that they are not secretly optimized for the interest of the company/organization that made the model).
If you believe that democratic outcome with centralized AI is possible, then how could that work? Who specifically would decide what the goal of AI is? And how would you ensure that it will be set to what has been decided and not to something else?
“Problems and needs: Sometimes it’s really valuable to hear user problems or needs rather than suggestions. Is there something that frustrates you when you interact with the Forum? Is there anything you find yourself looking for or wanting when you’re using the Forum?”
My biggest problem/frustration is when I write a post that I think is very important and has a lot of value, but I don’t get any reaction (or close to 0) - no upvotes and no critique/feedback.
I have some idea how to solve that, but for now I just share a problem.
Edit: I was writing the comment when I was a little frustrated, and I think my post was not very logical. I still think that there is too much authority bias, but I think a feature like curation can potentially help with that, if implemented properly.
Personally, I’m against curating and track records. I asked Gemini to generate a report on “Is there too much authority bias in the world?” (some authority bias is rational, but the question is it too much) with a neutral prompt and the result which was based on empirical evidence was there is too much authority bias. I would like to see more posts from people who don’t get a lot of attention.
I know from experience that when I want to communicate something to people, it’s very hard to do that because people seem to have short attention span when you don’t have credibility and it’s very hard to strike a balance between conciseness and clarity in the post, when people are not willing to give a lot of attention.
I would like to be able to see all posts and I’m currently confused about the user interface. I click “Advanced filters and sorting”, then I choose “New”, “All posts” and “Show low Karma” and I can see “Frontpage posts” above all lists. The word “frontpage” suggests that those are the most upvoted posts. And almost all of them have many upvotes (which suggests that those are not all posts). I want to see all posts.
So, previously my conclusion was that it’s a trade-off mostly between equality + contribution to alignment (I believe that my solution increases the probability of alignment) + AI faster so less people will die vs the risk of biological weapons.
I think the impact of equality and other things is higher because:
I asked 8 AI models about what is the expected number of people who will die from biological weapons of which development was assisted by AI (more or less, that wasn’t the exact question) to have an idea how serious the problem is, since I don’t have a lot of knowledge about biology. I removed the outliers (the biggest and the lowest answer) and got the average. The average was small relative to how many people die in 1 year from normal reasons. So, that suggests that the other problems are larger.
The idea is that if the powerful artificial intelligence is created later than sooner, then there is more time to prepare for biological weapons. But the event of humanity being prepared at X + Y time is dependent on the event of humanity being prepared at time X. Because the probability of both of those events depends on the same things like: are humans smart and considerate enough to prepare for that. Therefore, if the problem is not solved by time X, then it’s also less likely to be solved by the time X + Y.
I also considered the impact on animals and other non-human agents. One thing that I paid attention to is the following fact. Humans have some empathy, so it’s likely that they share some resources and/or benefits of AI with humans. If there is one person in the position of power, then there is a high variance of how much benefits animals (and other agents) will get because it depends on one person. For example, if that person will turn out to be a psychopath, then animals might suffer from that outcome. However, the more people there are in the position of power, the lower the variance, because the amount of benefits that animals will get will depend on more people. For animals to get nothing, all people in the position of power would have to be psychopaths. Therefore, all else equal, lack of concentration of power among humans is the safer option which is desirable given the diminishing returns of resources (utility = log resources). I’m not saying that that is the only factor at play, but I think it’s worth noting.
If anyone has something to add, feel free.
There was a mistake in “Empirical verification” section. Part of the suggested question/prompt was missing, and the reasoning didn’t make sense because of that. I have now corrected the question/prompt.