A 6 line argument for AGI risk
(1) Sufficient intelligence has capitalities that are ultimately limited by physics and computability
(2) An AGI could be sufficiently intelligent that it’s limited by physics and computability but humans can’t be
(3) An AGI will come into existence
(4) If the AGIs goals aren’t the same as humans, human goals will only be met for instrumental reasons and the AGIs goals will be met
(5) Meeting human goals won’t be instrumentally useful in the long run for an unaligned AGI
(6) It is more morally valuable for human goals to be met than an AGIs goals
Thank you, those both look like exactly what I’m looking for
But thank you for replying, in hindsight by reply seems a bit dismissive :)
Not really because that paper is essentially just making the consequentialist claim that axiological long termism implies that the action we should take are those which help the long run future the most. The Good is still prior to the Right.
Hi Alex, the link isn’t working
I’m worried about associating Effective altruism and rationality closely in public. I think rationality is reasonably likely to make enemies. The existence of r/sneerclub is maybe the strongest evidence of this, but also the general dislike that lots of people have for silicon valley and ideas that have a very silicon valley feel to them. I’m unsure to degree people hate Dominic Cummings because he’s a rationality guy, but I think it’s some evidence to think that rationality is good at making enemies. Similarly, the whole NY times-Scott Alexander crazyness makes me think there’s the potential for lots of people to be really anti rationality.
I think empirical claims can be discriminatory. I was struggling with how to think about this for a while, but I think I’ve come to two conclusions. The first way I think that empirical claims can be discrimory is if they express discriminatory claims with no evidence, and people refusing to change their beliefs based on evidence. I think the other way that they can be discriminatory is when talking about the definitions of socially constructed concepts where we can, in some sense and in some contexts, decide what is true.
I think the relevant split is between people who have different standards and different preferences for enforcing discourse norms. The ideal type position on the SJ side is that a significant number of claims relating to certain protected characteristics are beyond the pale and should be subject to strict social sanctions. The facebook group seems to on the over side of this divide.
I think using Bayesian regret misses a number of important things.
It’s somewhat unclear if it means utility in the sense of a function that maps preference relations to real numbers, or utility in axiological sense. If it’s in the former sense then I think it misses a number of very important things. The first is that preferences are changed by the political process. The second is that people have stable preferences for terrible things like capital punishment.
If it means it in the axiological sense then I don’t think we have strong reason to believe that how people vote will be closely related and I think we have reason to believe it will be different systematically. This also makes it vulnerable to some people having terrible outcomes.
Lots of what I’m worried about with elected leaders are negative externalities. For instance, quite plausibly the main reasons Trump was bad was his opposition to climate change and rejecting democratic norms. The former harms mostly people in other countries and future generations, and the latter mostly future generations (and probably people in other countries too more than Americans although it’s not obviously true.)
It also doesn’t account for dynamic affects of parties changing their platforms. My claim is that the overton window is real and important.
I think that having strong political parties which the electoral system protects is good for stopping these things in rich democracies because I think the gatekeepers will systematically support the system that put them in power. I also think the set of polices the elite support is better in the axiological sense than those supported by the voting population. The catch here is that the US has weak political parties that are supported by electoral system.
Yeah I mean this is a pretty testable hypothesis and I’m tempted to actually test it. My guess is that the level of vote splitting that electoral system has won’t have an effect and that that whether not voting is compulsory, number of young people, level of education and level of trust will explain most of the variation in rich democracies.
Two books I recommend on structural causes and solutions to global poverty. The bottom billion by Paul Collier focuses on the question how can you get failed and failing states in very poor countries to middle income status and has a particular focus on civil war. It also looks at some solutions and thinks about the second order effects of aid. How Asia works by Joe Studwell focus on the question of how can you get poor countries with high quality, or potentially high quality governance and reasonably good political economy to become high income countries. It focuses exclusively on the Asian developmental state model and compares it with neoliberalish models in other parts of Asia that are now mostly middle income countries.
Maybe this isn’t something people on the forum do, but it is something I’ve heard some EAs suggest. People often have a problem when they become EAs that they now believe this really strange thing that potentially is quite core to their identity now and that can feel quite isolating. A suggestion that I’ve heard is that people should find new, EA friends to solve this problem. It is extremely important that this does not come off as saying that people should cut ties with friends and family who aren’t EAs. It is extremely important that this is not what you mean. It would be deeply unhealthy for us as community is this became common.
Yeah that sounds right, I don’t even know how many people are working on strategy based around India becoming a superpower, which seems completely plausible.
Is there anyone doing research from an EA perspective on the impact of Nigeria becoming a great power by the end of this century? Nigeria is projected to be the 3rd largest country in the world by 2100 and potentially appears to be having exponential growth in GDP per capita. I’m not claiming that it’s a likely outcome that Nigeria is a great power in 2100 but nor does it seem impossible. It isn’t clear to me that Nigeria has dramatically worse institutions than India but I expect India to be a great power by 2100. It seems like it’d be really valuable for someone to do some work on this given it seems really neglected.
I think China is basically in a similar situation to Prussia/Germany from 1848 to 1914. The revolutions of 1848 were unsuccessful in both Prussia and the independent South German states but they gave the aristocratic elites one hell of fright. The formal institutions of government didn’t change very much, nor did who was running the show—in the Prussia then Germany the aristocratic-military Junker class. They still put people they didn’t like in prison sometimes and still had kings with a large amount of formal power. However, they liberalised pretty spectacularly in lots of way—for instance trade unions were unbanned and the SPD (communist party at the time) grew to be the largest party in Germany, contract law was made equal between employers and workers and a market economy was allowed to flourish independently of the state and the old organisations of guilds and the vestiges of feudalism allowed to die.
To see how dramatic this change was one can look at the state of Prussian agriculture before and after 1848. Prior to 1848 agriculture was still in important ways governed by the Conservative mode of economic organisation—production, exchange and consumption were decided by what tradition dictated, was insulated from market forces by tariffs, and dominated by old aristocratic families. After 1848 Prussian agriculture was allowed to become a part of the market economy and become dominated by bourgeois men who ran their farms to make a profit and hired and fired workers as they pleased, and the market dictated the price of grain. It is hard to overstate how different this is from how agriculture was organised in, say, 1830.
I think China is doing something pretty similar now. 20 years ago individuals lives were controlled in lots of ways by their work units. Your factory unit provided you with your job, your house, your pension, your healthcare and it was controlled by the party. This is now not the case. People move freely between jobs (that’s mostly but not entirely true) , regional newspapers report on government failures and people bring lawsuits against big powerful companies and sometimes they win.
Prussia/Germany was able to achieve growth at the frontier after 1848 and I think it’s plausible China does the same. Basically, I think that both governments are acting something like monopolists would in a contestable market. From the outside it’s looks uncompetitive and like the monopolist should be extracting big rents, but actually they’re keeping prices low because they’re shit scared that someone’s going to come and take their place if they start trying to get monopoly profits.
Now, having said all that, the Chinese economy has some big structural problems that look like classic extractive institutions problems. The two biggest to me at least look like the urban-rural divide and the massive about of infrastructure spending fueled by local government spending based land value prices. The Hukou system increases the cost of individuals moving from one administrative district to another by making it extremely difficult to access public services. This has created an underclass of poorly educated, low productivity migrants in the big cities who’ve left their children back home who go to low quality schools and just have the poor life chances associated by being raised not by one’s parents. China then also has the classic authoritarian problem of being really good at producing loads of infrastructure and then producing way too much of it relative. The political economy reason behind this is in the China case is that big infrastructure projects offer opportunities for graft and make regional GDP numbers look good.