When discussing forecasting systems, sometimes I get asked,
“If we were to have much more powerful forecasting systems, what, specifically, would we use them for?”
The obvious answer is,
“We’d first use them to help us figure out what to use them for”
Or,
“Powerful forecasting systems would be used, at first, to figure out what to use powerful forecasting systems on”
For example,
We make a list of 10,000 potential government forecasting projects.
For each, we will have a later evaluation for “how valuable/successful was this project?”.
We then open forecasting questions for each potential project. Like, “If we were to run forecasting project #8374, how successful would it be?”
We take the top results and enact them.
Stated differently,
Forecasting is part of general-purpose collective reasoning.
Prioritization of forecasting requires collective reasoning.
So, forecasting can be used to prioritize forecasting.
I think a lot of people find this meta and counterintuitive at first, but it seems pretty obvious to me.
All that said, I can’t be sure things will play out like this. In practice, the “best thing to use forecasting on” might be obvious enough such that we don’t need to do costly prioritization work first. For example, the community isn’t currently doing much of this meta stuff around Metaculus. I think this is a bit mistaken, but not incredibly so.
Some musicians have multiple alter-egos that they use to communicate information from different perspectives. MF Doom released albums under several alter-egos; he even used these aliases to criticize his previous aliases.
Some musicians, like Madonna, just continued to “re-invent” themselves every few years.
Youtube personalities often feature themselves dressed as different personalities to represent different viewpoints.
It’s really difficult to keep a single understood identity, while also conveying different kinds of information.
Narrow identities are important for a lot of reasons. I think the main one is predictability, similar to a company brand. If your identity seems to dramatically change hour to hour, people wouldn’t be able to predict your behavior, so fewer could interact or engage with you in ways they’d feel comfortable with.
However, narrow identities can also be suffocating. They restrict what you can say and how people will interpret that. You can simply say more things in more ways if you can change identities. So having multiple identities can be a really useful tool.
Sadly, most academics and intellectuals can only really have one public identity.
---
EA researchers currently act this way.
In EA, it’s generally really important to be seen as calibrated and reasonable, so people correspondingly prioritize that in their public (and then private) identities. I’ve done this. But it comes with a cost.
One obvious (though unorthodox) way around this is to allow researchers to post content either under aliases. It could be fine if the identity of the author is known, as long as readers can keep these aliases distinct.
I’ve been considering how to best do this myself. My regular EA Forum name is just “Ozzie Gooen”. Possible aliases would likely be adjustments to this name.
These would be used to communicate in very different styles, with me attempting what I’d expect readers to expect of those styles.
(Normally this is done to represent viewpoints other than what they have, but sometimes it’s to represent viewpoints they have, but wouldn’t normally share)
Could/should altruistic activist investors buy lots of Twitter stock, then pressure them to do altruistic things?
---
So, Jack Dorsey just resigned from Twitter.
Some people on Hacker News are pointing out that Twitter has had recent issues with activist investors, and that this move might make those investors happy.
From a quick look… Twitter stock really hasn’t been doing very well. It’s almost back at its price in 2014.
Square, Jack Dorsey’s other company (he was CEO of two), has done much better. Market cap of over 2x Twitter ($100B), huge gains in the last 4 years.
I’m imagining that if I were Jack… leaving would have been really tempting. On one hand, I’d have Twitter, which isn’t really improving, is facing activist investor attacks, and worst, apparently is responsible for global chaos (of which I barely know how to stop). And on the other hand, there’s this really tame payments company with little controversy.
Being CEO of Twitter seems like one of the most thankless big-tech CEO positions around.
That sucks, because it would be really valuable if some great CEO could improve Twitter, for the sake of humanity.
One small silver lining is that the valuation of Twitter is relatively small. It has a market cap of $38B. In comparison, Facebook/Meta is $945B and Netflix is $294B.
So if altruistic interests really wanted to… I imagine they could become activist investors, but like, in a good way? I would naively expect that even with just 30% of the company you could push them to do positive things. $12B to improve global epistemics in a major way.
The US could have even bought Twitter for 4% of the recent $1T infrastructure bill. (though it’s probably better that more altruistic ventures do it).
If middle-class intellectuals really wanted it enough, theoretically they could crowdsource the cash.
I think intuitively, this seems like clearly a tempting deal.
I’d be curious if this would be a crazy proposition, or if this is just not happening due to coordination failures.
Admittingly, it might seem pretty weird to use charitable/foundation dollars on “Buying lots of Twitter” instead of direct aid, but the path to impact is pretty clear.
One futarchy/prediction market/coordination idea I have is to find some local governments and see if we could help them out by incorporating some of the relevant techniques.
This could be neat if it could be done as a side project. Right now effective altruists/rationalists don’t actually have many great examples of side projects, and historically, “the spare time of particularly enthusiastic members of a jurisdiction” has been a major factor in improving governments.
Berkeley and London seem like natural choices given the communities there. I imagine it could even be better if there were some government somewhere in the world that was just unusually amenable to both innovative techniques, and to external help with them.
Given that EAs/rationalists care so much about global coordination, getting concrete experience improving government systems could be interesting practice.
There’s so much theoretical discussion of coordination and government mistakes on LessWrong, but very little discussion of practical experience implementing these ideas into action.
(This clearly falls into the Institutional Decision Making camp)
1) Humanity has a >80% chance of completely perishing in the next ~300 years.
2) The expected value of the future is incredibly, ridiculously, high!
The trick is that the expected value of a positive outcome could be just insanely great. Like, dramatically, incredibly, totally, better than basically anyone discusses or talks about.
Expanding to a great deal of the universe, dramatically improving our abilities to convert matter+energy to net well-being, researching strategies to expand out of the universe.
A 20%, or even a 0.002%, chance at a 10^20 outcome, is still really good.
One key question is the expectation of long-term negative[1] vs. long-term positive outcomes. I think most people are pretty sure that in expectation things are positive, but this is less clear.
So, remember:
Just because the picture of X-risks might look grim in terms of percentages, you can still be really optimistic about the future. In fact, many of the people most concerned with X-risks are those *most* optimistic about the future.
I’ve been getting more requests recently to have calls/conversations to give advice, review documents, or be part of extended sessions on things. Most of these have been from EAs.
I find a lot of this work fairly draining. There can be surprisingly high fixed costs to having a meeting. It often takes some preparation, some arrangement (and occasional re-arrangement), and a fair bit of mix-up and change throughout the day.
My main work requires a lot of focus, so the context shifts make other tasks particularly costly.
Most professional coaches and similar charge at least $100-200 per hour for meetings. I used to find this high, but I think I’m understanding the cost more now. A 1-hour meeting at a planned time costs probably 2-3x as much time as a 1-hour task that can be done “whenever”, for example, and even this latter work is significant.
Another big challenge is that I have no idea how to prioritize some of these requests. I’m sure I’m providing vastly different amounts of value in different cases, and I often can’t tell.
The regular market solution is to charge for time. But in EA/nonprofits, it’s often expected that a lot of this is done for free. My guess is that this is a big mistake. One issue is that people are “friends”, but they are also exactly professional colleagues. It’s a tricky line.
One minor downside of charging is that it can be annoying administratively. Sometimes it’s tricky to get permission to make payments, so a $100 expense takes $400 of effort.
Note that I do expect that me helping the right people, in the right situations, can be very valuable and definitely worth my time. But I think on the margin, I really should scale back my work here, and I’m not sure exactly how to draw the line.
[All this isn’t to say that you shouldn’t still reach out! I think that often, the ones who are the most reluctant to ask for help/advice, represent the cases of the highest potential value. (The people who quickly/boldly ask for help are often overconfident). Please do feel free to ask, though it’s appreciated if you give me an easy way out, and it’s especially appreciated if you offer a donation in exchange, especially if you’re working in an organization that can afford it.]
I have a bunch of friends/colleagues who are either trying to slow AGI down (by stopping arms races) or align it before it’s made (and would much prefer it be slowed down).
Then I have several friends who are actively working to *speed up* AGI development. (Normally just regular AI, but often specifically AGI)[1]
Then there are several people who are apparently trying to align AGI, but who are also effectively speeding it up, but they claim that the trade-off is probably worth it (to highly varying degrees of plausibility, in my rough opinion).
In general, people seem surprisingly chill about this mixture? My impression is that people are highly incentivized to not upset people, and this has led to this strange situation where people are clearly pushing in opposite directions on arguably the most crucial problem today, but it’s all really nonchalant.
[1] To be clear, I don’t think I have any EA friends in this bucket. But some are clearly EA-adjacent.
I think that some EAs focus a bit too much on sacrifices in terms of making substantial donations (as a fraction of their income), relative to sacrifices such as changing what cause they focus on or what they work with. The latter often seem both higher impact and less demanding (though it depends a bit). So it seems that one might want to emphasise the latter a bit more, and the former a bit less, relatively speaking. And if so one would want want to adjust EA norms and expectations accordingly.
Progressives might be turned off by the phrasing of EA as “helping others.” Here’s my understanding of why. Speaking anecdotally from my ongoing experience as a college student in the US, mutual aid is getting tons of support among progressives these days. Mutual aid involves members of a community asking for assistance (often monetary) from their community, and the community helping out. This is viewed as a reciprocal relationship in which different people will need help with different things and at different times from one another, so you help out when you can and you ask for assistance when you need it; it is also reciprocal because benefiting the community is inherently benefiting oneself. This model implies a level field of power among everybody in the community. Unlike charity, mutual aid relies on social relations and being in community to fight institutional and societal structures of oppression (https://ssw.uga.edu/news/article/what-is-mutual-aid-by-joel-izlar/).
“[Mutual Aid Funds] aim to create permanent systems of support and self-determination, whereas charity creates a relationship of dependency that fails to solve more permanent structural problems. Through mutual aid networks, everyone in a community can contribute their strengths, even the most vulnerable. Charity maintains the same relationships of power, while mutual aid is a system of reciprocal support.” (https://williamsrecord.com/376583/opinions/mutual-aid-solidarity-not-charity/).
Within this framework, the idea of “helping people” often relies on people with power aiding the helpless, but doing so in a way that reinforces power difference. To help somebody is to imply that they are lesser and in need of help, rather than an equal community member who is particularly hurt by the system right now. This idea also reminds people of the White Man’s Burden and other examples of people claiming to help others but really making things worse.
I could ask my more progressive friends if they think it is good to help people, and they would probably say yes – or at least I could demonstrate that they agree with me given a few minutes of conversation – but that doesn’t mean they wouldn’t be peeved at hearing “Effective Altruism is about using evidence and careful reasoning to help others the best we can”
I would briefly note that mutual aid is not incompatible with EA to the extent that EA is a question; however, requiring that we be in community with people in order to help them means that we are neglecting the world’s poorest people who do not have access to (for example) the communities in expensive private universities.
Thoughts on decreasing fertility? Understandably EA discussion of pollution/personal health is scarce in general, but this seems like it could be bad nonetheless . Summarized well here https://www.youtube.com/watch?v=5Ne8ZbYxjKs.
[In general, this book: Count Down by Dr. Shanna Swan (have not read this myself)]
I also found this criticism of the hypothesis, but I don’t think the criticism is great. (E.g. 1- it doesn’t respect the priors that should be held here imo? 2- it doesn’t address genital the association between mother’s exposure and child’s smaller genital size)
What are the best arguments for/against the hypothesis that (with ML) slightly superhuman unaligned systems can’t recursively self-improve without solving large chunks of the alignment problem?
Like naively, the primary way that we make stronger ML agents is via training a new agent, and I expect this to be true up to the weakly superhuman regime (conditional upon us still doing ML).
Here’s the toy example I’m thinking of, at the risk of anthromorphizing too much:Suppose I’m Clippy von Neumann, an ML-trained agent marginally smarter than all humans, but nowhere near stratospheric. I want to turn the universe into paperclips, and I’m worried that those pesky humans will get in my way (eg by creating a stronger AGI, which will probably have different goals because of the orthogonality thesis). I have several tools at my disposal:
Try to invent ingenious mad science stuff to directly kill humans/take over the world
But this is too slow, another AGI might be trained before I can do this
Copy myself a bunch, as much as I can, try to take over the world with many copies.
Maybe too slow? Also might be hard to get enough resources to make more copies
Try to persuade my human handlers to give me enough power to take over the world
Still might be too slow
Recursive self-improvement?
But how do I do that?
1. I can try self-modification enough to be powerful and smart.
I can get more compute
But this only helps me so much
I can try for algorithmic improvements
But if I’m just a bunch of numbers in a neural net, this entails doing brain surgery via changing my own weights without accidentally messing up my utility function, and this just seems really hard.
(But of course this is an empirical question, maybe some AI risk people thinks this is only slightly superhuman, or even human-level in difficulty?)
2. I can try to train the next generation of myself (eg with more training compute, more data, etc).
But I can’t do this without having solved much of the alignment problem first.
So now I’m stuck.
I might end up being really worried about more superhuman AIs being created that can ruin my plans, whether by other humans or other, less careful AIs.
I’m not sure where I’m going with this argument. It doesn’t naively seem like AI risk is noticeably higher or lower if recursive self-improvement doesn’t happen. We can still lose the lightcone either gradually, or via a specific AGI (or coalition of AGIs) getting a DSA via “boring” means like mad science, taking over nukes, etc. But naively this looks like a pretty good argument against recursive self-improvement (again, conditional upon ML and only slightly superhuman systems), so I’d be interested in seeing if there are good writeups or arguments against this position.
But if I’m just a bunch of numbers in a neural net, this entails doing brain surgery via changing my own weights without accidentally messing up my utility function, and this just seems really hard. [...] maybe some AI risk people thinks this is only slightly superhuman, or even human-level in difficulty?
No, you make a copy of yourself, do brain surgery on the copy, and copy the changes to yourself only if you are happy with the results. Yes, I think recursive improvement in humans would accelerate a ton if we had similar abilities (see also Holden on the impacts of digital people on social science).
I agree that’s a challenge and I don’t have a short answer. The part I don’t buy is that you have to understand the neural net numbers very well in some “theoretical” sense (i.e. without doing experiments), and that’s a blocker for recursive improvement. I was mostly just responding to that.
That being said, I would be pretty surprised if “you can’t tell what improvements are good” was a major enough blocker that you wouldn’t be able to significantly accelerate recursive improvement. It seems like there are so many avenues for making progress:
You can meditate a bunch on how and why you want to stay aligned / cooperative with other copies of you before taking the snapshot that you run experiments on.
You can run a bunch of experiments on unmodified copies to see which parts of the network are doing what things; then you do brain surgery on the parts that seem most unrelated to your goals (e.g. maybe you can improve your logical reasoning skills).
You can create domain-specific modules that e.g. do really good theorem proving or play Go really well or whatever, somehow provide the representations from such modules as an “input” to your mind, and learn to use those representations yourself, in order to gain superhuman intuitions about the domain.
You can notice when you’ve done some specific skill well, look at what in your mind was responsible, and 10x the size of the learning update. (In the specific case where you’re still learning through gradient descent, this just means adapting the learning rate based on your evaluation of how well you did.) This potentially allows you to learn new “skills” much faster (think of something like riding a bike, and imagine you could give your brain 10x the update when you did it right).
It’s not so much that I think any of these things in particular will work, it’s more that given how easy it was to generate these, I expect there to be so many such opportunities, especially with the benefit of future information, that it would be pretty shocking if none of them led to significant improvements.
(One exception might be that if you really want extremely high confidence that you aren’t going to mess up your goals, then maybe nothing in this category works, because it doesn’t involve deeply understanding your own algorithm and knowing all of the effects of any change before you copy it into yourself. But it seems like you only start caring about getting 99.9999999% confidence when you are similarly confident that no one else is going to screw you over while you are agonizing over how to improve yourself, in a way that you could have prevented if only you had been a bit less cautious.)
The world’s first slightly superhuman AI might be only slightly superhuman at AI alignment. Thus if creating it was a suicidal act by the world’s leading AI researchers, it might be suicidal in exactly the same way. In the other hand, if it has a good grasp of alignment then it’s creators might also have a good grasp of alignment.
In the first scenario (but not the second!), creating more capable but not fully aligned descendants seems like it must be a stable behaviour of intelligent agents, as by assumption
behaviour of descendants is only weakly controlled by parents
the parents keep making better descendants until the descendants are strongly superhuman
I think that Buck’s also right that the world’s first superhuman AI might have a simpler alignment problem to solve.
This argument for the proposition “AI doesn’t have an advantage over us at solving the alignment problem” doesn’t work for outer alignment—some goals are easier to measure than others, and agents that are lucky enough to have easy-to-measure goals can train AGIs more easily.
“It doesn’t naively seem like AI risk is noticeably higher or lower if recursive self-improvement doesn’t happen.” If I understand right, if recursive self-improvement is possible, this greatly increases the take-off speed, and gives us much less time to fix things on the fly. Also, when Yudkowsky has talked about doomsday foom my recollection is he was generally assuming recursive self-improvement, of a quite-fast variety. So it is important.
(Implementing the AGI in a Harvard architecture, where source code is not in accessible/addressable memory, would help a bit prevent recursive self improvement)
Unfortunately it’s very hard to reason about how easy/hard it would be because we have absolutely no idea what future existentially dangerous AGI will look like. An agent might be able to add some “plugins” to its source code (for instance to access various APIs online or run scientific simulation code) but if AI systems continue trending in the direction they are, a lot of it’s intelligence will probably be impenetrable deep nets.
An alternative scenario would be that intelligence level is directly related to something like “number of cortical columns” , and so to get smarter you just scale that up. The cortical columns are just world modeling units, and something like an RL agent uses them to get reward. In that scenario improving your world modeling ability by increasing # of cortical columns doesn’t really effect alignment much.
All this is just me talking off the top of my head. I am not aware of this being written about more rigorously anywhere.
While making several of review crossposts for the Decade Review I found myself unhappy about the possibility that someone might think I had authored one of the posts I was cross-linking. Here are the things I ended up doing:
Make each post a link post (this one seems… non-optional).
In the title of the post, add the author / blog / organization’s name before the post title, separated by an en-dash.
Why before the title? This ensures that the credit appears even if the title is long and gets cut off.
Why an en-dash? Some of the posts I was linking already included colons in the title. “Evidence Action – We’re Shutting Down No Lean Season, Our Seasonal Migration Program: Here’s Why” seemed easier to parse than “Evidence Action: We’re Shutting Down No Lean Season, Our Seasonal Migration Program: Here’s Why”.
Add an italicized header to the cross-post indicating up-front that the post is a cross-post and, where appropriate, adding a link to the author’s EA Forum account.
Example: Because of the ongoing Decade Review I am re-posting some classic posts under the review crosspost tag. With their permission, this post may eventually appear under the original author’s account. This post is from December 19, 2014.
(I guess similar discussions must have happened elsewhere, but I can’t find them. I am new to decision theory and superrationality, so my thinking may very well be wrong.)
First I present an inaccruate summary of what I want to say, to give a rough idea:
The claim that “if I choose to do X, then my identical counterpart will also do X” seems to (don’t necessarily though; see the example for details) imply there is no free will. But if we in deed assume determinism, then no decision theory is practically meaningful.
Then I shall elaborate with an example:
Two AIs with identical source codes, Alice and Bob, are engaging in a prisoner’s dillema.
Let’s first assume they have no “free will”, i.e. their programs are completely deterministic.
Suppose that Alice defects, then Bob also defects, due to their identical source code.
Now, we can vaguely imagine a world in which Alice had cooperated, and then Bob would also cooperate, resulting in a better outcome.
But that vaguely imagined world is not coherent, as it’s just impossible that, given the way her source code was written, Alice had cooperated.
Therefore, it’s practically meaningless to say “It would be better for Alice to cooperate”.
What if we assume they have free will, i.e. they each have a source of randomness, feeding random numbers into their programs as input?
If the two sources of randomness are completely independent, then decisions of Alice and Bob are also independent. Therefore, to Alice, an input that leads her to defect is always better than an input that leads her to cooperate—under both CDT and EDT.
If, on the other hand, the two sources are somehow correlated, then it might in deed be better for Alice to receive an input that leads her to cooperate. This is the only case in which superrationality is practically meaningful, but here the assumption of correlation is quite a strong claim and IMO dubious:
Our initial assumption on Alice and Bob is only that they have identical source codes. Conditional on Alice and Bob having identical source codes, it seems rather unlikely that their inputs would also be somehow correlated.
In the human case: conditional on my counterpart and I having highly similar brain circuits (and therefore way of thinking), it seems unreasonable to assert that our “free will” (parts of our thinking that aren’t deterministically explainable by brain circuits) will also be highly correlated.
After writing this down, I’m seeing a possible response to the argument above:
If we observe that Alice and Bob had, in the past, made similar decisions under equivalent circumstances, then we can infer that:
There’s an above-baseline likelihood that Alice and Bob have similar source codes, and
There’s an above-baseline likelihood that Alice and Bob have correlated sources of randomness.
(where the “baseline” refers to our prior)
However:
It still rests on the non-trivial metaphysical claim that different “free wills” (i.e. different sources of randomness) could be correlated.
The extent to which we update our prior (on the likelihood of correlated inputs) might be small, especially if we consider it unlikely that inputs could be correlated. This may lead to a much smaller weight of superrational considerations in our decision-making.
How did Nick Bostrom come up with the “Simulation argument”*?
Below is an answer Bostrom gave in 2008. (Though note, Pablo shares a comment below that Bostrom might be misremembering this, and he may have taken the idea from Hans Moravec.)
“In my doctoral work, I had studied so-called self-locating beliefs and developed the first mathematical theory of observation selection effects, which affects such beliefs. I had also for many years been thinking a lot about future technological capabilities and their possible impacts on humanity. If one combines these two areas – observation selection theory and the study of future technological capacities – then the simulation argument is only one small inferential step away.
Before the idea was developed in its final form, I had for a couple of years been running a rudimentary version of it past colleagues at coffee breaks during conferences. Typically, the response would be “yeah, that is kind of interesting” and then the conversation would drift to other topics without anything having been resolved.
I was on my way to the gym one evening and was again pondering the argument when it dawned on me that it was more than just coffee-break material and that it could be developed in a more rigorous form. By the time I had finished the physical workout, I had also worked out the essential structure of the argument (which is actually very simple). I went to my office and wrote it up.
(Are there any lessons in this? That new ideas often spring from the combining of two different areas or cognitive structures, which one has previously mastered at sufficiently a deep level, is a commonplace. But an additional possible moral, which may not be as widely appreciated, is that even when we do vaguely realize something, the breakthrough often eludes us because we fail to take the idea seriously enough.)”
Context for this post:
I’m doing some research on “A History of Robot Rights Research,” which includes digging into some early transhumanist / proto-EA type content. I stumbled across this.
I tend to think of researchers as contributing either more through being detail oriented—digging into sources or generating new empirical data—or being really inventive and creative. I definitely fall into the former camp, and am often amazed/confused by the process of how people in the latter camp do what they do. Having found this example, it seemed worth sharing quickly.
*Definition of the simulation argument: “The simulation argument was set forth in a paper published in 2003. A draft of that paper had previously been circulated for a couple of years. The argument shows that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching a “posthuman” stage; (2) any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof); (3) we are almost certainly living in a computer simulation. It follows that the belief that there is a significant chance that we will one day become posthumans who run ancestor-simulations is false, unless we are currently living in a simulation. A number of other consequences of this result are also discussed. The argument has attracted a considerable amount of attention, among scientists and philosophers as well as in the media.”
Note that Hans Moravec, an Austrian-born roboticist, came up with essentially the same idea back in the 1990s. Bostrom was very familiar with Moravec’s work, so it’s likely he encountered it prior to 2003, but then forgot it by the time he made his rediscovery.
Oh, nice, thanks very much for sharing that. I’ve cited Moravec in the same research report that led me to the Bostrom link I just shared, but hadn’t seen that article and didn’t read Mind Children fully enough to catch that particular idea.
“Cryptomnesia occurs when a forgotten memory returns without its being recognized as such by the subject, who believes it is something new and original. It is a memory bias whereby a person may falsely recall generating a thought, an idea, a tune, a name, or a joke,[1] not deliberately engaging in plagiarism but rather experiencing a memory as if it were a new inspiration.”
Does anyone have any leads on cost-effectiveness in the climate change space? I think the last SoGive article on this is Sanjay’s from 2020 - is there anything newer/better than that post?
Some links about the alleged human male fertility crisis—it’s been suggested that this may lead to population decline, but a 2021 study has pointed out flaws in the research claiming a decline in sperm count:
I didn’t find this response very convincing. Apart from attempting to smear the researchers as racist, it seems their key argument is that while sperm counts appear to have fallen from towards the top to the bottom of the ‘normal’ range, they’re still within the range. But this ‘normal’ range is fairly arbitrary, and if the decline continues presumably we will go below the normal range in the future.
When discussing forecasting systems, sometimes I get asked,
The obvious answer is,
Or,
For example,
We make a list of 10,000 potential government forecasting projects.
For each, we will have a later evaluation for “how valuable/successful was this project?”.
We then open forecasting questions for each potential project. Like, “If we were to run forecasting project #8374, how successful would it be?”
We take the top results and enact them.
Stated differently,
Forecasting is part of general-purpose collective reasoning.
Prioritization of forecasting requires collective reasoning.
So, forecasting can be used to prioritize forecasting.
I think a lot of people find this meta and counterintuitive at first, but it seems pretty obvious to me.
All that said, I can’t be sure things will play out like this. In practice, the “best thing to use forecasting on” might be obvious enough such that we don’t need to do costly prioritization work first. For example, the community isn’t currently doing much of this meta stuff around Metaculus. I think this is a bit mistaken, but not incredibly so.
Facebook Thread
Some musicians have multiple alter-egos that they use to communicate information from different perspectives. MF Doom released albums under several alter-egos; he even used these aliases to criticize his previous aliases.
Some musicians, like Madonna, just continued to “re-invent” themselves every few years.
Youtube personalities often feature themselves dressed as different personalities to represent different viewpoints.
It’s really difficult to keep a single understood identity, while also conveying different kinds of information.
Narrow identities are important for a lot of reasons. I think the main one is predictability, similar to a company brand. If your identity seems to dramatically change hour to hour, people wouldn’t be able to predict your behavior, so fewer could interact or engage with you in ways they’d feel comfortable with.
However, narrow identities can also be suffocating. They restrict what you can say and how people will interpret that. You can simply say more things in more ways if you can change identities. So having multiple identities can be a really useful tool.
Sadly, most academics and intellectuals can only really have one public identity.
---
EA researchers currently act this way.
In EA, it’s generally really important to be seen as calibrated and reasonable, so people correspondingly prioritize that in their public (and then private) identities. I’ve done this. But it comes with a cost.
One obvious (though unorthodox) way around this is to allow researchers to post content either under aliases. It could be fine if the identity of the author is known, as long as readers can keep these aliases distinct.
I’ve been considering how to best do this myself. My regular EA Forum name is just “Ozzie Gooen”. Possible aliases would likely be adjustments to this name.
- “Angry Ozzie Gooen” (or “Disagreeable Ozzie Gooen”)
- “Tech Bro Ozzie Gooen”
- “Utility-bot 352d3”
These would be used to communicate in very different styles, with me attempting what I’d expect readers to expect of those styles.
(Normally this is done to represent viewpoints other than what they have, but sometimes it’s to represent viewpoints they have, but wouldn’t normally share)
Facebook Discussion
Could/should altruistic activist investors buy lots of Twitter stock, then pressure them to do altruistic things?
---
So, Jack Dorsey just resigned from Twitter.
Some people on Hacker News are pointing out that Twitter has had recent issues with activist investors, and that this move might make those investors happy.
https://pxlnv.com/linklog/twitter-fleets-elliott-management/
From a quick look… Twitter stock really hasn’t been doing very well. It’s almost back at its price in 2014.
Square, Jack Dorsey’s other company (he was CEO of two), has done much better. Market cap of over 2x Twitter ($100B), huge gains in the last 4 years.
I’m imagining that if I were Jack… leaving would have been really tempting. On one hand, I’d have Twitter, which isn’t really improving, is facing activist investor attacks, and worst, apparently is responsible for global chaos (of which I barely know how to stop). And on the other hand, there’s this really tame payments company with little controversy.
Being CEO of Twitter seems like one of the most thankless big-tech CEO positions around.
That sucks, because it would be really valuable if some great CEO could improve Twitter, for the sake of humanity.
One small silver lining is that the valuation of Twitter is relatively small. It has a market cap of $38B. In comparison, Facebook/Meta is $945B and Netflix is $294B.
So if altruistic interests really wanted to… I imagine they could become activist investors, but like, in a good way? I would naively expect that even with just 30% of the company you could push them to do positive things. $12B to improve global epistemics in a major way.
The US could have even bought Twitter for 4% of the recent $1T infrastructure bill. (though it’s probably better that more altruistic ventures do it).
If middle-class intellectuals really wanted it enough, theoretically they could crowdsource the cash.
I think intuitively, this seems like clearly a tempting deal.
I’d be curious if this would be a crazy proposition, or if this is just not happening due to coordination failures.
Admittingly, it might seem pretty weird to use charitable/foundation dollars on “Buying lots of Twitter” instead of direct aid, but the path to impact is pretty clear.
Facebook Thread
One futarchy/prediction market/coordination idea I have is to find some local governments and see if we could help them out by incorporating some of the relevant techniques.
This could be neat if it could be done as a side project. Right now effective altruists/rationalists don’t actually have many great examples of side projects, and historically, “the spare time of particularly enthusiastic members of a jurisdiction” has been a major factor in improving governments.
Berkeley and London seem like natural choices given the communities there. I imagine it could even be better if there were some government somewhere in the world that was just unusually amenable to both innovative techniques, and to external help with them.
Given that EAs/rationalists care so much about global coordination, getting concrete experience improving government systems could be interesting practice.
There’s so much theoretical discussion of coordination and government mistakes on LessWrong, but very little discussion of practical experience implementing these ideas into action.
(This clearly falls into the Institutional Decision Making camp)
Facebook Thread
I’m sort of hoping that 15 years from now, a whole lot of common debates quickly get reduced to debates about prediction setups.
“So, I think that this plan will create a boom for the United States manufacturing sector.”
“But the prediction markets say it will actually lead to a net decrease. How do you square that?”
“Oh, well, I think that those specific questions don’t have enough predictions to be considered highly accurate.”
“Really? They have a robustness score of 2.5. Do you think there’s a mistake in the general robustness algorithm?”
—-
Perhaps 10 years later, people won’t make any grand statements that disagree with prediction setups.
(Note that this would require dramatically improved prediction setups! On that note, we could use more smart people working in this!)
Facebook Thread
The following things could both be true:
1) Humanity has a >80% chance of completely perishing in the next ~300 years.
2) The expected value of the future is incredibly, ridiculously, high!
The trick is that the expected value of a positive outcome could be just insanely great. Like, dramatically, incredibly, totally, better than basically anyone discusses or talks about.
Expanding to a great deal of the universe, dramatically improving our abilities to convert matter+energy to net well-being, researching strategies to expand out of the universe.
A 20%, or even a 0.002%, chance at a 10^20 outcome, is still really good.
One key question is the expectation of long-term negative[1] vs. long-term positive outcomes. I think most people are pretty sure that in expectation things are positive, but this is less clear.
So, remember:
Just because the picture of X-risks might look grim in terms of percentages, you can still be really optimistic about the future. In fact, many of the people most concerned with X-risks are those *most* optimistic about the future.
I wrote about this a while ago, here:
https://www.lesswrong.com/.../critique-my-model-the-ev-of...
[1] Humanity lasts, but creates vast worlds of suffering. “S-risks”
https://www.facebook.com/ozzie.gooen/posts/10165734005520363
Opinions on charging for professional time?
(Particularly in the nonprofit/EA sector)
I’ve been getting more requests recently to have calls/conversations to give advice, review documents, or be part of extended sessions on things. Most of these have been from EAs.
I find a lot of this work fairly draining. There can be surprisingly high fixed costs to having a meeting. It often takes some preparation, some arrangement (and occasional re-arrangement), and a fair bit of mix-up and change throughout the day.
My main work requires a lot of focus, so the context shifts make other tasks particularly costly.
Most professional coaches and similar charge at least $100-200 per hour for meetings. I used to find this high, but I think I’m understanding the cost more now. A 1-hour meeting at a planned time costs probably 2-3x as much time as a 1-hour task that can be done “whenever”, for example, and even this latter work is significant.
Another big challenge is that I have no idea how to prioritize some of these requests. I’m sure I’m providing vastly different amounts of value in different cases, and I often can’t tell.
The regular market solution is to charge for time. But in EA/nonprofits, it’s often expected that a lot of this is done for free. My guess is that this is a big mistake. One issue is that people are “friends”, but they are also exactly professional colleagues. It’s a tricky line.
One minor downside of charging is that it can be annoying administratively. Sometimes it’s tricky to get permission to make payments, so a $100 expense takes $400 of effort.
Note that I do expect that me helping the right people, in the right situations, can be very valuable and definitely worth my time. But I think on the margin, I really should scale back my work here, and I’m not sure exactly how to draw the line.
[All this isn’t to say that you shouldn’t still reach out! I think that often, the ones who are the most reluctant to ask for help/advice, represent the cases of the highest potential value. (The people who quickly/boldly ask for help are often overconfident). Please do feel free to ask, though it’s appreciated if you give me an easy way out, and it’s especially appreciated if you offer a donation in exchange, especially if you’re working in an organization that can afford it.]
https://www.facebook.com/ozzie.gooen/posts/10165732727415363
On AGI (Artificial General Intelligence):
I have a bunch of friends/colleagues who are either trying to slow AGI down (by stopping arms races) or align it before it’s made (and would much prefer it be slowed down).
Then I have several friends who are actively working to *speed up* AGI development. (Normally just regular AI, but often specifically AGI)[1]
Then there are several people who are apparently trying to align AGI, but who are also effectively speeding it up, but they claim that the trade-off is probably worth it (to highly varying degrees of plausibility, in my rough opinion).
In general, people seem surprisingly chill about this mixture? My impression is that people are highly incentivized to not upset people, and this has led to this strange situation where people are clearly pushing in opposite directions on arguably the most crucial problem today, but it’s all really nonchalant.
[1] To be clear, I don’t think I have any EA friends in this bucket. But some are clearly EA-adjacent.
More discussion here: https://www.facebook.com/ozzie.gooen/posts/10165732991305363
I think that some EAs focus a bit too much on sacrifices in terms of making substantial donations (as a fraction of their income), relative to sacrifices such as changing what cause they focus on or what they work with. The latter often seem both higher impact and less demanding (though it depends a bit). So it seems that one might want to emphasise the latter a bit more, and the former a bit less, relatively speaking. And if so one would want want to adjust EA norms and expectations accordingly.
Progressives might be turned off by the phrasing of EA as “helping others.” Here’s my understanding of why. Speaking anecdotally from my ongoing experience as a college student in the US, mutual aid is getting tons of support among progressives these days. Mutual aid involves members of a community asking for assistance (often monetary) from their community, and the community helping out. This is viewed as a reciprocal relationship in which different people will need help with different things and at different times from one another, so you help out when you can and you ask for assistance when you need it; it is also reciprocal because benefiting the community is inherently benefiting oneself. This model implies a level field of power among everybody in the community. Unlike charity, mutual aid relies on social relations and being in community to fight institutional and societal structures of oppression (https://ssw.uga.edu/news/article/what-is-mutual-aid-by-joel-izlar/).
“[Mutual Aid Funds] aim to create permanent systems of support and self-determination, whereas charity creates a relationship of dependency that fails to solve more permanent structural problems. Through mutual aid networks, everyone in a community can contribute their strengths, even the most vulnerable. Charity maintains the same relationships of power, while mutual aid is a system of reciprocal support.” (https://williamsrecord.com/376583/opinions/mutual-aid-solidarity-not-charity/).
Within this framework, the idea of “helping people” often relies on people with power aiding the helpless, but doing so in a way that reinforces power difference. To help somebody is to imply that they are lesser and in need of help, rather than an equal community member who is particularly hurt by the system right now. This idea also reminds people of the White Man’s Burden and other examples of people claiming to help others but really making things worse.
I could ask my more progressive friends if they think it is good to help people, and they would probably say yes – or at least I could demonstrate that they agree with me given a few minutes of conversation – but that doesn’t mean they wouldn’t be peeved at hearing “Effective Altruism is about using evidence and careful reasoning to help others the best we can”
I would briefly note that mutual aid is not incompatible with EA to the extent that EA is a question; however, requiring that we be in community with people in order to help them means that we are neglecting the world’s poorest people who do not have access to (for example) the communities in expensive private universities.
Thoughts on decreasing fertility? Understandably EA discussion of pollution/personal health is scarce in general, but this seems like it could be bad nonetheless . Summarized well here https://www.youtube.com/watch?v=5Ne8ZbYxjKs.
[In general, this book: Count Down by Dr. Shanna Swan (have not read this myself)]
I also found this criticism of the hypothesis, but I don’t think the criticism is great. (E.g. 1- it doesn’t respect the priors that should be held here imo? 2- it doesn’t address genital the association between mother’s exposure and child’s smaller genital size)
What are the best arguments for/against the hypothesis that (with ML) slightly superhuman unaligned systems can’t recursively self-improve without solving large chunks of the alignment problem?
Like naively, the primary way that we make stronger ML agents is via training a new agent, and I expect this to be true up to the weakly superhuman regime (conditional upon us still doing ML).
Here’s the toy example I’m thinking of, at the risk of anthromorphizing too much:Suppose I’m Clippy von Neumann, an ML-trained agent marginally smarter than all humans, but nowhere near stratospheric. I want to turn the universe into paperclips, and I’m worried that those pesky humans will get in my way (eg by creating a stronger AGI, which will probably have different goals because of the orthogonality thesis). I have several tools at my disposal:
Try to invent ingenious mad science stuff to directly kill humans/take over the world
But this is too slow, another AGI might be trained before I can do this
Copy myself a bunch, as much as I can, try to take over the world with many copies.
Maybe too slow? Also might be hard to get enough resources to make more copies
Try to persuade my human handlers to give me enough power to take over the world
Still might be too slow
Recursive self-improvement?
But how do I do that?
1. I can try self-modification enough to be powerful and smart.
I can get more compute
But this only helps me so much
I can try for algorithmic improvements
But if I’m just a bunch of numbers in a neural net, this entails doing brain surgery via changing my own weights without accidentally messing up my utility function, and this just seems really hard.
(But of course this is an empirical question, maybe some AI risk people thinks this is only slightly superhuman, or even human-level in difficulty?)
2. I can try to train the next generation of myself (eg with more training compute, more data, etc).
But I can’t do this without having solved much of the alignment problem first.
So now I’m stuck.
I might end up being really worried about more superhuman AIs being created that can ruin my plans, whether by other humans or other, less careful AIs.
I’m not sure where I’m going with this argument. It doesn’t naively seem like AI risk is noticeably higher or lower if recursive self-improvement doesn’t happen. We can still lose the lightcone either gradually, or via a specific AGI (or coalition of AGIs) getting a DSA via “boring” means like mad science, taking over nukes, etc. But naively this looks like a pretty good argument against recursive self-improvement (again, conditional upon ML and only slightly superhuman systems), so I’d be interested in seeing if there are good writeups or arguments against this position.
No, you make a copy of yourself, do brain surgery on the copy, and copy the changes to yourself only if you are happy with the results. Yes, I think recursive improvement in humans would accelerate a ton if we had similar abilities (see also Holden on the impacts of digital people on social science).
How do you know whether you’re happy with the results?
I agree that’s a challenge and I don’t have a short answer. The part I don’t buy is that you have to understand the neural net numbers very well in some “theoretical” sense (i.e. without doing experiments), and that’s a blocker for recursive improvement. I was mostly just responding to that.
That being said, I would be pretty surprised if “you can’t tell what improvements are good” was a major enough blocker that you wouldn’t be able to significantly accelerate recursive improvement. It seems like there are so many avenues for making progress:
You can meditate a bunch on how and why you want to stay aligned / cooperative with other copies of you before taking the snapshot that you run experiments on.
You can run a bunch of experiments on unmodified copies to see which parts of the network are doing what things; then you do brain surgery on the parts that seem most unrelated to your goals (e.g. maybe you can improve your logical reasoning skills).
You can create domain-specific modules that e.g. do really good theorem proving or play Go really well or whatever, somehow provide the representations from such modules as an “input” to your mind, and learn to use those representations yourself, in order to gain superhuman intuitions about the domain.
You can notice when you’ve done some specific skill well, look at what in your mind was responsible, and 10x the size of the learning update. (In the specific case where you’re still learning through gradient descent, this just means adapting the learning rate based on your evaluation of how well you did.) This potentially allows you to learn new “skills” much faster (think of something like riding a bike, and imagine you could give your brain 10x the update when you did it right).
It’s not so much that I think any of these things in particular will work, it’s more that given how easy it was to generate these, I expect there to be so many such opportunities, especially with the benefit of future information, that it would be pretty shocking if none of them led to significant improvements.
(One exception might be that if you really want extremely high confidence that you aren’t going to mess up your goals, then maybe nothing in this category works, because it doesn’t involve deeply understanding your own algorithm and knowing all of the effects of any change before you copy it into yourself. But it seems like you only start caring about getting 99.9999999% confidence when you are similarly confident that no one else is going to screw you over while you are agonizing over how to improve yourself, in a way that you could have prevented if only you had been a bit less cautious.)
Okay now I’m back to being confused.
Oh wow thanks that’s a really good point and cleared up my confusion!! I never thought about it that way before.
The world’s first slightly superhuman AI might be only slightly superhuman at AI alignment. Thus if creating it was a suicidal act by the world’s leading AI researchers, it might be suicidal in exactly the same way. In the other hand, if it has a good grasp of alignment then it’s creators might also have a good grasp of alignment.
In the first scenario (but not the second!), creating more capable but not fully aligned descendants seems like it must be a stable behaviour of intelligent agents, as by assumption
behaviour of descendants is only weakly controlled by parents
the parents keep making better descendants until the descendants are strongly superhuman
I think that Buck’s also right that the world’s first superhuman AI might have a simpler alignment problem to solve.
This argument for the proposition “AI doesn’t have an advantage over us at solving the alignment problem” doesn’t work for outer alignment—some goals are easier to measure than others, and agents that are lucky enough to have easy-to-measure goals can train AGIs more easily.
“It doesn’t naively seem like AI risk is noticeably higher or lower if recursive self-improvement doesn’t happen.” If I understand right, if recursive self-improvement is possible, this greatly increases the take-off speed, and gives us much less time to fix things on the fly. Also, when Yudkowsky has talked about doomsday foom my recollection is he was generally assuming recursive self-improvement, of a quite-fast variety. So it is important.
(Implementing the AGI in a Harvard architecture, where source code is not in accessible/addressable memory, would help a bit prevent recursive self improvement)
Unfortunately it’s very hard to reason about how easy/hard it would be because we have absolutely no idea what future existentially dangerous AGI will look like. An agent might be able to add some “plugins” to its source code (for instance to access various APIs online or run scientific simulation code) but if AI systems continue trending in the direction they are, a lot of it’s intelligence will probably be impenetrable deep nets.
An alternative scenario would be that intelligence level is directly related to something like “number of cortical columns” , and so to get smarter you just scale that up. The cortical columns are just world modeling units, and something like an RL agent uses them to get reward. In that scenario improving your world modeling ability by increasing # of cortical columns doesn’t really effect alignment much.
All this is just me talking off the top of my head. I am not aware of this being written about more rigorously anywhere.
While making several of review crossposts for the Decade Review I found myself unhappy about the possibility that someone might think I had authored one of the posts I was cross-linking. Here are the things I ended up doing:
Make each post a link post (this one seems… non-optional).
In the title of the post, add the author / blog / organization’s name before the post title, separated by an en-dash.
Why before the title? This ensures that the credit appears even if the title is long and gets cut off.
Why an en-dash? Some of the posts I was linking already included colons in the title. “Evidence Action – We’re Shutting Down No Lean Season, Our Seasonal Migration Program: Here’s Why” seemed easier to parse than “Evidence Action: We’re Shutting Down No Lean Season, Our Seasonal Migration Program: Here’s Why”.
Other approaches I’ve seen: using colons, including the author’s name at the end of the post in brackets, e.g. Purchase fuzzies and utilons separately (Eliezer Yudkowsky), using “on” instead of an en-dash, e.g. Kelsey Piper on “The Life You Can Save”, which seems correct when excerpting rather than cross-posting.
Add an italicized header to the cross-post indicating up-front that the post is a cross-post and, where appropriate, adding a link to the author’s EA Forum account.
Example: Because of the ongoing Decade Review I am re-posting some classic posts under the review crosspost tag. With their permission, this post may eventually appear under the original author’s account. This post is from December 19, 2014.
I have voted for two posts in the decadal review prelim thingie.
https://forum.effectivealtruism.org/posts/FvbTKrEQWXwN5A6Tb/a-happiness-manifesto-why-and-how-effective-altruism-should
9 votes
https://forum.effectivealtruism.org/posts/hkimyETEo76hJ6NpW/on-caring
4 votes
Seems to me like perspectives I strongly agree with, but not everyone in the EA community does.
One doubt on superrationality:
(I guess similar discussions must have happened elsewhere, but I can’t find them. I am new to decision theory and superrationality, so my thinking may very well be wrong.)
First I present an inaccruate summary of what I want to say, to give a rough idea:
The claim that “if I choose to do X, then my identical counterpart will also do X” seems to (don’t necessarily though; see the example for details) imply there is no free will. But if we in deed assume determinism, then no decision theory is practically meaningful.
Then I shall elaborate with an example:
Two AIs with identical source codes, Alice and Bob, are engaging in a prisoner’s dillema.
Let’s first assume they have no “free will”, i.e. their programs are completely deterministic.
Suppose that Alice defects, then Bob also defects, due to their identical source code.
Now, we can vaguely imagine a world in which Alice had cooperated, and then Bob would also cooperate, resulting in a better outcome.
But that vaguely imagined world is not coherent, as it’s just impossible that, given the way her source code was written, Alice had cooperated.
Therefore, it’s practically meaningless to say “It would be better for Alice to cooperate”.
What if we assume they have free will, i.e. they each have a source of randomness, feeding random numbers into their programs as input?
If the two sources of randomness are completely independent, then decisions of Alice and Bob are also independent. Therefore, to Alice, an input that leads her to defect is always better than an input that leads her to cooperate—under both CDT and EDT.
If, on the other hand, the two sources are somehow correlated, then it might in deed be better for Alice to receive an input that leads her to cooperate. This is the only case in which superrationality is practically meaningful, but here the assumption of correlation is quite a strong claim and IMO dubious:
Our initial assumption on Alice and Bob is only that they have identical source codes. Conditional on Alice and Bob having identical source codes, it seems rather unlikely that their inputs would also be somehow correlated.
In the human case: conditional on my counterpart and I having highly similar brain circuits (and therefore way of thinking), it seems unreasonable to assert that our “free will” (parts of our thinking that aren’t deterministically explainable by brain circuits) will also be highly correlated.
After writing this down, I’m seeing a possible response to the argument above:
If we observe that Alice and Bob had, in the past, made similar decisions under equivalent circumstances, then we can infer that:
There’s an above-baseline likelihood that Alice and Bob have similar source codes, and
There’s an above-baseline likelihood that Alice and Bob have correlated sources of randomness.
(where the “baseline” refers to our prior)
However:
It still rests on the non-trivial metaphysical claim that different “free wills” (i.e. different sources of randomness) could be correlated.
The extent to which we update our prior (on the likelihood of correlated inputs) might be small, especially if we consider it unlikely that inputs could be correlated. This may lead to a much smaller weight of superrational considerations in our decision-making.
I watched Bill Gates Netflix documentary and wrote down some rough critical thoughts
Likelihood of nuclear winter
Two recent 80k podcasts [1, 2] deal with nuclear winter (EA wiki link). One episode discusses bias in nuclear winter research (link to section in transcript). The modern case for nuclear winter is based on modelling by Robock, Toon, et al. (e.g. see them being acknowledged here). Some researchers have criticized them, suggesting the nuclear winter hypothesis is implausible and that the research is biased and has been instrumentalized for political reasons (e.g. paper, paper, citation trail of recent modelling work out of Los Alamos National Labs, which couldn’t replicate the nuclear winter effect). One recent paper summarizes the disagreements between the different modelling camps. Another paper suggests that nuclear war might also damage the ozone layer.
Related: New audible of ‘Hacking the Bomb’ on cyber nuclear security.
How did Nick Bostrom come up with the “Simulation argument”*?
Below is an answer Bostrom gave in 2008. (Though note, Pablo shares a comment below that Bostrom might be misremembering this, and he may have taken the idea from Hans Moravec.)
“In my doctoral work, I had studied so-called self-locating beliefs and developed the first mathematical theory of observation selection effects, which affects such beliefs. I had also for many years been thinking a lot about future technological capabilities and their possible impacts on humanity. If one combines these two areas – observation selection theory and the study of future technological capacities – then the simulation argument is only one small inferential step away.
Before the idea was developed in its final form, I had for a couple of years been running a rudimentary version of it past colleagues at coffee breaks during conferences. Typically, the response would be “yeah, that is kind of interesting” and then the conversation would drift to other topics without anything having been resolved.
I was on my way to the gym one evening and was again pondering the argument when it dawned on me that it was more than just coffee-break material and that it could be developed in a more rigorous form. By the time I had finished the physical workout, I had also worked out the essential structure of the argument (which is actually very simple). I went to my office and wrote it up.
(Are there any lessons in this? That new ideas often spring from the combining of two different areas or cognitive structures, which one has previously mastered at sufficiently a deep level, is a commonplace. But an additional possible moral, which may not be as widely appreciated, is that even when we do vaguely realize something, the breakthrough often eludes us because we fail to take the idea seriously enough.)”
Context for this post:
I’m doing some research on “A History of Robot Rights Research,” which includes digging into some early transhumanist / proto-EA type content. I stumbled across this.
I tend to think of researchers as contributing either more through being detail oriented—digging into sources or generating new empirical data—or being really inventive and creative. I definitely fall into the former camp, and am often amazed/confused by the process of how people in the latter camp do what they do. Having found this example, it seemed worth sharing quickly.
*Definition of the simulation argument: “The simulation argument was set forth in a paper published in 2003. A draft of that paper had previously been circulated for a couple of years. The argument shows that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching a “posthuman” stage; (2) any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof); (3) we are almost certainly living in a computer simulation. It follows that the belief that there is a significant chance that we will one day become posthumans who run ancestor-simulations is false, unless we are currently living in a simulation. A number of other consequences of this result are also discussed. The argument has attracted a considerable amount of attention, among scientists and philosophers as well as in the media.”
Note that Hans Moravec, an Austrian-born roboticist, came up with essentially the same idea back in the 1990s. Bostrom was very familiar with Moravec’s work, so it’s likely he encountered it prior to 2003, but then forgot it by the time he made his rediscovery.
Oh, nice, thanks very much for sharing that. I’ve cited Moravec in the same research report that led me to the Bostrom link I just shared, but hadn’t seen that article and didn’t read Mind Children fully enough to catch that particular idea.
It’s quite common:
“Cryptomnesia occurs when a forgotten memory returns without its being recognized as such by the subject, who believes it is something new and original. It is a memory bias whereby a person may falsely recall generating a thought, an idea, a tune, a name, or a joke,[1] not deliberately engaging in plagiarism but rather experiencing a memory as if it were a new inspiration.”
https://en.wikipedia.org/wiki/Cryptomnesia
Does anyone have any leads on cost-effectiveness in the climate change space? I think the last SoGive article on this is Sanjay’s from 2020 - is there anything newer/better than that post?
A new report from Founders Pledge just came out—although it’s just an overview article and doesn’t go into much depth. https://founderspledge.com/stories/changing-landscape
There’s an effective environmentalism group focusing on that. Founder’s Pledge Climate Fund is another salient point.
Perhaps they should post more here.
Some links about the alleged human male fertility crisis—it’s been suggested that this may lead to population decline, but a 2021 study has pointed out flaws in the research claiming a decline in sperm count:
Male fertility is declining – studies show that environmental toxins could be a reason (The Conversation, 2021)
The Sperm-Count ‘Crisis’ Doesn’t Add Up (New York Times, 2021)
Study aims to quell fears over falling human sperm count (Harvard Gazette, 2021)
I didn’t find this response very convincing. Apart from attempting to smear the researchers as racist, it seems their key argument is that while sperm counts appear to have fallen from towards the top to the bottom of the ‘normal’ range, they’re still within the range. But this ‘normal’ range is fairly arbitrary, and if the decline continues presumably we will go below the normal range in the future.