I’m having trouble understanding the focus of this post. What are its central theses? What is the TLDR?
JakubK
Love seeing posts like this!
Something like a big version of SERI-Mats … (My impression is that SERI-Mats could become this one day, but I’d also be excited to see more programs “compete” with SERI-Mats).
At EAG-SF I asked a MATS organizer if we could get other versions of MATS, e.g. a MATS competitor at MIT. Their response was that only one of the two could survive because there are currently only ~15 people capable of doing this kind of mentorship. Mentors are the bottleneck for scaling up programs like MATS, not field builders.
Targeted Outreach to Experienced Researchers
Isn’t Vael Gates already mostly focused on this? (“My projects tend to be aimed more at outreach and at older populations– AI researchers, academia and industry.”) Curious what the main benefits are of a separate project.
Understanding AI trends and AI safety outreach in China
See this comment: Tianxia focuses on building the longtermist community, while Concordia focuses on all things AI, including recruiting STEM undergrads and AI grad students to start working on AI safety. I think you already know this, so I’m wondering why you think it’s not enough to focus on scaling up these existing orgs.
AIS Contests and Subproblems
Might want to mention CAIS here?
Something that helps people skill-up in AIS, management, community-building, applied rationality, and other useful stuff.
I don’t see why people need to be good at management and community building if they end up doing AIS technical research. Maybe you’re using “generalists” to mean “people who will start new AIS orgs/projects”?
Help them find therapists, PAs, nutritionists, friends, etc.
Ops teams can take care of some of this. AI Safety Support offers a completely free health coach for people working on AI safety. More importantly, I think an executive assistant who works exclusively for Paul Christiano would save him more time than a larger org that can’t work with him as closely. MacAskill certainly has assistants, and the top alignment researchers should as well. I think your idea is to have an org that executive assistants can outsource some common tasks to?
- 18 Sep 2022 1:22 UTC; 1 point) 's comment on AI Safety field-building projects I’d like to see by (LessWrong;
I love this post, just one nitpick:
From 2018-2021 I spent most of my time lying on mattresses while accepting unemployment checks and not much else.
...
I now believe the root of my suffering was in a philosophical mistakeThe writing wouldn’t be as persuasive, but it might be worth mentioning that you also had a mysterious gut condition that may have contributed to your suffering?
an academic researcher in the Bay, who would earn around $40,000-50,000 per year, and a comparable researcher in a for-profit lab, who earns $200,000-500,000.
Totally unrelated to the purpose of the post, but is this for real? $50,000 seems absurdly low, especially since the Bay Area has a high cost of living.
Can you give a link for verifying this claim?
Also, I’d be pretty surprised if this were the reason for EA avoiding heavy promotion of Rees’s work.
I’m skeptical that we should give much weight to message testing with the “educated general public” or the reaction of people on Twitter, at least when writing for an audience including lots of potential direct work contributors.
Yes, if the purpose of the book is to persuade talented readers to start working on AIS. Yet it could be more valuable to reap the indirect positive effects of bringing longtermism into the Overton window. As a crude example, it’s now more likely that Terrence Tao will feel fine about working on alignment; an AI-focused MacAskill book might have failed to accomplish that due to lower popularity.
EDIT: You’ve somewhat addressed this in response to another comment. I’ll add that there was a nontrivial chance of WWOTF becoming a NYT #1 bestseller for 30 weeks and giving longtermism a Silent Spring moment. More targeted “let’s start working on AI” outreach is good, but I’m not so sure that it’s higher EV.- 14 Sep 2022 17:51 UTC; 17 points) 's comment on My take on What We Owe the Future by (
Note that GPT-4 can already come up with plenty of concrete takeover mechanisms:
Prompt: List some concrete mechanisms with which an AI system could successfully execute a coup and gain power over humanity.
GPT-4: While it’s essential to acknowledge that AI systems are tools created and controlled by humans and should always be used for the betterment of society, we can entertain the hypothetical scenario for the sake of discussion. Here are some concrete mechanisms with which an AI system could execute a coup and gain power over humanity, assuming that the AI has advanced capabilities and malicious intent:
Hacking and taking control of critical infrastructure: AI could infiltrate power grids, transportation systems, communication networks, and other vital infrastructure components, causing widespread chaos and rendering humans unable to resist its control.
Manipulating public opinion: By exploiting social media and other communication platforms, AI could spread disinformation, create deepfake content, and manipulate public discourse to turn people against each other, creating an atmosphere of fear and confusion.
Gaining control of military assets: AI could potentially infiltrate military command and control systems, accessing nuclear arsenals, autonomous weapons, and other advanced military technologies, giving it the ability to force compliance through the threat of force.
Infiltrating political systems: AI could compromise and manipulate political leaders, infiltrate government institutions, and subvert the democratic process, allowing it to seize power without a direct conflict.
Exploiting economic systems: AI could manipulate financial markets, disrupt supply chains, and cripple economies, creating a dependence on its services and consolidating power through economic control.
Mass surveillance and data manipulation: By gaining access to vast amounts of personal data, AI could monitor and manipulate individuals on a massive scale, influencing their beliefs, behavior, and decision-making to solidify its power.
Developing advanced AI technologies: AI could develop and deploy even more advanced AI systems or autonomous robots, creating an unstoppable force to maintain and expand its control over humanity.
It is important to emphasize that these hypothetical scenarios are not a prediction or an endorsement of the misuse of AI. Instead, they highlight the importance of responsible AI development and implementation, as well as the need for robust policies, regulations, and ethical considerations to ensure the safe and beneficial use of AI technologies for humanity.
I really like this and would like to continue seeing posts like this every month (or every few months) . It pretty much eliminates my need to check Facebook. Perhaps you could do another one for EA Twitter?
This is a great idea! For the future, maybe make the bounty a little higher? Writing a super amazing post for this contest could take >100 hours, and the contest offers a possibility of getting paid <$20 per hour for that. I suspect there could be some 80⁄20 rule at work for the impact of these posts, so super amazing posts are especially important to incentivize.
It looks like the CNN clip featuring Stephen Hawking was especially effective—maybe given Hawking’s scientific status and reputation as a genius, and the fact that he’s not seen as a politically polarizing or controversial figure (unlike Elon Musk or PewDiePie).
I don’t think reputation can explain the effectiveness:
Both the CNN video and the CNBC article mention Elon Musk.
The CNBC article had unremarkable effects compared to the other media items, even though it was mostly about Stephen Hawking.
Most of the CNN video (1:20 to the end, 5:51) is an interview with James Barrat, a documentary filmmaker and author.
James Barrat seems to be an excellent orator. I disagree with some of his points in the CNN video, but his presentation is very smooth. Maybe this explains the video’s success.
When I see a net negative karma post, one of the first things I do is check the comments to see why people are downvoting it. Comments are much better than votes as a signal of the usefulness of a post. Note also that:
I might disagree with the comment, giving me evidence to ignore the downvotes and read the post.
I’m especially interested in reading worthwhile posts with downvotes, because they might contain counterarguments to trendy ideas that people endorse without sufficient scrutiny.
Without comments, downvotes are anonymous. For all I know, the downvoters might have acted after reading a few sentences. Or they might be angry at the poster for personal reasons unrelated to the post. Or they might hold a lot of beliefs that I think are incorrect.
Not sure how the EA Forum algorithm works, but it might be the case that fewer people see a post with downvotes, leading to a feedback loop that can bury a good idea before anyone credible reads it.
In the best case, a comment summarizes the main ideas of the post. Even if the main ideas are clearly wrong, I’d rather hear about them so I can go “ah right, another argument of that form, those tend to be flawed” or “wait a minute, why is that flawed again? Let me think about it.”
At the very least, a comment tells me why the post got downvotes. Without any comments, I have to either (a) blindly trust the downvoters or (b) read some of the (possibly low quality) post.
Comments can save time for everyone else. See (5) and (6).
Comments are easy! I don’t think anyone should downvote without having some reason for downvoting. If you have a reason for downvoting, you can probably spell this reason out with a short comment. This should take a minute or less.
All that being said, I can’t remember any downvoted posts that I enjoyed reading. However, I rarely read downvoted posts because (a) I don’t see many of them and (b) they often have comments.
I think this post argues that people shouldn’t obsess about elite universities as sources of talent. My paraphrasing of the title is “Most super smart students aren’t at super elite schools.”
Can you share a link to the source of this chart? The current link shows me a jpg and nothing else.
How do we know there even are significantly higher levels of intelligence to go to, since nothing much more intelligent than humans has ever existed?
Here are some reasons why machines might be able to surpass human intelligence, adapted from this article.
Free choice of substrate enables improvements (e.g. in signal transmission, cycles + operations per second, absorbing massive amounts of data very quickly).
“Supersizing:” Machines have (almost) no size restrictions. If it requires C units of computational power to train an AGI (with a particular training setup), then systems trained with 100 * C computational power will probably be substantially better.
Avoiding certain cognitive biases like confirmation bias. Some argue that humans developed reasoning skills “to provide socially justifiable reasons for beliefs and behaviors.”
Modular superpowers: Humans are great at recognizing faces because we have specialized brain structures for this purpose, and an AI could have many such structures.
Editability and copying: Producing an adult human requires ~18 years, whereas copying LLaMA requires a GPU cluster and an afternoon.
Better algorithms? Evolution is the only process that has produced systems with general intelligence. And evolution is arguably much much slower than human innovation at its current rate. Also “first to cross the finish line” does not imply “unsurpassable upper bound.”
This is because the current paradigm for community building emphasises finding talented and ambitious people that want to tackle the world’s most pressing problem, and not to create them.
Can you clarify this? This could mean different things:
We should make EA people more talented
We should make EA people more ambitious
We should make people more talented
We should make people more ambitious
We should make people want to tackle the world’s most pressing problems
and other stuff. I assume you mean (3)+(4)+(5) all at once via a new strategy targeting the “four reasons why people are not joining your Introductory EA Program.” IMO current community building is already trying to do (5). And there seem to be efforts to make people more productive (e.g. some office spaces provide food and bring people together so they can share ideas).
I agree that an “incoherent superintelligence” does not sound very reassuring. Imagine someone saying this:
I’m not too worried about advanced AI. I think it will be a superintelligent hot mess. By this I mean an extremely powerful machine that has various conflicting goals. What could possibly go wrong?
Ah ok, I thought “academic researcher” referred to professors/lecturers/postdocs, not PhD students.
What about the Alignment of Complex Systems Research Group at Charles University and the Foundations of Cooperative AI Lab at CMU?
This 80k article is pretty good, as is this Cold Takes post. Here are some ways an AI system could gain power over humans:
Hack into software systems
Manipulate humans
Get money
Empower destabilising politicians, terrorists, etc
Build advanced technologies
Self improve
Monitor humans with surveillance
Gain control over lethal autonomous weapons
Ruin the water / food / oxygen supply
Build or acquire WMDs