trevor1

We are already in a per­sua­sion-trans­formed world and must take precautions

• trevor1Nov 4, 2023, 5:25 AM

  2 points

  2 ∶ 0

  on: trevor1′s Quick takes

  For more than a century, science fiction was basically the only institution paying people to think about the future. This sculpted the cultural landscape pretty significantly (try thinking of examples!)

AI Safety is Drop­ping the Ball on Clown Attacks

• trevor1Oct 7, 2023, 11:19 AM

  2 points

  1 ∶ 0

  on: Fix­ing In­sider Threats in the AI Sup­ply Chain

  Overall I think this post was well done and introduces valuable approaches, especially the focus on social engineering and the limits of psychometric data collection available to most firms, but generally a lot of different and valuable topics which I expect to be unfamiliar to most readers. Have you thought about cross posting this to Lesswrong, where users are friendlier to AI safety?

  Something that’s worth keeping in mind is that, although base rates of this seem low relative to employee incompetence, it’s also true that a sufficiently sophisticated adversary will be highly capable of framing specific employees for the attack. This is important for AI labs, which will be noticed by unusually powerful adversaries, and yet nonetheless must get everything right. They should expect their top-performing security staff to start getting bumped off one by one, the same way they would expect that to happen to leadership.

  Train employees on personal cybersecurity (securing smart home devices, managing logins, financial scams, etc.).

  Securing smart home devices isn’t possible, not for anyone in or adjacent to EA anyway. The idea that you can make smart devices themselves safe is ludicrous and dangerous; unless by “securing smart home devices” you meant mitigating personal exposure to them, which I absolutely agree would reduce risk. The threat of smart home devices creating massive psychometric datasets and researching social engineering with sample sizes in the millions is a security nightmare, that every developed country has gotten entangled in; just because everyone’s doing it doesn’t make it sensible or reasonable, just like religion or meat consumption. The current paradigm of constant smart device exposure is already wildly inadequate for the basic infosec that AI labs currently require, let alone for the transformative slow takeoff world that many anticipate over the next 1-2 decades.

• trevor1Oct 6, 2023, 11:09 PM

  0 points

  1 ∶ 2

  on: trevor1′s Quick takes

  Effective thinking is like building a house of cards. There are a lot of unique negative feedback loops that could scuttle the whole thing, and also increased risk of collapse from more height and time.

  But the winning strategy is to get in the habit of routinely/​passively stacking cards; even if collapse is frequent and frustrating, it’s well worth it, it’s the best way to eventually end up making a stack of cards higher than anyone else ever has. That’s how Peter Singer and other philosophers became the first humans on Earth to discover the core arguments of Effective Altruism decades ago. Before that, nobody anywhere had stacked the cards high enough.

• trevor1Sep 26, 2023, 8:11 AM

  16 points

  4 ∶ 1

  in reply to: Ben_West🔸’s comment on: Ben_West’s Shortform

  The Forum feels like it’s in a better place to me than when FTX declared bankruptcy: the moderation team at the time was Lizka, Lorenzo, and myself, but it is now six people, and they’ve put in a number of processes to make it easier to deal with a sudden growth in the number of heated discussions. We have also made a number of design changes, notably to the community section.

  This is a huge relief to hear. I noticed some big positive differences, but I couldn’t tell where from. Thank you.

• trevor1Sep 26, 2023, 4:10 AM

  −6 points

  0 ∶ 1

  in reply to: erickb’s comment on: Ama­zon to in­vest up to $4 billion in Anthropic

  Your concept of “redundancy” in AI labs is confusing and the implied connection to safety is tenuous.

  Sorry to nitpick, but I think this specific sentence isn’t true at all; my concept of “redundancy” wasn’t confusing and the implied connection to safety isn’t tenuous.

• trevor1Sep 26, 2023, 4:02 AM

  0 points

  1 ∶ 2

  in reply to: Davis_Kingsley’s comment on: Ama­zon to in­vest up to $4 billion in Anthropic

  That’s plausibly good for community-building, but from the infosec approach, you don’t really know what kinds of people are reading the comments, or what kind of person they will be in a year or so. In an extreme scenario, people could start getting turned. But the more likely outcome is that people hired by various bigcorps (and possibly intelligence agencies) are utilizing EAforum for open-source intelligence; this is far more prevalent than most people think.

• trevor1Sep 26, 2023, 3:53 AM

  6 points

  0 ∶ 0

  in reply to: lilly’s comment on: Ama­zon to in­vest up to $4 billion in Anthropic

  This makes sense and it changed my mind, rudeness should stay on Lesswrong where Bayes Points rule the scene. Also, at the time I’m leaving this comment, the distribution of support on this page has shifted such that the ratio of opposition to the deal to uncertainty about the deal is less terrible; it was pretty bad when I wrote this comment.

  I still think that people are too harsh on Anthropic, and that has consequences. I was definitely concerned as well when I first found out about this; Amazon plays hardball, and is probably much more capable of doing cultural investigations and appearing harmless than Anthropic thinks. Nickliang’s comment might have been far more carefully worded than I thought. But at the same time, if Dustin opposes the villainization of Anthropic and Yudkowsky is silent on the matter, that seems like mobbing Anthropic is the wrong move with serious real-life consequences.

• trevor1Sep 26, 2023, 1:49 AM

  1 point

  0 ∶ 0

  in reply to: titotal’s comment on: “X dis­tracts from Y” as a thinly-dis­guised fight over group sta­tus /​ politics

  I think this comment describes a really important crux here- the mentality of AI safety being like demonic possession is a mentality that’s one of the core tools in the toolbox of anyone trying to criticize AI safety in bad faith.

  This is a routinely recurring dynamic, in public statements by celebrities like Mitchell, in the news, etc, and ignoring it because it’s stupid, is to ignore a key gear in any information warfare battlespace (a defensive one in this case). They might be afraid to mention it outright, but it can still be between the lines a majority of the time, a dog whistle, etc.

• trevor1Sep 25, 2023, 8:23 PM

  −20 points

  4 ∶ 7

  in reply to: NickLaing’s comment on: Ama­zon to in­vest up to $4 billion in Anthropic

  I was going to reply to this comment, but after seeing the comments here, I’ve decided to abstain from sharing information on this specific post. The confidence that people here have about this being bad news, rather than uncertain news, indicates very dangerous levels of incompetence, narrow-mindedness, and even unfamiliarity with race dynamics (e.g. how one of the main risks of accelerating AI, even early on, comes from the creation of executives and AI engineers who neurotically pursue AI acceleration).

  NickLaing is just one person and if one person doesn’t have a complete picture then that’s not a big deal, that’s random error and it happens to everyone. When a dozen or more people each have an incomplete picture and confidently take aggressive stances against Anthropic, then that’s a very serious issue. I now have a better sense of why Yudkowsky became apprehensive about writing about AI publicly, or why Dustin Moscovitz throws his weight behind Anthropic and insists that they’re the good guys. If the people here would like to attempt to develop a perspective on race dynamics, they can start with the Yudkowsky Christiano debate which is balanced, or Yudkowsky’s List of Lethalities and Christiano’s response. Johnswentworth just put up a great post relevant to the topic. Or just read Christiano’s response or Holden’s Cold Takes series, the important thing here isn’t balance, it’s about having any perspective at all on race dynamics before you decide whether to tear into Anthropic’s reputation.

• trevor1Sep 23, 2023, 2:36 AM

  1 point

  0 ∶ 0

  in reply to: Will Aldred’s comment on: Will Al­dred’s Shortform

  I think it’s valuable to note that the type of people who do well in government are a specific type of person with a specific type of approach to reality, and they are spending many hours of the day in a completely different mindset (buried in a less-nerdy, more toxic environment) than most people in and around EA (buried in a more-nerdy, less toxic environment). A culture of futility is very pervasive in government and possibly important in order to do well at all. People in government roles are pretty far out-of-distribution relative to EA as a whole and possibly also potentially have a biased view of government as well due to higher access to jobs in parts of government with higher turnover and lower patriotism, even if those specific parts aren’t very representative of the parts that matter. Of course, it’s also possible that they got where they are mostly because they’re just that good.

  Such a gap would unambiguously be worth analyzing, but probably not in a public forum post imo (especially because it’s probably already being done by people privately).

• trevor1Sep 22, 2023, 4:17 PM

  2 points

  2 ∶ 2

  on: AI is cen­tral­iz­ing by de­fault; let’s not make it worse

  research to better control AIs being far easier than research to control humans

  Sorry to nitpick, but this is a bad and dangerous assumption. Just because you don’t have access to information about research to control humans, doesn’t mean that research doesn’t exist. Not all lines of research have an intrinsic tendency to get open-sourced.

  It’s not that research about controlling/​influencing humans is super secret, it’s just that civilization is currently in a state where most people don’t know where to start. A great example of a research starter is chapter 1 and 4 of Joseph Nye’s Soft Power (2005) which is critical for geopolitical forecasting, especially for AI geopolitics in the US-China, because it demonstrates that, unlike conventional and nuclear war, information wars can actually be fought and won by today’s global powers without mutually assured destruction, incentivizing more interest from intelligence agencies relative to interest in conventional military force. Most people in the 2020s know that politicians lie in order to steer public opinion, many fewer people knew that in the 1950s even though it was an open secret back then too. It follows that there are probably other open secrets in the 2020s that most people don’t know about.

• trevor1Sep 21, 2023, 1:14 PM

  4 points

  0 ∶ 0

  in reply to: richard_ngo’s comment on: richard_ngo’s Shortform

  I think that the 2-scenario model described here is very important, and should be a foundation for thinking about the future of AI safety.

  However, I think that both scenarios will also be compromised to hell. The attack surface for the AI safety community will be massive in both scenarios, ludicrously massive in scenario #2, but nonetheless still nightmarishly large in scenario #1.

  Assessment of both scenarios revolves around how inevitable you think slow takeoff is- I think that some aspects of slow takeoff, such as intelligence agencies, already started around 10 years ago and at this point just involve a lot of finger crossing and hoping for the best.

• trevor1Sep 21, 2023, 12:48 PM

  1 point

  0 ∶ 0

  in reply to: Stephen Clare’s comment on: Care­less talk on US-China AI com­pe­ti­tion? (and crit­i­cism of CAIS cov­er­age)

  Agreed on the choice of an older edition.

  Mearsheimer still makes for a good base, especially for someone whose main exposure to international affairs and military/​government was from reading the news and high school civics class (e.g. constitutional checks and balances), which unfortunately is the level that many non-international focused regulation-specialists in EA are still at. I don’t think this speaks badly to their skill level and certainly not their potential, just that they start out in a really unfair circumstance, with a head filled with a bunch of bullshit that just needs to be thrown out as cleanly as possible, and Mearsheimer is a great way to do that. I don’t remember much about the benefits from reading my first IR textbook, but I remember feeling extremely confused and disoriented, whereas Mearsheimer left me with a clear model that I could tweak and criticize and add gears to.

  Mearsheimer starts you out with “yes, this is how it is, and the stuff you grew up with and still see in the news is total bullshit and you’re going to have a bad time trying to build a solid model off of that” and people can’t do well with modelling international affairs unless they’re prepared to bite the bullet and do that at some point. Mearsheimer’s model itself is of course insufficient on its own, but its empirical base and predictiveness are strong, and it sets up the student to add in their own gears (a big one being information warfare). That’s really important for being able to forecast how slow takeoff will disrupt and transform the system in historically unprecedented ways.

• trevor1Sep 21, 2023, 12:34 PM

  1 point

  0 ∶ 0

  in reply to: Ben Millwood🔸’s comment on: Care­less talk on US-China AI com­pe­ti­tion? (and crit­i­cism of CAIS cov­er­age)

  Agreed.

  I’m not really the right kind of person to be commenting on this here, I thought I might end up being the only person responding (this was the first comment) so I left a comment that seemed significantly better than nothing. The helpfulness of my comment was substantially outclassed by subsequent comments.

• trevor1Sep 20, 2023, 2:01 PM

  1 point

  0 ∶ 0

  on: Care­less talk on US-China AI com­pe­ti­tion? (and crit­i­cism of CAIS cov­er­age)

  Have you done a lot of US-China or have any kind of IR background? Lots of people recommend starting out with reading the first couple chapters of Mearsheimer’s Tragedy of Great Power Politics in order to get a sense of how things work at the macro scale, I think that the first two chapters of Thomas Schelling’s Arms and Influence are neat but it’s pretty old.

• trevor1Sep 20, 2023, 1:10 PM

  −8 points

  1 ∶ 9

  in reply to: Linch’s comment on: Linch’s Shortform

  If someone consistently generates combinations of words in an obviously adversarial/​adverse way, then there are two major explanations:

  1. They have chosen to be disruptive

     1. the average intelligence of an EAforum user allows them to easily find clever ways to do this while simultaneously skirting the rules, or even with charisma on top of that (optimize for gaining status among onlookers)

     2. refuting their argument gives them attention and puts you on a playing field that they are willing to invest way more time into thinking about

  2. They don’t realize that they are being disruptive, e.g. due to language barrier, in which case Linch needs to know because then it is worth his time to make the effort to resolve the issue

     1. They don’t realize they are being disruptive, due to a personality issue that they’re only dimly aware of and calling attention to it might help them

  In both scenarios, asking them if they’re a native language speaker appears to be a net positive choice. I think that Linch was tearing down a Chesterton-Schelling fence here, causing an unforseen consequence, such as dozens of people going absolutely bananas.

• trevor1Sep 20, 2023, 1:45 AM

  9 points

  0 ∶ 5

  on: Protest against Meta’s ir­re­versible pro­lifer­a­tion (Sept 29, San Fran­cisco)

  The decision to attend this particular protest is actually a difficult one. Normally, most EA-minded people consistently do not vote or attend protests, since whether or not the protest succeeds depends on the non-EA masses who don’t do EV evaluations. Your decision not to attend predicts the decision of other EA-minded people not to attend, but does not predict the decision of non-EA people who almost entirely determine whether the protest/​vote succeeds or fails.

  However, with this specific protest, EA-minded people are the ones who almost entirely determine whether the protest succeeds or fails, because this is a protests by elites, against elites, and the general population is unwilling/​unable to do EV calculations and will not attend either away. Therefore, your decision not to attend predicts whether this protest succeeds or fails. If a third of EA-affiliated people attend, then it actually probably intimidates Facebook quite a bit, whereas if it fails and only 12 people attend, then it might even embolden Facebook.

  I’m someone who would normally not go to protests, because, in my own words, “that is obviously something that the world already has plenty of people doing”, and many people affiliated with EA have an extremely similar knee-jerk response to public protests. But this situation is different.

• trevor1Sep 17, 2023, 2:33 PM

  3 points

  0 ∶ 0

  in reply to: MathiasKB🔸’s comment on: Mathi­asKirkBonde’s Shortform

  Do a large proportion of people come back from EAGs infected with a variant of COVID, relative to other large gatherings?