vaniver

Karma: 387

vaniver 11 Nov 2022 20:30 UTC
28 points
3 ∶ 0
in reply to: RAB’s comment on: The FTX Future Fund team has resigned
From The Snowball, dealing with Warren Buffett’s son’s stint as a director and PR person for ADM:
The second the FBI agents left, Howie called his father, flailing, saying, I don’t know what to do, I don’t have the facts, how do I know if these allegations are true? My name is on every press release. How can I be the spokesman for the company worldwide? What should I do, should I resign?
Buffett refrained from the obvious response, which was that, of his three children, only Howie could have wound up with an FBI agent in his living room after taking his first job in the corporate world. He listened to the story non-judgmentally and told Howie that it was his decision whether to stay at ADM. He gave only one piece of advice: Howie had to decide within the next twenty-four hours. If you stay in longer than that, he said, you’ll become one of them. No matter what happens, it will be too late to get out.
That clarified things. Howie now realized that waiting was not a way to get more information to help him decide, it was making the decision to stay. He had to look at his options and understand as of right now what they meant.
If he resigned and they were innocent, he would lose friends and look like a jerk.
If he stayed and they were guilty, he would be viewed as consorting with criminals.
The next day Howie went in, resigned, and told the general counsel that he would take legal action against the company if they put his name on any more press releases. Resigning from the board was a major event. For a director to resign was like sending up a smoke signal that said the company was guilty, guilty, guilty. People at ADM did not make it easy for Howie. They pushed for reprieve, they asked how he could in effect convict them without a trial. Howie held firm, however, and got out.
What links here?
- Dancer's comment on In Defense of SBF by Austin (14 Nov 2022 17:37 UTC; 11 points)

vaniver 31 Aug 2020 18:47 UTC
26 points
0 ∶ 0
in reply to: Lukas_Gloor’s comment on: Some thoughts on the EA Munich // Robin Hanson incident
I’m a bit puzzled why it has to be edgy on top of just talking with fewer filters.
Presumably every filter is associated with an edge, right? Like, the ‘trolley problem’ is a classic of philosophy, and yet it is potentially traumatic for the victims of vehicular violence or accidents. If that’s a group you don’t want to upset or offend, you install a filter to catch yourself before you do, and when seeing other people say things you would’ve filtered out, you perceive them as ‘edgy’. “Don’t they know they shouldn’t say that? Are they deliberately saying that because it’s edgy?”
[A more real example is that a friend once collected a list of classic examples and thought experiments, and edited all of the food-based ones to be vegan, instead of the original food item. Presumably the people who originally generated those thought experiments didn’t perceive them as being ‘edgy’ or ‘over the line’ in some way.]
but also some element of deliberate provocation.
I read a lot of old books; for example, it’s interesting to contrast the 1934 and 1981 editions of How to Win Friends and Influence People. Deciding to write one of the ‘old-version’ sentences in 2020 would probably be seen as a deliberate provocation, and yet it seems hugely inconsistent to see Dale Carnegie as out to deliberately provoke people.
Now, I’m not saying Hanson isn’t deliberately edgy; he very well might be. But there are a lot of ways in which you might offend someone, and it takes a lot of computation to proactively notice and prevent all of them, and it’s very easy to think your filters are “common knowledge” or “obvious” when in fact they aren’t. As a matter of bounded computation, thoughts spent on filters are thoughts not spent on other things, and so there is a real tradeoff here, where the fewer filters are required the more thoughts can be spent on other things, but this is coming through a literal increase in carelessness.
What links here?
- vaniver's comment on Some thoughts on the EA Munich // Robin Hanson incident by Aaron Gertler (2 Sep 2020 19:13 UTC; 7 points)

vaniver 1 Sep 2020 21:07 UTC
23 points
0 ∶ 0
in reply to: Kirsten’s comment on: Some thoughts on the EA Munich // Robin Hanson incident
Comparing trolley accidents to rape is pretty ridiculous for a few reasons:
I think you’re missing my point; I’m not describing the scale, but the type. For example, suppose we were discussing racial prejudice, and I made an analogy to prejudice against the left-handed; it would be highly innumerate of me to claim that prejudice against the left-handed is as damaging as racial prejudice, but it might be accurate of me to say both are examples of prejudice against inborn characteristics, are perceived as unfair by the victims, and so on.
And so if you’re not trying to compare expected trauma, and just come up with rules of politeness that guard against any expected trauma above a threshold, setting the threshold low enough that both “prejudice against left-handers” and “prejudice against other races” are out doesn’t imply that the damage done by both are similar.

That said, I don’t think I agree with the points on your list, because I used the reference class of “vehicular violence or accidents,” which is very broad. I agree there’s an important disanalogy in that ‘forced choices’ like in the trolley problem are highly atypical for vehicular accidents, most of which are caused by negligence of one sort or another, and that trolleys themselves are very rare compared to cars, trucks, and trains, and so I don’t actually expect most sufferers of MVA PTSD to be triggered or offended by the trolley problem. But if they were, it seems relevant that (in the US) motor vehicle accidents are more common than rape, and lead to more cases of PTSD than rape (at least, according to 2004 research; I couldn’t quickly find anything more recent).
I also think that utilitarian thought experiments in general radiate the “can’t be trusted to abide by norms” property; in the ‘fat man’ or ‘organ donor’ variants of the trolley problem, for example, the naive utilitarian answer is to murder, which is also a real risk that could make the conversation include an implicit threat.

vaniver 20 Dec 2023 22:59 UTC
21 points
5 ∶ 0
in reply to: TracingWoodgrains’s comment on: Nonlinear’s Evidence: Debunking False and Misleading Claims
So you would have lost 40 hours of productive time? Respectfully: so what? You have sources actively claiming you are about to publish directly false information about them and asking for time to provide evidence that information is directly false.
Also, I think it is worth Oli/Ben estimating how many productive hours were lost to the decision to not delay; it would not surprise me if much of the benefit here was illusory.

vaniver 1 Sep 2020 21:26 UTC
21 points
0 ∶ 0
in reply to: Lukas_Gloor’s comment on: Some thoughts on the EA Munich // Robin Hanson incident
you didn’t want to voice unambiguous support for the view that the comment wordings were in fact not easy to improve on given the choice of topic.
I’m afraid this sentence has too many negations for me to clearly point one way or the other, but let me try to restate it and say why I made a comment:
The mechanistic approach to avoiding offense is to keep track of the ways things you say could be interpreted negatively, and search for ways to get your point across while not allowing for any of the negative interpretations. This is a tax on saying anything, and it especially taxes statements on touchy subjects, and the tax on saying things backpropagates into a tax on thinking them.
When we consider people who fail at the task of avoiding giving offense, it seems like there are three categories to consider:
1. The Blunt, who are ignoring the question of how the comment will land, and are just trying to state their point clearly (according to them).
2. The Blithe, who would put effort into rewording their point if they knew how to avoid giving offense, but whose models of the audience are inadequate to the task.
3. The Edgy, who are optimizing for being ‘on the line’ or in the ‘plausible deniability’ region, where they can both offend some targets and have some defenders who view their statements as unobjectionable.
While I’m comfortable predicting those categories will exist, confidently asserting that someone falls into any particular category is hard, because it involves some amount of mind-reading (and I think the typical mind fallacy makes it easy to think people are being Edgy, because you assume they see your filters when deciding what to say). That said, my guess is that Hanson is Blunt instead of Edgy or Blithe.

vaniver 7 Feb 2023 18:45 UTC
20 points
4 ∶ 0
on: A personal reflection on SBF
When people make big and persistent mistakes, the usual cause (in my experience) is not something that comes labeled with giant mental “THIS IS A MISTAKE” warning signs when you reflect on it.
Instead, tracing mistakes back to their upstream causes, I think that the cause tends to look like a tiny note of discord that got repeatedly ignored—nothing that mentally feels important or action-relevant, just a nagging feeling that pops up sometimes.
To do better, then, I want to take stock of those subtler upstream causes, and think about the flinch reactions I exhibited on the five-second level and whether I should have responded to them differently.
I don’t see anything in the lessons on the question of whether or not your stance on drama has changed, which feels like the most important bit?
That is, suppose I have enough evidence to not-be-surprised-in-retrospect if one of my friends is abusing their partner, and also I have a deliberate stance of leaving other people’s home lives alone. The former means that if I thought carefully about all of my friends, I would raise that hypothesis to attention; the latter means that even if I had the hypothesis, I would probably not do anything about it. In this hypothetical, I only become a force against abuse if I decide to become a meddler (which introduces other costs and considerations).

vaniver 19 Nov 2019 17:58 UTC
19 points
0 ∶ 0
in reply to: MikeJohnson’s comment on: I’m Buck Shlegeris, I do research and outreach at MIRI, AMA
FWIW I agree with Buck’s criticisms of the Symmetry Theory of Valence (both content and meta) and also think that some other ideas QRI are interested in are interesting. Our conversation on the road trip was (I think) my introduction to Connectome Specific Harmonic Waves (CSHW), for example, and that seemed promising to think about.
I vaguely recall us managing to operationalize a disagreement, let me see if I can reconstruct it:
A ‘multiple drive’ system, like PCT’s hierarchical control system, has an easy time explaining independent desires and different flavors of discomfort. (If one both has a ‘hunger’ control system and a ‘thirst’ control system, one can easily track whether one is hungry, thirsty, both, or neither.) A ‘single drive’ system, like expected utility theories more generally, has a somewhat more difficult time explaining independent desires and different flavors of discomfort, since you only have the one ‘utilon’ number.
But this is mostly because we’re looking at different parts of the system as the ‘value’. If I have a vector of ‘control errors’, I get the nice multidimensional property. If I have a utility function that’s a function of a vector, the gradient of that function will be a vector that gives me the same nice multidimensional property.
CSHW gives us a way to turn the brain into a graph and then the graph activations into energies in different harmonics. Then we can look at an energy distribution and figure out how consonant or dissonant it is. This gives us the potentially nice property that ‘gradients are easy’, where if ‘perfect harmony’ (= all consonant energy) corresponds to the ‘0 error’ case in PCT, being hungry will look like missing some consonant energy or having some dissonant energy.
Here we get the observational predictions: for PCT, ‘hunger’ and ‘thirst’ and whatever other drives just need to be wire voltages somewhere, but for QRI’s theory as I understand it, they need to be harmonic energies with particular numerical properties (such that they are consonant or dissonant as expected to make STV work out).
Of course, it could be the case that there are localized harmonics in the connectome, such that we get basically the same vector represented in the energy distribution, and don’t have a good way to distinguish between them.
On that note, I remember we also talked about the general difficulty of distinguishing between theories in this space; for example, my current view is that Friston-style predictive coding approaches and PCT-style hierarchical control approaches end up predicting very similar brain architecture, and the difference is ‘what seems natural’ or ‘which underlying theory gets more credit.’ (Is it the case that the brain is trying to be Bayesian, or the brain is trying to be homeostatic, and embedded Bayesianism empirically performs well at that task?) I expect a similar thing could be true here, where whether symmetry is the target or the byproduct is unclear, but in such cases I normally find myself reaching for ‘byproduct’. It’s easy to see how evolution could want to build homeostatic systems, and harder to see how evolution could want to build Bayesian systems; I think a similar story goes through for symmetry and brains.
This makes me more sympathetic to something like “symmetry will turn out to be a marker for something important and good” (like, say, ‘focus’) than something like “symmetry is definitionally what feeling good is.”

vaniver 25 Jun 2022 18:32 UTC
18 points
0 ∶ 0
on: Impact markets may incentivize predictably net-negative projects
This reminds me a lot of limited liability (see also Austin’s comment, where he compares it to the for-profit startup market, which because of limited liability for corporations bounds prices below by 0).
This is a historically unusual policy (full liability came first), and seems to me to have basically the same downsides (people do risky things, profiting if they win and walking away if they lose), and basically the same upsides (according to the theory supporting LLCs, there’s too little investment and support of novel projects).
Can you say more about why you think this consideration is sufficient to be net negative? (I notice your post seems very ‘do-no-harm’ to me instead of ‘here are the positive and negative effects, and we think the negative effects are larger’, I’m also interested in Owen’s impression on whether or not impact markets lead to more or less phase 2 work.)

vaniver 8 May 2023 20:44 UTC
17 points
2 ∶ 0
on: vaniver’s Shortform
I’m thinking about the matching problem of “people with AI safety questions” and “people with AI safety answers”. Snoop Dogg hears Geoff Hinton on CNN (or wherever), asks “what the fuck?”, and then tries to find someone who can tell him what the fuck.
I think normally people trust their local expertise landscape—if they think the CDC is the authority on masks they adopt the CDC’s position, if they think their mom group on Facebook is the authority on masks they adopt the mom group’s position—but AI risk is weird because it’s mostly unclaimed territory in their local expertise landscape. (Snoop also asks “is we in a movie right now?” because movies are basically the only part of the local expertise landscape that has had any opinion on AI so far, for lots of people.) So maybe there’s an opportunity here to claim that territory (after all, we’ve thought about it a lot!).
I think we have some ‘top experts’ who are available for, like, mass-media things (podcasts, blog posts, etc.) and 1-1 conversations with people they’re excited to talk to, but are otherwise busy / not interested in fielding ten thousand interview requests. Then I think we have tens (hundreds?) of people who are expert enough to field ten thousand interview requests, given that the standard is “better opinions than whoever they would talk to by default” instead of “speaking to the whole world” or w/e. But just like connecting people who want to pay to learn calculus and people who know calculus and will teach it for money, there’s significant gains from trade from having some sort of clearinghouse / place where people can easily meet. Does this already exist? Is anyone trying to make it? (Do you want to make it and need support of some sort?)

vaniver 12 Dec 2023 18:46 UTC
15 points
9 ∶ 3
in reply to: Kirsten’s comment on: Nonlinear’s Evidence: Debunking False and Misleading Claims
While it’s generally poor form to attempt to de-anonymize stories, since it’s at issue here it seems potentially worth it. It seems like this could be Kat’s description of Kat’s experience of Ben, which she (clearly) consents to sharing.

Prediction Markets for Science

vaniver2 Jan 2023 17:55 UTC

14 points

4 comments1 min readEA link

vaniver 1 Nov 2023 0:37 UTC
12 points
1 ∶ 0
on: Thoughts on the AI Safety Summit company policy requests and responses
(I’m Matthew Gray)
Inflection is a late addition to the list, so Matt and I won’t be reviewing their AI Safety Policy here.
My sense from reading Inflection’s response now is that they say the right things about red teaming and security and so on, but I am pretty worried about their basic plan / they don’t seem to be grappling with the risks specific to their approach at all. Quoting from them in two different sections:
Inflection’s mission is to build a personal artificial intelligence (AI) for everyone. That means an AI that is a trusted partner: an advisor, companion, teacher, coach, and assistant rolled into one.
Internally, Inflection believes that personal AIs can serve as empathetic companions that help people grow intellectually and emotionally over a period of years or even decades.** Doing this well requires an understanding of the opportunities and risks that is grounded in long-standing research in the fields of psychology and sociology.** We are presently building our internal research team on these issues, and will be releasing our research on these topics as we enter 2024.
I think AIs thinking specifically about human psychology—and how to convince people to change their thoughts and behaviors—are very dual use (i.e. can be used for both positive and negative ends) and at high risk for evading oversight and going rogue. The potential for deceptive alignment seems quite high, and if Inflection is planning on doing any research on those risks or mitigation efforts specific to that, it doesn’t seem to have shown up in their response.
I don’t think this type of AI is very useful for closing the acute risk window, and so probably shouldn’t be made until much later.

vaniver 3 Jun 2022 16:51 UTC
11 points
0 ∶ 0
on: Announcing a contest: EA Criticism and Red Teaming
I’m interested in fleshing out “what you’re looking for”; do you have some examples of things written in the past which changed your minds, which you would have awarded prizes to?
For example, I thought about my old comment on patient long-termism, which observes that in order to say “I’m waiting to give later” as a complete strategy you need to identify the conditions under which you would stop waiting (as otherwise, your strategy is to give never). On the one hand, it feels “too short” to be considered, but on the other hand, it seems long enough to convey its point (at least, embedded in context as it was), and so any additional length would be ‘more cost without benefit’.

vaniver 17 Jul 2020 2:41 UTC
11 points
0 ∶ 0
in reply to: Geoffrey Irving’s comment on: A list of good heuristics that the case for AI X-risk fails
Presumably there are two categories of heuristics, here: ones which relate to actual difficulties in discerning the ground truth, and ones which are irrelevant or stem from a misunderstanding. I think it seems bad that this list implicitly casts the heuristics as being in the latter category, and rather than linking to why each is irrelevant or a misunderstanding it does something closer to mocking the concern.
For example, I would decompose the “It’s not empirically testable” heuristic into two different components. The first is something like “it’s way easier to do good work when you have tight feedback loops, and a project that relates to a single shot opportunity without a clear theory simply cannot have tight feedback loops.” This was the primary reason I stayed away from AGI safety for years, and still seems to me like a major challenge to research work here. [I was eventually convinced that it was worth putting up with this challenge, however.]
The second is something like “only trust claims that have been empirically verified”, which runs into serious problems with situations where the claims are about the future, or running the test is ruinously expensive. A claim that ‘putting lamb’s blood on your door tonight will cause your child to be spared’ is one that you have to act on (or not) before you get to observe whether or not it will be effective, and so whether or not this heuristic helps depends on whether or not it’s possible to have any edge ahead of time on figuring out which such claims are accurate.

vaniver 5 Nov 2020 17:03 UTC
9 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Thoughts on whether we’re living at the most influential time in history
Then, if the expected cost-effectiveness of the best opportunities varies substantially over time, there will be just one point in time at which your philanthropy will have the most impact, and you should try to max out your philanthropy at that time period, donating all your philanthropy at that time if you can.
Tho I note that the only way one would ever take such opportunities, if offered, is by developing a view of what sorts of opportunities are good that is sufficiently motivating to actually take action at least once every few decades.
For example, when the most attractive opportunity so far appears in year 19 of investing and assessing opportunities, will our patient philanthropist direct all their money towards it, and then start saving again? Will they reason that they don’t have sufficient evidence to overcome their prior that year 19 is not more attractive than the years to come? Will they say “well, I’m following the Secretary Problem solution, and 19 is less than 70/e, so I’m still in info-gathering mode”?
They won’t, of course, know which path had higher value in their particular world until they die, but it seems to me like most of the information content of a strategy that waits to pull the trigger is in when it decides to pull the trigger, and this feels like the least explicit part of your argument.
Compare to investing, where some people are fans of timing the market, and some people are fans of dollar-cost-averaging. If you think the attractiveness of giving opportunities is going to be unpredictably volatile, then doing direct work or philanthropy every year is the optimal approach. If instead you think the attractiveness of giving opportunities is predictably volatile, or predictably stable, then doing patient philanthropy makes more sense.
What seems odd to me is simultaneously holding the outside view sense that we have insufficient evidence to think that we’re correctly assessing a promising opportunity now, and having the sense that we should expect that we will correctly assess the promising opportunities in the future when they do happen.
What links here?
- vaniver's comment on Announcing a contest: EA Criticism and Red Teaming by Lizka (3 Jun 2022 16:51 UTC; 11 points)

Weight by Impact

vaniver21 May 2023 14:37 UTC

7 points

0 comments1 min readEA link

vaniver 2 Sep 2020 19:13 UTC
7 points
0 ∶ 0
in reply to: Lukas_Gloor’s comment on: Some thoughts on the EA Munich // Robin Hanson incident
I felt confused about why I was presented with a fully general argument for something I thought I indicated I already considered.
In my original comment, I was trying to resolve the puzzle of why something would have to appear edgy instead of just having fewer filters, by pointing out the ways in which having unshared filters would lead to the appearance of edginess. [On reflection, I should’ve been clearer about the ‘unshared’ aspect of it.]

vaniver 28 Nov 2019 0:05 UTC
7 points
0 ∶ 0
in reply to: RobBensinger’s comment on: I’m Buck Shlegeris, I do research and outreach at MIRI, AMA
I certainly don’t think agents “should” try to achieve outcomes that are impossible from the problem specification itself.
I think you need to make a clearer distinction here between “outcomes that don’t exist in the universe’s dynamics” (like taking both boxes and receiving $1,001,000) and “outcomes that can’t exist in my branch” (like there not being a bomb in the unlucky case). Because if you’re operating just in the branch you find yourself in, many outcomes whose probability an FDT agent is trying to affect are impossible from the problem specification (once you include observations).
And, to be clear, I do think agents “should” try to achieve outcomes that are impossible from the problem specification including observations, if certain criteria are met, in a way that basically lines up with FDT, just like agents “should” try to achieve outcomes that are already known to have happened from the problem specification including observations.
As an example, if you’re in Parfit’s Hitchhiker, you should pay once you reach town, even though reaching town has probability 1 in cases where you’re deciding whether or not to pay, and the reason for this is because it was necessary for reaching town to have had probability 1.

vaniver 5 Nov 2020 17:13 UTC
6 points
0 ∶ 0
in reply to: Buck’s comment on: Thoughts on whether we’re living at the most influential time in history
What I’m saying is that if you believe that x-risk is 0.1%, then you think we’re at least one in a million.
I think you’re saying “if you believe that x-risk this century is 0.1%, then survival probability this century is 99.9%, and for total survival probability over the next trillion years to be 0.01%, there can be at most 9200 centuries with risk that high over the next trillion years (.999^9200=0.0001), which means we’re in (most generously) a one-in-one-million century, as a trillion years is 10 billion centuries, which divided by ten thousand is a million.” That seem right?

vaniver 10 Aug 2020 21:55 UTC
6 points
0 ∶ 0
in reply to: dominicroser’s comment on: Long-term investment fund at Founders Pledge
Benjamin Franklin, in his will, left £1,000 pounds each to the cities of Boston and Philadelphia, with the proviso that the money should be invested for 100 years, with 25 percent of the principal to be invested for a further 100 years.
Also of note is that he gave conditions on the investments; the money was to be lent to married men under 25 who had finished an apprenticeship, with two people willing to co-sign the loan for them. So in that regard it was something like a modern microlending program, instead of just trying to maximize returns for benefits in the future.

vaniver

Pre­dic­tion Mar­kets for Science

Weight by Impact

Prediction Markets for Science