Sharmake

Karma: 937

Sharmake Mar 12, 2025, 5:41 PM
7 points
0 ∶ 0
in reply to: Eevee🔹’s comment on: evelynciara’s Shortform
Late comment, I basically agree with the point being made here that we should avoid committing a fallacy of assuming work done is constant/lump of labor fallacies, but I don’t think this weakens the argument that human work will be replaced by AI work totally, for 2 reasons:
1. In a world where you can copy AI labor hugely readily, wages fall for the same reason why prices fall when more goods are supplied, and in particular humans have a biological minimum wage of 20-100 watts that fundamentally makes them unemployable once AIs can be run for cheaper than this, and human wages are likely to fall below subsistence if AIs are copied hugely.
2. While more work will happen from growing the economy, it is still better to invest in AIs to do the work than it is to invest in humans, and thus even while labor grows, human labor specifically can fall to essentially 0, so the automation hypothesis is at least a consistent hypothesis to hold economically.

Sharmake Mar 10, 2025, 8:18 PM
1 point
0 ∶ 0
on: How confident are you that it’s preferable for America to develop AGI before China does?
To be honest, even if we grant the assumption that AI alignment is achieved and it matters who achieves AGI/ASI, I’d be much, much less confident in America racing, and think that it’s weakly negative to race.

One big reason for this is that the pressures AGI introduces are closer to cross-cutting pressures than pressures that are dependent on nations, like the intelligence curse sort of scenario where elites have incentives to invest in their automated economy, and leave the large non-elite population to starve/be repressed:

https://lukedrago.substack.com/p/the-intelligence-curse

Sharmake Jan 24, 2025, 7:30 PM
1 point
0 ∶ 0
in reply to: Greg_Colbourn ⏸️ ’s comment on: Are AI safetyists crying wolf?
I think this might not be irrationality, but a genuine difference in values.

In particular, I think something like a discount rate disagreement is at the core of a lot of disagreements on AI safety, and to be blunt, you shouldn’t expect convergence unless you successfully persuade them of this.

Sharmake Nov 23, 2024, 11:15 PM
5 points
1 ∶ 0
on: Donation Election Discussion Thread
I ultimately decided to vote for the animal welfare groups, because I believe that animal welfare, in both it’s farmed and wild variants is probably one of the most robust and large problems in the world, and with the exception of groups that are the logistical/epistemic backbone of the movements (they are valuable for gathering data and making sure that the animal welfare groups can do their actions), I’ve become more skeptical that other causes were robustly net-positive, especially reducing existential risks.

Sharmake Nov 20, 2024, 4:51 PM
12 points
1 ∶ 0
in reply to: sapphire’s comment on: US government commission pushes Manhattan Project-style AI initiative
This sounds very much like the missile gap/bomber gap narrative, and yeah this is quite bad news if they actually adopt the commitments pushed here.

The evidence that China is racing to AGI is quite frankly very little, and I see a very dangerous arms race that could come:

https://forum.effectivealtruism.org/posts/cXBznkfoPJAjacFoT/are-you-really-in-a-race-the-cautionary-tales-of-szilard-and

Sharmake Oct 29, 2024, 12:43 AM
3 points
0 ∶ 0
on: Tentatively against making AIs ‘wise’
I honestly agree with this post, and to best translate this into my own thinking, we should rather have AI that is superhuman at faithful COT reasoning than it is at wise forward pass thinking.

The Peter Singer/Einstein/Legible reasoning corresponds to COT reasoning, whereas a lot of the directions for intuitive wise/illegible thinking depend on making the forward pass thinking more capable, which is not a great direction for reasons of trust and alignment.

Sharmake Oct 23, 2024, 9:18 PM
1 point
0 ∶ 0
in reply to: Sharmake’s comment on: Linch’s Shortform
In retrospect, I agree more with 3, and while I do still think AI timelines are plausibly very short, I do think that after-2030 timelines are reasonably plausible from my perspective.

I have become less convinced that takeoff speed from the perspective of the state will be slow, slightly due to entropix reducing my confidence in a view where algorithmic progress doesn’t suddenly go critical and make AI radically better, and more so because I now think there will be less flashy/public progress, and more importantly I think the gap between consumer AI and internal AI used in OpenAI will only widen, so I expect a lot of the GPT-4 moments where people wowed and got very concerned at AI to not happen again.

So I expect the landscape of AI governance to have less salience when AIs can automate AI research than the current AI governance field thinks, which means overall I’ve reduced my probability of a strong societal response from say 80-90% likely to only 45-60% likely.

Sharmake Sep 17, 2024, 2:50 PM
3 points
0 ∶ 0
in reply to: SummaryBot’s comment on: My disagreements with “AGI ruin: A List of Lethalities”
This is mostly correct as a summary of my position, but for point 6, I want to point out while this is technically true, I do fear economic incentives are against this path.

Agree with the rest of the summary though.

My disagreements with “AGI ruin: A List of Lethalities”

SharmakeSep 15, 2024, 5:22 PM

16 points

2 comments1 min readEA link

Sharmake Feb 22, 2024, 9:03 PM
1 point
0 ∶ 0
in reply to: Linch’s comment on: Linch’s Shortform
I basically grant 2, sort of agree with 1, and drastically disagree with three (that timelines will be long.)

Which makes me a bit weird, since while I do have real confidence in the basic story that governments are likely to influence AI a lot, I do have my doubts that governments will try to regulate AI seriously, especially if timelines are short enough.

Sharmake Dec 15, 2023, 7:34 PM
42 points
15 ∶ 2
in reply to: DPiepgrass’s comment on: Nonlinear’s Evidence: Debunking False and Misleading Claims
Yeah, at least several comments have much more severe issues than tone or stylistic choices, like rewording ~every claim by Ben, Chloe and Alice, and then assuming that the transformed claims had the same truth value as the original claim.

I’m in a position very similar to Yarrow here: While I think Kat Woods has mostly convinced me that the most incendiary claims are likely false, and I’m sympathetic to the case for suing Ben and Habryka, there was dangerous red flags in the responses, so much so that I’d stop funding Nonlinear entirely, and I think it’s quite bad that Kat Woods responded the way they did.

Sharmake Nov 21, 2023, 4:06 PM
5 points
0 ∶ 0
in reply to: David Mathers🔸’s comment on: JWS’s Shortform
I unendorsed primarily because apparently, the board didn’t fire because of safety concerns, though I’m not sure this is accurate.

Sharmake Nov 20, 2023, 4:31 PM
6 points
1 ∶ 1
on: Sam Altman fired from OpenAI
It seems like the board did not fire Sam Altman for safety reasons, but instead for other reasons instead. Utterly confusing, and IMO demolishes my previous theory, though a lot of other theories also lost out.

Sources below, with their archive versions included:

https://twitter.com/norabelrose/status/1726635769958478244

https://twitter.com/eshear/status/1726526112019382275

https://archive.is/dXRgA

https://archive.is/FhHUv

Sharmake Nov 20, 2023, 3:32 AM
11 points
2 ∶ 2
in reply to: Lukas_Gloor’s comment on: JWS’s Shortform
While I generally agree that they almost certainly have more information on what happened, which is why I’m not really certain on this theory, my main reason here is that for the most part, AI safety as a cause basically managed to get away with incredibly weak standards of evidence for a long time, until the deep learning era in 2019-, especially with all the evolution analogies, and even now it still tends to have very low standards (though I do believe it’s slowly improving right now). This probably influenced a lot of EA safetyists like Ilya, who almost certainly imbibed the norms of the AI safety field, and one of them is that there is a very low standard of evidence needed to claim big things, and that’s going to conflict with corporate/legal standards of evidence.

But I don’t think most people who hold influential positions within EA (or EA-minded people who hold influential positions in the world at large, for that matter) are likely to be that superficial in their analysis of things. (In particular, I’m strongly disagreeing with the idea that it’s likely that the board “basically had no evidence except speculation from the EA/LW forum”. I think one thing EA is unusually good at – or maybe I should say “some/many parts of EA are unusually good at” – is hiring people for important roles who think for themselves and have generally good takes about things and acknowledge the possibility of being wrong about stuff. [Not to say that there isn’t any groupthink among EAs. Also, “unusually good” isn’t necessarily that high of a bar.])

I agree with this weakly, in the sense that being high up in EA is at least a slight update towards them actually thinking through things and being able to make actual cases. My disagreement here is that this effect is probably not strong enough to wash away the cultural effects of operating in a cause area where they don’t need to meet any standard of evidence except long-winded blog posts and getting rewarded, for many reasons.

Also, the board second-guessed it’s decision, which would be evidence for the theory that they couldn’t make a case that actually abided to the standard of evidence for a corporate/legal setting.

If it was any other cause like say GiveWell or some other causes in EA, I would trust them much more that they do have good reason. But AI safety has been so reliant on very low-non-existent standards of evidence or epistemics that they probably couldn’t explain themselves in a way that would abide by the strictness of a corporate/legal standard of evidence.

Edit: The firing wasn’t because of safety related concerns.

Sharmake Nov 19, 2023, 9:49 PM
7 points
1 ∶ 4
in reply to: JWS 🔸’s comment on: JWS’s Shortform

If Ilya can say “we’re pushing capabilities down a path that is imminently highly dangerous, potentially existentially, and Sam couldn’t be trusted to manage this safely” with proof that might work—but then why not say that?

I suspect this is due to the fact that quite frankly, the concerns they had about Sam Altman being unsafe on AI basically had no evidence except speculation from the EA/LW forum, which is not enough evidence at all in the corporate world/legal world, and to be quite frank, the EA/LW standard of evidence on AI risk being a big deal enough to investigate is very low, sometimes non-existent, and that simply does not work once you have to deal with companies/the legal system.

More generally, EA/LW is shockingly loose, sometimes non-existent in its standards of evidence for AI risk, which doesn’t play well with the corporate/legal system.

This is admittedly a less charitable take than say, Lukas Gloor’s take.

Sharmake Nov 19, 2023, 7:41 PM
3 points
1 ∶ 0
in reply to: JWS 🔸’s comment on: JWS’s Shortform
My general thoughts on this can be stated as: I’m mostly of the opinion that EA will survive this, bar something massively wrong like the board members willfully lying or massive fraud from EAs, primarily because most of the criticism is directed to the AI safety wing, and EA is more than AI safety, after all.

Nevertheless, I do think that this could be true for the AI safety wing, and they may have just hit a key limit to their power. In particular, depending on how this goes, I could foresee a reduction in AI safety power and influence, and IMO this was completely avoidable.

Sharmake Nov 19, 2023, 2:54 AM
13 points
11 ∶ 1
in reply to: JWS 🔸’s comment on: Sam Altman fired from OpenAI
Yeah, this is one of the few times where I believe that the EAs on the board likely overreached here, because they probably didn’t give enough evidence to justify their excoriating statement there that Sam Altman was dishonest, and he might be coming back to lead the company.

I’m not sure how to react to all of this, though.

Edit: My reaction is just WTF happened, and why did they completely play themselves? Though honestly, I just believe that they were inexperienced.

Sharmake Nov 2, 2023, 10:07 PM
4 points
1 ∶ 0
in reply to: JWS 🔸’s comment on: My thoughts on the social response to AI risk

The Bay Area rationalist scene is a hive of techno-optimisitic libertarians.[1] These people have a negative view of state/government effectiveness at a philosophical and ideological level, so their default perspective is that the government doesn’t know what it’s doing and won’t do anything. [edit: Re-reading this paragraph it comes off as perhaps mean as well as harsh, which I apologise for]

Yeah, I kinda of have to agree with this, I think the Bay Area rationalist scene underrates government competence, though even I was surprised at how little politicking happened, and how little it ended up being politicized.

Similary, ‘Politics is the Mind-Killer’ might be the rationalist idea that has aged worst—especially for its influences on EA. EA is a political project—for example, the conclusions of Famine, Affluence, and Morality are fundamentally political.

I think that AI was a surprisingly good exception to the rule that politicizing something would make it harder to get, and I think this is mostly due to the popularity of AI regulations. I will say though that there’s clear evidence that at least for now, AI safety is in a privileged position, and the heuristic no longer applies.

Overly-short timelines and FOOM. If you think takeoff is going to be so fast that we get no firealarms, then what governments do doesn’t matter. I think that’s quite a load bearing assumption that isn’t holding up too well

Not just that though, I also think being overly pessimistic around AI safety sort of contributed, as a lot of people’s mental health was almost certainly not great at best, making them catastrophize the situation and being ineffective.

This is a real issue in the climate change movement, and I expect that AI safety’s embrace of pessimism was not good at all for thinking clearly.

Thinking of AI x-risk as only a technical problem to solve, and undervaluing AI Governance. Some of that might be comparative advantage (I’ll do the coding and leave political co-ordination to those better suited). But it’d be interesting to see x-risk estimates include effectiveness of governance and attention of politicians/the public to this issue as input parameters.

I agree with this, at least for the general problem of AI governance, though I disagree if we talk about AI alignment, though I agree that rationalists underestimate the governance work required to achieve a flourishing future.

Sharmake Oct 31, 2023, 7:24 PM
−1 points
0 ∶ 1
in reply to: evhub’s comment on: AI Pause Will Likely Backfire
Okay, my crux is that the simplicity/Kolmogorov/Solomonoff prior is probably not very malign, assuming we could run it, and in general I find the prior not to be malign except for specific situations.

This is basically because it relies on the IMO dubious assumption that the halting oracle can only be used once, and notably once we use the halting/Solomonoff oracle more than once, the Solomonoff oracle loses it’s malign properties.

More generally, if the Solomonoff Oracle is duplicatable, as modern AIs generally are, then there’s a known solution to mitigate the malignancy of the Solomonoff prior: Duplicate it, and let multiple people run the Solomonoff inductor in parallel to increase the complexity of manipulation. The goal is essentially to remove the uniqueness of 1 Solomonoff inductor, and make an arbitrary number of such oracles to drive up the complexity of manipulation.

So under a weak assumption, the malignancy of the Solomonoff prior goes away. This is described well in the link below, and the important part is that we need either a use-once condition, or we need to assume uniqueness in some way. If we don’t have either assumption holding, as is likely to be the case, then the Solomonoff/Kolmogorov prior isn’t malign.

https://www.lesswrong.com/posts/f7qcAS4DMKsMoxTmK/the-solomonoff-prior-is-malign-it-s-not-a-big-deal#Comparison_

And that’s if it’s actually malign, which it might not be, at least in the large-data limit:

https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign#fDEmEHEx5EuET4FBF

More specifically, it’s this part of John Wentworth’s comment:

In Solomonoff Model, Sufficiently Large Data Rules Out Malignness

There is a major outside-view reason to expect that the Solomonoff-is-malign argument must be doing something fishy: Solomonoff Induction (SI) comes with performance guarantees. In the limit of large data, SI performs as well as the best-predicting program, in every computably-generated world. The post mentions that:

A simple application of the no free lunch theorem shows that there is no way of making predictions that is better than the Solomonoff prior across all possible distributions over all possible strings. Thus, agents that are influencing the Solomonoff prior cannot be good at predicting, and thus gain influence, in all possible worlds.

… but in the large-data limit, SI’s guarantees are stronger than just that. In the large-data limit, there is no computable way of making better predictions than the Solomonoff prior in any world. Thus, agents that are influencing the Solomonoff prior cannot gain long-term influence in any computable world; they have zero degrees of freedom to use for influence. It does not matter if they specialize in influencing worlds in which they have short strings; they still cannot use any degrees of freedom for influence without losing all their influence in the large-data limit.

Takeaway of this argument: as long as we throw enough data at our Solomonoff inductor before asking it for any outputs, the malign agent problem must go away. (Though note that we never know exactly how much data that is; all we have is a big-O argument with an uncomputable constant.)

As far as the actual practical question, there is a very important limitation on inner-misaligned agents by SGD, primarily because gradient hacking is very difficult to do, and is an underappreciated limitation on misalignment, since SGD has powerful tools to remove inner-misaligned circuits/TMs/Agents in the link below:

https://www.lesswrong.com/posts/w2TAEvME2yAG9MHeq/gradient-hacking-is-extremely-difficult

Sharmake Oct 30, 2023, 10:15 PM
11 points
2 ∶ 0
in reply to: Geoffrey Miller’s comment on: OpenAI’s new Preparedness team is hiring
I want to flag that I see quite a lot of inappropriate binarization happening, and I generally see quite a lot of dismissals of valid third options.

Either they take the extinction risks seriously, or they don’t.

There are other important possibilities, like a potential belief in AI progress helping or solving the existential risk, thinking that the intervention of increasing AI progress is actually the best strategy, etc. More generally, once we make weaker or no assumptions about AI risk, we no longer obtain the binary you’ve suggested.

So this doesn’t really work, because it basically requires us to assume the conclusion, especially for near-term people.

My conclusion is that they say they take the extinction risks seriously, but they’re lying, or they’re profoundly self-deceived. In any case, their revealed preferences are that they prefer a little extra money, power, and status for themselves over a lot of extra safety for everybody else—and for themselves.

Sharmake

My dis­agree­ments with “AGI ruin: A List of Lethal­ities”

My disagreements with “AGI ruin: A List of Lethalities”