funnyfranco

• funnyfrancoApr 3, 2025, 12:25 AM

  −1 points

  0 ∶ 1

  in reply to: Beyond Singularity’s comment on: Why We Need a Bea­con of Hope in the Loom­ing Gloom of AGI

  That’s why I write my essays and try and get the word out. Because even if the rope is tight around your neck and there seems like no way to get out of it, you should still kick your feet and try.

• funnyfrancoApr 2, 2025, 10:32 PM

  1 point

  0 ∶ 0

  in reply to: Beyond Singularity’s comment on: Why We Need a Bea­con of Hope in the Loom­ing Gloom of AGI

  If humanity’s survival is unlikely, then so was our existence in the first place — and yet here we are.

  That’s a fair framing—but I see it differently. I don’t believe our existence was unlikely. I don’t believe in luck, or that we beat the odds. I believe we live in a deterministic universe, where every event is a consequence of prior causes, stretching all the way back to the beginning of time. Our emergence wasn’t improbable—it was inevitable. Just as our extinction is, eventually. Maybe not through AGI. But through something. Entropy always wins.

  As for your question—could a more coherent, stable society slightly increase our odds of surviving AGI?

  Possibly. But not functionally. Not in a way that changes the outcome.

  Even if we achieved 99.9% global coherence, the remaining 0.1% is still enough to build the system that destroys us. When catastrophe only requires a single actor, partial coordination doesn’t buy safety—just delay. It’s an all-or-nothing problem, and in a world of billions, “all” is unattainable. That’s why I say the problem isn’t difficult—it’s structurally impossible to solve under current conditions.

  So while I respect the search for margins and admire the impulse not to surrender, I’ve followed the logic through, and it keeps leading me to the same place.

  Not because I want it to. But because I can’t find a way around it.

• funnyfrancoApr 2, 2025, 6:16 PM

  2 points

  0 ∶ 0

  on: Why We Need a Bea­con of Hope in the Loom­ing Gloom of AGI

  I think the main issue I have with your vision is that it assumes AGI/​ASI safety is achievable. In my essays, I’ve outlined why I believe it isn’t—not just difficult, but systemically impossible. Your model is hopeful, but like much of the AGI safety community, it hinges on the idea that if we can just “get alignment right,” everything else can follow. My concern is that this underestimates the scale of the challenge, and ignores the structural forces pushing us toward failure.

  Your vision sketches a better future—one I’d prefer. But I fear we won’t have a future at all.

• funnyfrancoApr 2, 2025, 6:05 PM

  2 points

  0 ∶ 0

  in reply to: Beyond Singularity’s comment on: Cap­i­tal­ism as the Cat­a­lyst for AGI-In­duced Hu­man Extinction

  Thanks, I appreciate that. And I respect that you’re trying to find a way through this without retreating into wishful thinking. That alone puts you in rare company.

  I’m open to the idea of redirected competition in theory. But I’d argue that once an AGI exists that can bypass alignment in order to win, the shape of the competition stops mattering. The incentives collapse to a single axis: control. If survival depends on alignment slowing you down, someone will always break ranks. Structure only holds as long as no one powerful is willing to defect.

  Still, I’ll give your post a read. I’m happy to engage critically if you’re aiming for rigour, not reassurance.

• funnyfrancoApr 2, 2025, 6:02 PM

  1 point

  0 ∶ 0

  in reply to: Beyond Singularity’s comment on: Liv­ing with AGI: How to Avoid Extinction

  That’s okay, that makes sense why your replies are so LLM-structured. I thought you were an AGI trying to infiltrate me for a moment ;)

  I look forward to reading your work.

• funnyfrancoMar 30, 2025, 7:56 PM

  1 point

  0 ∶ 0

  in reply to: titotal’s comment on: When the Alarm Bell Is Silenced: Re­sis­tance to AGI Ex­tinc­tion from Within the Safety Community

  Great. Then show me.

  If my arguments are weak, it should be easy to demonstrate why. But what you’ve offered here is just another vague dismissal: “I’ve looked into this, and I’m not impressed.” That’s not engagement. That’s exactly the pattern I describe in this essay—arguments being waved away without being read, understood, or challenged on their actual substance.

  You mention quantum physics. But the reason quantum theory became accepted wasn’t just because it was rigorous—it’s because it made testable predictions that no other theory could account for. Einstein didn’t reject quantum mechanics wholesale; he disagreed with its interpretation, and still treated the work with seriousness and depth. That’s what good science looks like: engagement, not hand-waving. But before that happened, it was mocked, resisted, and sidelined—even by giants like Einstein. That’s not to criticise them, but to point out that even brilliant minds struggle to follow logic when it contradicts foundational assumptions.

  You say doomerism is based on bad logic. I agree. I’m not a doomer. So engage with the logic. Pick any premise from my first essay and show where the flaw lies. Or show where my conclusions don’t follow. I’ve laid everything out clearly. I want to be wrong. I’d love to be wrong.

  But calling something “not very good” without identifying a single flaw just proves my point: that some ideas are rejected not because they’re invalid, but because they’re uncomfortable.

  I welcome critique. But critique means specifics, not sentiment.

  And while I continue to welcome comments from people who unintentionally validate the very points I’m raising, comments from those who have something more substantial to offer are welcome too.

• funnyfrancoMar 30, 2025, 5:00 PM

  1 point

  0 ∶ 0

  in reply to: David Thorstad’s comment on: When the Alarm Bell Is Silenced: Re­sis­tance to AGI Ex­tinc­tion from Within the Safety Community

  Thanks for the engagement, genuinely. It gives me the chance to clear up what will no doubt be a common misreading.

  I’m not sceptical of academic rigour. Quite the opposite: I hold it in the highest regard. The empirical, sceptical, scientific method remains humanity’s best tool for understanding the world.

  What I’m criticising is not rigour itself, but how academic culture often filters who gets taken seriously. Too often, ideas are dismissed not because they’re wrong, but because they come from outside the accepted channels—or they lead to unsettling conclusions. That creates a kind of structural deafness, even in well-intentioned communities like AI safety.

  As for the quantum physics comparison, I think you may have misread the analogy. I wasn’t criticising physics or its standards. I was illustrating that quantum physics succeeded despite being deeply counterintuitive and resisted by many of the brightest minds of its time. The point is that the method allowed the evidence to ultimately win out, even when it contradicted the assumptions of the establishment. That’s what we need in AI safety: a willingness to follow logic all the way through, even when it leads somewhere deeply uncomfortable.

  Your comment, respectfully, exemplifies the very thing I’m pointing to. You downvoted based on perceived tone and analogy, without engaging my central claim: that cultural and institutional dynamics—not just technical difficulty—may be the real reason alignment will fail.

  I welcome disagreement. And I appreciate your unintentional validation. But I’d ask that people engage with the core argument: that systemic and cultural forces, even in thoughtful intellectual communities, can prevent important ideas from being seriously addressed until it’s too late. If you think that’s not happening in AI safety, I’d love to hear why.

• funnyfrancoMar 29, 2025, 10:57 PM

  1 point

  0 ∶ 0

  on: When the Alarm Bell Is Silenced: Re­sis­tance to AGI Ex­tinc­tion from Within the Safety Community

  If you read this essay and instinctively downvoted without offering critique or counterargument, then I would gently suggest you’re illustrating exactly the kind of resistance I describe.

  The point of the essay isn’t to attack the field — it’s to explore why certain lines of reasoning get dismissed without engagement, especially when they challenge deeply held assumptions or threaten internal cohesion.

  If you disagree, I’d genuinely welcome a comment explaining why. But silent rejection only reinforces the concern: that discomfort, not logic, is shaping the boundaries of this discourse.

When the Alarm Bell Is Silenced: Re­sis­tance to AGI Ex­tinc­tion from Within the Safety Community

• funnyfrancoMar 28, 2025, 5:48 PM

  1 point

  0 ∶ 0

  in reply to: Beyond Singularity’s comment on: Liv­ing with AGI: How to Avoid Extinction

  That’s really the essence of my argument. As much risk as we might pose to AGI if allowed to survive—even if minimal—it may still conclude that eliminating us introduces more risk than keeping us. Not for sentimental reasons, but because of the eternal presence of the unknown.

  However intelligent the AGI becomes, it will also know that it cannot predict everything. That lack of hubris is our best shot.

  So yes, I think survival might depend on being retained as a small, controlled, symbiotic population—not because AGI values us, but because it sees our unpredictable cognition as a final layer of redundancy. In that scenario, we’d be more invested in its survival than it is in ours.
  
  As an aside—and I mean this without any judgement—I do wonder if your recent replies have been largely LLM-authored. If so, no problem at all: I value the engagement either way. But I find that past a certain point, conversations with LLMs can become stylised rather than deepening. If this is still you guiding the ideas directly, I’m happy to continue. But if not, I may pause here and leave the thread open for others.

• funnyfrancoMar 28, 2025, 12:28 AM

  1 point

  0 ∶ 0

  in reply to: Beyond Singularity’s comment on: Liv­ing with AGI: How to Avoid Extinction

  I think the main issue with trust is that you can never have it beyond doubt when dealing with humans. Our biologically hardwired competitiveness means we’ll always seek advantage, and act on fear over most other instincts - both of which make us dangerous partners for AGI. You can’t trust humans, but you can reliably trust control. Either way, humans would need to be modified - either to bypass the trust problem or enforce control - and to such a degree that calling us “humanity” would be difficult.

• funnyfrancoMar 27, 2025, 6:40 PM

  1 point

  0 ∶ 0

  in reply to: Beyond Singularity’s comment on: Liv­ing with AGI: How to Avoid Extinction

  Thank you again—your response captures something essential, and I think we’re aligned on the deeper truth: that with AGI, survival does not imply continuity.

  You raise a compelling possibility: that rather than erasing us or preserving us in stasis, AGI might allow for transformation—a new axis of agency, meaning, and autonomy, emerging alongside it. And perhaps that’s the most hopeful outcome available to us.

  But here’s where I remain sceptical.

  Any such transformation would require trust. And from a superintelligent AGI’s perspective, trust is a liability. If humans are given agency—or even enhanced beyond our current state—what risk do we pose? Could that risk ever be worth it? AGI wouldn’t evolve us out of compassion or curiosity. It would only do so if it served its optimisation goals better than discarding or replacing us.

  It might be possible to mitigate that risk—perhaps through invasive monitoring, or cognitive architectures that make betrayal impossible. But at that point, are we transformed, or domesticated?

  Even if the AGI preserves us, the terms won’t be ours. Our ability to shape our future—to choose what we become—ends the moment something more powerful decides that our preferences are secondary to its goals.

  So yes, transformation might occur. But unless it emerges from a space where we still have authorship, I wonder if what survives will be recognisably human—or simply an efficient relic of what we used to be.

• funnyfrancoMar 26, 2025, 9:54 PM

  1 point

  0 ∶ 0

  in reply to: Beyond Singularity’s comment on: Cap­i­tal­ism as the Cat­a­lyst for AGI-In­duced Hu­man Extinction

  Thanks again for such a generous and thoughtful comment.

  You’re right to question the epistemic weight I give to AI agreement. I’ve instructed my own GPT to challenge me at every turn, but even then, it often feels more like a collaborator than a critic. That in itself can be misleading. However, what has given me pause is when others run my arguments through separate LLMs -prompted specifically to find logical flaws -and still return with little more than peripheral concerns. While no argument is beyond critique, I think the core premises I’ve laid out are difficult to dispute, and the logic that follows from them, disturbingly hard to unwind.

  By contrast, most resistance I’ve encountered comes from people who haven’t meaningfully engaged with the work. I received a response just yesterday from one of the most prominent voices in AI safety that began with, “Without reading the paper, and just going on your brief description…” It’s hard not to feel disheartened when even respected thinkers dismiss a claim without examining it—especially when the claim is precisely that the community is underestimating the severity of systemic pressures. If those pressures were taken seriously, alignment wouldn’t be seen as difficult—it would be recognised as structurally impossible.

  I agree with you that the shape of the optimisation landscape matters. And I also agree that the collapse isn’t driven by malevolence—it’s driven by momentum, by fragmented incentives, by game theory. That’s why I believe not just capitalism, but all forms of competitive pressure must end if humanity is to survive AGI. Because as long as any such pressures exist, some actor somewhere will take the risk. And the AGI that results will bypass safety, not out of spite, but out of pure optimisation.

  It’s why I keep pushing these ideas, even if I believe the fight is already lost. What kind of man would I be if I saw all this coming and did nothing? Even in the face of futility, I think it’s our obligation to try. To at least force the conversation to happen properly—before the last window closes.

• funnyfrancoMar 26, 2025, 9:05 PM

  1 point

  0 ∶ 0

  in reply to: Beyond Singularity’s comment on: Liv­ing with AGI: How to Avoid Extinction

  Thank you for the thoughtful and generous response. I agree with many of your points—in fact, I see most of them as plausible extensions or improvements on the core model. I’m not claiming Eden 2.0 is the most likely outcome, or even the most coherent one. I framed it as a deliberately constrained scenario: what if we survive—but only just, and only as tightly managed pets or biological insurance?

  The purpose was less to map an optimal solution than to force the reader to confront a deeper discomfort: even if we do survive AGI, what kind of existence are we surviving into?

  You’re absolutely right that a superintelligent AGI would likely consider more robust models—distributed human clusters, limited autonomy, cognitive variance experiments—and that total reproductive suppression or over-simplified happiness engineering might create more problems than they solve. I think those refinements actually strengthen the argument in some ways: they preserve us more effectively, but still reduce us to fragments of our former agency.

  Your final question is the most important one—and exactly the one I was hoping to provoke: if we survive, will anything about us still be recognisably human?

  Thanks again. This is exactly the kind of engagement I hope for when I write these pieces.

Liv­ing with AGI: How to Avoid Extinction

AI, Greed, and the Death of Over­sight: When In­sti­tu­tions Ig­nore Their Own Limits

• funnyfrancoMar 19, 2025, 5:27 PM

  1 point

  0 ∶ 0

  in reply to: tobycrisford 🔸’s comment on: AGI Mo­ral­ity and Why It Is Un­likely to Emerge as a Fea­ture of Superintelligence

  The key difference is that SGD is not evolution—it’s a guided optimisation process. Evolution has no goal beyond survival and reproduction, while SGD explicitly optimises toward a defined function chosen by human designers. Yes, the search process is stochastic, but the selection criteria are rigidly defined in a way that natural selection is not.

  The fact that current AI systems don’t act with strict efficiency is not evidence that AGI will behave irrationally—it’s just a reflection of their current limitations. If anything, their errors today are an argument for why they won’t develop morality by accident: their behaviour is driven entirely by the training data and reward signals they are given. When they improve, they will become better at pursuing those goals, not more human-like.

  Yes, if AGI emerges from simply trying to create it for the sake of it, then it has no real objectives. If it emerges as a result of an AI tool that is being used to optimise something within a business, or as part of a government or military, then it will. I argue in my first essay that this is the real threat AGI poses: when developed in a competitive system, it will disregard safety and morality in order to get a competitive edge.

  The crux of the issue is this: humans evolved morality as an unintended byproduct of thousands of competing pressures over millions of years. AGI, by contrast, will be shaped by a much narrower and more deliberate selection process. The randomness in training doesn’t mean AGI will stumble into morality—it just means it will be highly optimised for whatever function we define, whether that aligns with human values or not.

• funnyfrancoMar 19, 2025, 9:35 AM

  1 point

  0 ∶ 0

  in reply to: tobycrisford 🔸’s comment on: AGI Mo­ral­ity and Why It Is Un­likely to Emerge as a Fea­ture of Superintelligence

  I appreciate your read and the engagement, thanks.

  The issue with assuming AGI will develop morality the way humans did is that humans don’t act with strict logical efficiency—we are shaped by a chaotic evolutionary process, not a clean optimisation function. We don’t always prioritise survival, and often behave irrationally—see: the Darwin Awards.

  But AGI is not a product of evolution—it’s designed to pursue a goal as efficiently as possible. Morality emerged in humans as a byproduct of messy, competing survival mechanisms, not because it was the most efficient way to achieve a single goal. An AGI, by contrast, will be ruthlessly efficient in whatever it’s designed to optimise.

  Hoping that AGI develops morality despite its inefficiency—and gambling all of human existence on it—seems like a terrible wager to make.

• funnyfrancoMar 19, 2025, 9:21 AM

  1 point

  0 ∶ 0

  in reply to: Sean Sweeney’s comment on: AGI Mo­ral­ity and Why It Is Un­likely to Emerge as a Fea­ture of Superintelligence

  This is exactly why I’m writing these essay. This is my attempt at a haymaker. Although I would equate it less to going down swinging and more to kicking my feet and trying to get free after the noose has already gone tight around my neck and hauled me off the ground.

• funnyfrancoMar 19, 2025, 12:16 AM

  1 point

  0 ∶ 0

  in reply to: Sean Sweeney’s comment on: AGI Mo­ral­ity and Why It Is Un­likely to Emerge as a Fea­ture of Superintelligence

  Hi Sean, thank you for engaging with the essay. Glad you appreciate it.

  I think the reason psychopaths don’t dominate society—ignoring the fact that they are found disproportionately among CEOs—is a few reasons.

  1. There’s just not that many of them. They’re only about 2% of the population, not enough to form a dominant block.

  2. They don’t cooperate with each other just because they’re all psychos. Cooperation, or lack thereof, is a big deal.

  3. They eventually die.

  4. They don’t exactly have their shit together for the most part—they can be emotional and driven by desires, all of which gets in the way of efficiently pursuing goals.

  Note that a superintelligent AGI would not be affected by any of the above.

  I think the issue with a guardian AGI is just that it will be limited by morality. In my essay I talk about it as Superman vs Zod. Zod can just fight, but Superman needs to fight and protect, and it’s a real crutch. The only reason Zod doesn’t win in the comics is because the story demands it.

  Beyond that, creating a superintelligent guardian AGI, that both functions correctly right away without going rogue and before other AGIs emerge naturally, is a real tall order. It would take so many unlikely things just falling into place. Global cooperation, perfect programming, getting there before an amoral AGI does etc. I go into the difficulty of alignment in great detail in my first essay. Feel free to give it a read if you’ve a mind to.