tobycrisford 🔸

Karma: 864

tobycrisford 🔸 31 Dec 2025 16:12 UTC
1 point
0 ∶ 0
in reply to: David Mathers🔸’s comment on: Unknown Knowns: Five Ideas You Can’t Unsee
Maybe, although an election being tied is about the only way that particular example can be fuzzy, and there is a well defined process for what happens in that situation (like flipping a coin). There is ultimately only one winner, and it is possible for a single vote to make the difference.
Whether an experience is painful or not is extremely unclear, but if your metric is just something like “number of animals killed for meat each year” then again that is something well defined and precise, and it must in principle be possible to change it with an individual purchase.

tobycrisford 🔸 31 Dec 2025 8:43 UTC
5 points
1 ∶ 0
in reply to: David Mathers🔸’s comment on: Unknown Knowns: Five Ideas You Can’t Unsee
Ironically I might also be guilty of using some technical terminology incorrectly here!
I had in mind the discussion on valuing actions with imperceptible effects from the “Five Mistakes in Moral Mathematics” chapter in Reasons+Persons (relevant to all the examples mentioned in the IVT section of this post), where if I remember right Parfit makes an explicit comparison with the “paradox of the heap” (I think this is where I first came across the term).
It feels the same in that for both cases we have a function from natural numbers (number of grains of sand in our potential heap, or number of people voting/buying meat) to some other set (boolean ‘heap’ vs ‘not heap’, or winner of election, or number of animals harmed). And the point is that mathematically, this function must at some point change with the addition of a single +1 to the input, or it can never change at all. Moreover, the sum of the expected value of lots of potential additions must equal the expected value of all of them being applied together, so that if the collective has a large effect, the individual effects can’t be smaller, on average, than the collective effect divided by the number of consituents.
I suppose the point is that this paradox is non-trivial and possibly unsolved when the output is fuzzy (like whether some grains of sand are a heap or not) but trivially true when the output is precise or quantitative (like who wins an election or how many animals are harmed)?

tobycrisford 🔸 30 Dec 2025 10:09 UTC
9 points
2 ∶ 0
on: Unknown Knowns: Five Ideas You Can’t Unsee
I like this post!
The pedantic mathematician in me though didn’t like the concept in the “Intermediate Value Theorem” section being described with that name. I’m not sure that is actually the theorem being relied on here.
A couple of reasons:
- IVT only applies to functions from the real numbers to the real numbers, whereas the example you’re using it on is a mapping from a discrete set to a discrete set (or at a stretch, from rationals to rationals). IVT doesn’t apply there (e.g. the function x → x^2 on the rationals between x=0 and x=2 goes from 0 to 4 without ever passing through 2).
- I don’t think the IVT is “trivially easy”. It’s a theorem that the formal definition of continuity aligns with our intuition (which is not trivial), and really it is only true because the real numbers have been specifically constructed to make it true (it is not true over the rational numbers), and the construction of the real numbers is very weird and non-trivial!
The actual concept you’re relying on in that section is actually a lot simpler I think. It’s essentially just the “paradox of the heap”. As you say, if the value of N things is V, and the value of N+M things is W, then the sum of the changes of each addition to N must add to (W-V). You don’t need calculus for that. It’s not the Intermediate Value Theorem. It really is just “trivially easy” I think. I’m not sure if that theorem has a name.

tobycrisford 🔸 30 Nov 2025 9:04 UTC
10 points
2 ∶ 1
on: Why Does EA Focus on Veganism but Not Other Boycotts?
There is a big difference between veganism and most(?) other boycott campaigns. Every time you purchase an animal product then you are causing significant direct harm (in expectation, if you accept the vegan argument). This is because if demand for animal products increases by 1, then we should expect some fraction more of that product to be produced to meet that demand, on average (the particular fraction depending on price elasticity, since you also raise prices a bit which puts other consumers off).
A lot of other boycott campaigns aren’t like this. For example, take the boycott of products which have been tested on animals. Here you don’t do direct harm with each purchase in the same way (or at least if you do, it is probably orders of magnitude less). Instead, the motivation is that if enough people start acting like this, it will lead to policy change.
In the first case, it doesn’t matter if no one else in the world agrees with you, participating in the boycott can still do significant good. In the second case, a large number of people are required in order for it to have meaningful impact. It makes sense that impact minded EAs are more inclined to support a boycott of the first kind.
I think a lot of your examples probably fall under the second kind (though not all). And I think that’s a big part of the answer to your question. Also, for at least some of the ones in the first kind, I think most EAs probably just disagree with the fundamental argument. For example, the environmental impact of using LLMs isn’t actually that bad: https://andymasley.substack.com/p/a-cheat-sheet-for-conversations-about.

tobycrisford 🔸 24 Nov 2025 18:07 UTC
6 points
0 ∶ 0
in reply to: Yarrow Bouchard 🔸’s comment on: Unsolved research problems on the road to AGI
I don’t disagree with much of this comment (to the extent that it puts o3′s achievement in its proper context), but I think this is still inconsistent with your original “no progress” claim (whether the progress happened pre or post o3′s ARC performance isn’t really relevant). I suppose your point is that the “seed of generalization” that LLMs contain is so insignificant that it can be rounded to zero for practical purposes? That was true pre o3 and is still true now? Is that a fair summary of your position? I still think “no progress” is too bold!
But in addition, I think I also disagree with you that there is nothing exciting about o3′s ARC performance.
It seems obvious that LLMs have always had some ability to generalize. Any time that they produce a coherent response that has not appeared verbatim in their training data, they are doing some kind of generalisation. And I think even Chollet has always acknowledged that too. I’ve heard him characterize LLMs (pre ARC success) as combining dense sampling of the problem space with an extremely weak ability to generalize, contrasting that with the ability of humans to learn from only a few examples. But there is still an acknowledgement here that some non-zero generalization is happening.
But if this is your model of how LLMs work, that their ability to generalize is extremely weak, then you don’t expect them to be able to solve ARC problems. They shouldn’t be able to solve ARC problems even if they had access to unlimited inference time compute. Ok, so o3 had 1,024 attempts at each task, but that doesn’t mean it tried the task 1,024 times until it hit on the correct answer. That would be cheating. It means it tried the task 1,024 times and then did some statistics on all of its solutions before providing a single guess, which turned out to be right most of the time!
I think it is surprising and impressive that this worked! This wouldn’t have worked with GPT-3. You could have given it chain of thought prompting, let it write as much as it wanted per attempt, and given it a trillion attempts at each problem, but I still don’t think you would expect to find the correct answer dropping out at the end. In at least this sense, o3 was a genuine improvement in generalization ability.
And Chollet thought it was impressive too, describing it as a “genuine breakthrough”, despite all the caveats that go with that (that you’ve already quoted).
When LLMs can solve a task, but only with masses of training data, then I think it is fair to contrast their data efficiency with that of humans and write off their intelligence as memorization rather than generalization. But when they can only solve a task by expending masses of inference time compute, I think it is harder to write that off in the same way. Mainly because: we don’t really know how much inference time compute humans are using! (I don’t think? Unless we understand the brain a lot better than I thought we did). I wouldn’t be surprised at all if we find that AGI requires spending a lot of inference-time compute. I don’t think that would make it any less AGI.
The exteme inference time compute costs are really important context to bear in mind when forecasting how AI progress is going to go, and what kinds of things are going to be possible. But I don’t think it provides a reason to describe the intelligence as not “general”, in the way that extreme data inefficiency does.

tobycrisford 🔸 24 Nov 2025 10:08 UTC
5 points
1 ∶ 0
in reply to: Yarrow Bouchard 🔸’s comment on: Unsolved research problems on the road to AGI
Because the ARC benchmark was specifically designed to be a test of general intelligence (do you disagree that it successfully achieves this?) and because each problem takes the form of requiring you to spot a pattern from only a couple of examples.

tobycrisford 🔸 24 Nov 2025 6:54 UTC
3 points
1 ∶ 0
on: Unsolved research problems on the road to AGI
LLMs have made no progress on any of these problems.
I think this probably overstates things? For example, o3 was able to achieve human level performance on ARC-AGI-1, which I think counts as at least some kind of progress on the problems of generalization and data efficiency?

tobycrisford 🔸 18 Nov 2025 7:34 UTC
16 points
0 ∶ 0
in reply to: Connacher Murphy 🔸’s comment on: A major flaw in the Forecasting Research Institute’s “Longitudinal Expert AI Panel” survey
This is a really great exchange, and thank you for responding to the post.
I just wanted to leave a quick comment to say: It seems crazy to me that someone would say the “slow” scenario has “already been achieved”!
Unless I’m missing something, the “slow” scenario says that half of all freelance software engineering jobs taking <8 hours can be fully automated, that any task a competent human assistant can do in <1 hour can be fully automated with no drop in quality (what if I ask my human assistant to solve some ARC-2 problems for me?), that the majority of customer complaints in a typical business will be fully resolved by AI in those businesses that use it, and that AI will be capable of writing hit songs (at least if humans aren’t made aware that it is AI-generated)?

I suppose the scenario is framed only to say that AI is capable of all of the above, rather than that it is being used like this in practice. That still seems like an incorrect summary of current capability to me, but is slightly more understandable. But in that case, it seems the scenario should have just been framed that way: “Slow progress: No significant improvement in AI capabilities from 2025, though possibly a significant increase in adoption”. There could then be a separate question on what people think about the level that current capabilities are at?
Otherwise disagreements about current capabilities and progress are getting blurred in the single question. Describing the “slow” scenario as “slow” and putting it at the extreme end of the spectrum is inevitably priming people to think about current capabilities in a certain way. Still struggling to understand the point of view that says this is an acceptable way to frame this question.
What links here?
- Yarrow Bouchard 🔸's comment on A major flaw in the Forecasting Research Institute’s “Longitudinal Expert AI Panel” survey by Yarrow Bouchard 🔸 (18 Nov 2025 19:33 UTC; 2 points)

tobycrisford 🔸 14 Nov 2025 17:08 UTC
12 points
4 ∶ 2
on: A major flaw in the Forecasting Research Institute’s “Longitudinal Expert AI Panel” survey
This post is getting some significant downvotes. I would be interested if someone who has downvoted could explain the reason for that.
There’s plenty of room for disagreement on how serious a mistake this is, whether it has introduced a ‘framing’ bias into other results or not, and what it means for the report as a whole. But it just seems straightforwardly true that this particular question is phrased extremely poorly (it seems disingenuous to suggest that the question using the phrasing “best matching” covers you for not even attempting to include the full range of possibilities in your list).
I assume that people downvoting are objecting to the way that this post is using this mistake to call the entire report into question, with language like “major flaw”. They may have a point there. But I think you should have a very high bar for downvoting someone who is politely highlighting a legitimate mistake in a piece of research.
‘Disagree’ react to the ‘major flaw’ language if you like, and certainly comment your disagreements, but silently downvoting someone for finding a legitimate methodological problem in some EA research seems like bad EA forum behaviour to me!
What links here?
- Yarrow Bouchard 🔸's comment on NickLaing’s Quick takes by NickLaing (18 Nov 2025 2:15 UTC; 3 points)

tobycrisford 🔸 12 Nov 2025 18:20 UTC
10 points
3 ∶ 1
on: Effective Altruists Should Promote Frugality More
I think I broadly agree with this.
I am very confused about your number 1 con though! Why would promoting frugality be perceived as the rich promoting their own interests over those of the poor? Isn’t it exactly the other way around?
To the extent that EA is comfortable with people spending large sums of their money on unnecessary things, I think it is open to the ‘elitism’ criticism (think of the discussion around SBF’s place in the bahamas). People can justifiably argue: “it is easy to say we should all be donating a lot to charity when you are so rich that you will still have enough left over to live in luxury!”.
But if EA advocates frugality for everyone, including the super rich, then this seems like a powerful response to the elitism criticism. I would have put this near the top of the pros list!

tobycrisford 🔸 5 Nov 2025 21:39 UTC
8 points
0 ∶ 1
on: The Last Stop on the Crazy Train
I don’t think longtermism is a nice solution to this problem. If you’re open to letting astronomically large but unlikely scenarios dominate your expected value calculations, then I don’t think this rounds out nicely to simply “reduce existential risk”. The more accurate summary would be: reduce existential risk according to a worldview in which astronomical value is possible, which is likely to lead to very different recommendations than if you were to attempt to reduce existential risk unconditionally.
https://forum.effectivealtruism.org/posts/RCmgGp2nmoWFcRwdn/should-strong-longtermists-really-want-to-minimize

tobycrisford 🔸 5 Nov 2025 7:12 UTC
1 point
0 ∶ 0
in reply to: Davidmanheim’s comment on: Will Welfareans Get to Experience the Future?
It sounds like MichaelDickens’ reply is probably right, that we don’t need to consider identical experiences in order for this argument to go through.
But the question of whether identical copies of the same experience have any additional value is a really interesting one. I used to feel very confident that they have no value at all. I’m now a lot more uncertain, after realising that this view seems to be in tension with the many worlds interpretation of quantum mechanics: https://www.lesswrong.com/posts/bzSfwMmuexfyrGR6o/the-ethics-of-copying-conscious-states-and-the-many-worlds

tobycrisford 🔸 29 Oct 2025 18:47 UTC
1 point
0 ∶ 0
in reply to: Joey Marcellino’s comment on: Some thoughts on fanaticism
It seems very strange to me to treat reducing someone’s else chance of X differently to reducing your own (if you’re confident it would affect each of you similarly)! But thank you for engaging with these questions, it’s helping me understand your position better I think.
By ‘collapsing back to expected utility theory’ I only meant that if you consider a large enough reference class of similar decisions, it seems like it will in practice be the same as acting as if you had an extremely low discount threshold? But it sounds like I may just not have understood the original approach well enough.

tobycrisford 🔸 28 Oct 2025 18:33 UTC
3 points
0 ∶ 0
on: Is “cage free” really the most humane option for egg-laying hens?
Thanks for writing this up! I’d be interested to read a response to these points.

tobycrisford 🔸 28 Oct 2025 17:34 UTC
2 points
0 ∶ 0
in reply to: Joey Marcellino’s comment on: Some thoughts on fanaticism
To answer your question, personally, I think we should probably stick with standard expected value reasoning over the approach you are advocating here. So no, I wouldn’t tell you to wear one anyway. But I’m confused about exactly what it is you are advocating.
I’ll try to make the objection I am trying to articulate more forceful:
Suppose we are considering some awful painful experience, X, and some trivial inconvenience, Y. Suppose everyone on earth agrees that, when thinking altruistically about others, 8000 people having experience X would be worse than 8 billion people having experience Y (that’s how bad experience X is, and how trivial experience Y is).
Suppose also that everyone on earth adopts a discount threshold of just over 1 in a million.
Now suppose that everyone on earth is faced with the choice of experiencing Y or facing a 1 in a million chance of X. Since they have a discount threshold, they all choose to go with the 1 in a million chance of X.
Now, with extremely high probability, ~8,000 people on earth will experience X. Take the perspective of any one individual looking at what’s happened to everyone else. They will agree that the situation for everyone else is bad. They should, at least when thinking altruistically, wish that everyone else had chosen to experience Y instead (everyone agrees that it is worse for 8000 people to experience X than 8 billion to experience Y). But they can’t actually recommend that any particular person should have decided any differently, because they have done exactly the same thing themselves!
Something seems wrong here!

tobycrisford 🔸 28 Oct 2025 7:54 UTC
1 point
0 ∶ 0
in reply to: Joey Marcellino’s comment on: Some thoughts on fanaticism
Thanks for the reply!
On the grouping issue, suppose we take your suggestion of grouping all “decisions that might kill you” into one, and suppose that everyone on earth follows this policy. Suppose also that there is some precaution against painful death (like seatbelt wearing) that everyone decides not to do, in order to gain some trivial benefit. Suppose that integrating over their own life, this decision makes sense, because the risk is still below their discount threshold. Whereas on expected value terms it does not.
It might then be the case that if everyone follows this policy independently, that globally, over billions of people, we might still expect thousands of people to end up dying avoidable painful deaths. Which seems bad!
This seems like a strong case for integrating not just over “decisions that might kill you”, but over “decisions that anyone takes that might kill them”.. and I think through similar appeals you could imagine extending that to “decisions anyone takes that might cause any benefit/harm to any sentient being”, at which point, in a big universe, have you not just arrived back at expected utility theory again..?

tobycrisford 🔸 27 Oct 2025 8:04 UTC
2 points
0 ∶ 0
on: Some thoughts on fanaticism
This is an interesting approach!
I’m still a bit confused about exactly how to apply this method in practice though. If I am understanding it correctly, then if someone knows they will only drive 1 mile in their entire life, you would say that wearing a seatbelt becomes the wrong thing to do for them? On the other hand, if they know that they will drive 500 miles, then wearing a seatbelt for those 500 miles might make sense?
But what if they are in a situation where they do not know how long their journeys are going to be? They are taking 1 car journey in their life, and maybe the car will stop after 1 mile, or maybe after 1000. Maybe they have some subjective probability distribution over these possible journey lengths. How do they make their decision in this situation? I’d be interested to see a worked example here!
I’m also still confused about how you decide on the groupings in practice. If I know that I will travel 250 miles by car in my life, and 250 miles by bike, and each risk is below the discount threshold, does that mean I should wear neither a seatbelt nor a bike helmet? Or should I wear both, if the combined risk of both driving and cycling is together enough to cross the threshold? Should I treat seatbelts/bike-helmets as one decision or two separate ones?
If I’m treating it as two separate decisions, then this feels arbitrary (why not split the seatbelt decision into driving on main roads vs driving on side roads, to push things under the threshold when it previously would have been over?), but if I treat them together, then it feels like my actual discount threshold in practical situations is going to become far smaller than the one I decide on a priori (since I make a lot of decisions in my life!)

tobycrisford 🔸 26 Oct 2025 19:35 UTC
3 points
0 ∶ 0
in reply to: tobycrisford 🔸’s comment on: Rule High Stakes In, Not Out
Having thought this through some more, I’ve realised I’m wrong, sorry!

Person A shouldn’t say that the probability of extinction halves each century, but they can say that it will decay as 1/N, and that will still lead to an enormous future without them ever seeming implausibly overconfident.
A 1/N decay in extinction risk per century (conditional on making it that far) implies a O(1/N) chance of surviving >= N centuries, which implies a O(1/N^2) chance of going extinct in the Nth century (unconditional). Assuming that the value of the future with extinction in the Nth century is at least proportional to N (a modest assumption) then the value of the future is the sum of terms that decay no faster than 1/N, so this sum diverges, and we get a future with infinite expected value.
I think your original argument is right.
I still have separate reservations about allowing small chances of high stakes to infect our decision making like this, but I completely retract my original comment!

tobycrisford 🔸 26 Oct 2025 18:50 UTC
1 point
0 ∶ 0
in reply to: Richard Y Chappell🔸’s comment on: Rule High Stakes In, Not Out
I was assuming in my example that the “Time of perils” that Person A believes we might be living through is supposed to be over by the 50th century, so that the 50th century is already in the period where extinction risk is supposed to have become very low.
But suppose Person A adopts your alternative probabilities instead. Person A now believes in a 1/1000 chance of going extinct in the 50th century, conditional on reaching it, and then the probability halves in each century after that.
But if that’s what they believe, you can now just run my argument on the 100th century instead. Person A now proposes a probability of ~10^(-18) of going extinct in the 100th century (conditional on reaching it) which seems implausibly overconfident to me on the face of it!
I agree with you, that if we were considering the 50 millionth century, then a probability of 1/1000 would be far too high. I agree that it would be crazy to stipulate a probability for the Nth century that is much higher than 1/N, because surviving N centuries is evidence that typical extinction risk per century is lower than this (except maybe if we were considering centuries close to the time the sun is expected to die..?)
But my point is that in order to get a truly big future, with the kind of stakes that dominate our expected value calculations, then we need the probability of extinction to decay much faster than 1/N. We need the “Time of Perils” hypothesis. It needs to decay exponentially* (something like the halving that you’ve suggested). And before too long that exponential decay is going to lead to implausibly low probabilities of extinction.
*Edit: Actually not too confident on this claim now I think it through some more. Perhaps you can still get a very large future with sub-exponential decay. Maybe this is another way out for Person A in fact!

tobycrisford 🔸 24 Oct 2025 7:39 UTC
1 point
0 ∶ 0
on: Rule High Stakes In, Not Out
Edit: I no longer endorse this any more. The important point I was missing was that Person A’s probability of extinction per century only needs to decay as 1/N in order for the value of the future to remain enormous, and a 1/N decay is not implausibly overconfident.
You are saying that we do not need to assign high probability to the “time of perils” hypothesis in order to get high stakes. We only need to assign it non-vanishing probability. And assigning it vanishing probability would appear to be implausibly overconfident.
But I’m not sure this works, because I think it is impossible to avoid assigning vanishingly small probability to some outcome. If you just frame the question differently, the position that appears to be the overconfident one can be reversed.
Suppose you ask two people what credence they each have in the “time of perils” hypothesis. Person A replies with 10%, and Person B replies with 10^(-20). Person B sounds wildly overconfident.
But now ask each of them what the probability is that humanity (or humanity’s descendants/creations) will go extinct in the 50th century, conditional on surviving until that point. Person B may respond in many different ways. Maybe they say 1/1000. But Person A is now committed to giving a vanishingly small answer to this question, in order to be consistent with their 10% credence in “time of perils”. Now it is Person A who sounds overconfident!
Person A is committed to this, because Person A places a non-vanishing probability on the future being very large. But the probability of making it to the far future is just the product of the probabilities of making it through each century along the way (conditional on surviving the centuries prior). For there to be a non-vanishing probability of a large future, most of these probabilities must be extremely close to 1. Does that not also seem overconfident?
I don’t think this example tells us which of Person A or Person B are doing the right thing, but I think it shows that we can’t decide between them with an argument of the form: “this person’s view is implausible because they are assigning vanishingly small probability to something that seems, on the face of it, credible”.