Hi, all. Talk is cheap, and EA Forum karma may be insufficiently nuanced to convey substantive disagreements.
I’ve taken the liberty to sketch out several forecasting questions that might reflect underlying differences in opinion. Interested parties may wish to forecast on them (which the EA Forum should allow you to do directly, at least on desktop) and then make bets accordingly.
Feel free to also counterpropose (and make!) other questions if you think the existing question operationalizations are not sufficient (I’m far from knowledgeable in this field!).
By EOY 2026, will a single neuroscience paper by Mike Johnson or Andrés Emilsson have more than 100 citations?By EOY 2031, will there be more than 3000 citations by academic researchers of articles by QRI or direct affiliates on the Symmetry Theory of Valence?Will QRI exist and have a >250k/year budget in 2031? In 2031, will Luke Muelhauser say statements to the effect that the Symmetry Theory of Valence is substantively true?
Methods papers tend to be among the most highly cited, and e.g. Selen Atasoy’s original work on CSHW has been cited 208 times, according to Google Scholar. Some more recent papers are at significantly less than 100, though this may climb over time.
Anyway my sense is (1) is possible but depends on future direction, (2) is unlikely, (3) is likely, (4) is unlikely (high confidence).
Perhaps a better measure of success could be expert buy-in. I.e., does QRI get endorsements from distinguished scientists who themselves fit criteria (1) and/or (2)? Likewise, technological usefulness, e.g. has STV directly inspired the creation of some technical device that is available to buy or is used in academic research labs? I’m much more optimistic about these criteria than citation counts, and by some measures we’re already there.
Note that the 2nd question is about total citations rather than of one paper, and 3k citations doesn’t seem that high if you’re introducing an entirely new subfield (which is roughly what I’d expect if STV is true). The core paper of Friston’s free energy principle has almost 5,000 citations for example, and it seems from the outside that STV (if true) ought to be roughly as big a deal as free energy.
For a sense of my prior beliefs about EA-encouraged academic subfields, I think 3k citations in 10 years is an unlikely but not insanely high target for wild animal welfare (maybe 20-30%?), and AI risk is likely already well beyond that (eg >1k citations for Concrete Problems alone).
I’d say that’s a fair assessment — one wrinkle that isn’t a critique of what you wrote, but seems worth mentioning, is that it’s an open question if these are the metrics we should be optimizing for. If we were part of academia, citations would be the de facto target, but we have different incentives (we’re not trying to impress tenure committees). That said, the more citations the better of course.
As you say, if STV is true, it would essentially introduce an entirely new subfield. It would also have implications for items like AI safety and those may outweigh its academic impact. The question we’re looking at is how to navigate questions of support, utility, and impact here: do we put our (unfortunately rather small) resources toward academic writing and will that get us to the next step of support, or do we put more visceral real-world impact first (can we substantially improve peoples’ lives? How much and how many?), or do we go all out towards AI safety?
It’s of course possible to be wrong; I’m also understanding it’s possible to be right, but take the wrong strategic path and run out of gas. Basically I’m a little worried that racking up academic metrics like citations is less a panacea than it might appear, and we’re looking to hedge our bets here.
For what it’s worth, we’ve been interfacing with various groups working on emotional wellness neurotech and one internal metric I’m tracking is how useful a framework STV is to these groups; here’s Jay Sanguinetti explaining STV to Shinzen Young (first part of the interview):
I think of the metrics I mentioned above as proxies rather than as the underlying targets, which is some combination of:
a) Is STV true? b) Conditional upon STV being true, is it useful?
What my forecasting questions aimed to do is shedding light on a). I agree that academia and citations isn’t the best proxy. They may in some cases have conservatism bias (I think trusting the apparent academic consensus on AI risk in 2014 would’ve been a mistake for early EAs), but are also not immune to falseties/crankery (cf replication crisis). In addition, standards for truth and usefulness are different within EA circles than academia, partially because we are trying to answer different questions.
This is especially an issue as the areas that QRI is likely to interact with (consciousness, psychedelics) seem from the outside to be more prone than average to falseness and motivated cognition, including within academia.
This is what I was trying to get at with “will Luke Muelhauser say statements to the effect that the Symmetry Theory of Valence is substantively true?” because Luke is a non-QRI affiliated person within EA who’s a) respected and b) have thought about concepts adjacent to QRI’s work. Bearing in mind that Luke is very far from a perfect oracle, I would still trust Luke’s judgement on this more than an arbitrarily selected academic in an adjacent field.
I think the actual question I’m interested in is something like “In X year, will a panel of well-respected EAs a) not affiliated with QRI and b) have very different thoughts from each other and c)who have thought about things adjacent to QRI’s work have updated to believing STV to be substantively true” but I was unable to come up with a clean question operationalization in the relatively brief amount of time I gave myself to come up with this.
People are free to counterpropose and make their own questions.
Hi Linch, that’s very well put. I would also add a third possibility (c), which is “is STV false but generative.” — I explore this a little here, with the core thesis summarized in this graphic:
I.e., STV could be false in a metaphysical sense, but insofar as the brain is a harmonic computer (a strong reframe of CSHW), it could be performing harmonic gradient descent. Fully expanded, there would be four cases:
STV true, STHR true
STV true, STHR false
STV false, STHR true
STV false, STHR false
Of course, ‘true and false’ are easier to navigate if we can speak of absolutes; STHR is a model, and ‘all models are wrong; some are useful.’
Hi, all. Talk is cheap, and EA Forum karma may be insufficiently nuanced to convey substantive disagreements.
I’ve taken the liberty to sketch out several forecasting questions that might reflect underlying differences in opinion. Interested parties may wish to forecast on them (which the EA Forum should allow you to do directly, at least on desktop) and then make bets accordingly.
Feel free to also counterpropose (and make!) other questions if you think the existing question operationalizations are not sufficient (I’m far from knowledgeable in this field!).
Hi Linch, cool idea.
I’d suggest that 100 citations can be a rather large number for papers, depending on what reference class you put us in, 3000 larger still; here’s an overview of the top-cited papers in neuroscience for what it’s worth: https://www.frontiersin.org/articles/10.3389/fnhum.2017.00363/full
Methods papers tend to be among the most highly cited, and e.g. Selen Atasoy’s original work on CSHW has been cited 208 times, according to Google Scholar. Some more recent papers are at significantly less than 100, though this may climb over time.
Anyway my sense is (1) is possible but depends on future direction, (2) is unlikely, (3) is likely, (4) is unlikely (high confidence).
Perhaps a better measure of success could be expert buy-in. I.e., does QRI get endorsements from distinguished scientists who themselves fit criteria (1) and/or (2)? Likewise, technological usefulness, e.g. has STV directly inspired the creation of some technical device that is available to buy or is used in academic research labs? I’m much more optimistic about these criteria than citation counts, and by some measures we’re already there.
Note that the 2nd question is about total citations rather than of one paper, and 3k citations doesn’t seem that high if you’re introducing an entirely new subfield (which is roughly what I’d expect if STV is true). The core paper of Friston’s free energy principle has almost 5,000 citations for example, and it seems from the outside that STV (if true) ought to be roughly as big a deal as free energy.
For a sense of my prior beliefs about EA-encouraged academic subfields, I think 3k citations in 10 years is an unlikely but not insanely high target for wild animal welfare (maybe 20-30%?), and AI risk is likely already well beyond that (eg >1k citations for Concrete Problems alone).
I’d say that’s a fair assessment — one wrinkle that isn’t a critique of what you wrote, but seems worth mentioning, is that it’s an open question if these are the metrics we should be optimizing for. If we were part of academia, citations would be the de facto target, but we have different incentives (we’re not trying to impress tenure committees). That said, the more citations the better of course.
As you say, if STV is true, it would essentially introduce an entirely new subfield. It would also have implications for items like AI safety and those may outweigh its academic impact. The question we’re looking at is how to navigate questions of support, utility, and impact here: do we put our (unfortunately rather small) resources toward academic writing and will that get us to the next step of support, or do we put more visceral real-world impact first (can we substantially improve peoples’ lives? How much and how many?), or do we go all out towards AI safety?
It’s of course possible to be wrong; I’m also understanding it’s possible to be right, but take the wrong strategic path and run out of gas. Basically I’m a little worried that racking up academic metrics like citations is less a panacea than it might appear, and we’re looking to hedge our bets here.
For what it’s worth, we’ve been interfacing with various groups working on emotional wellness neurotech and one internal metric I’m tracking is how useful a framework STV is to these groups; here’s Jay Sanguinetti explaining STV to Shinzen Young (first part of the interview):
https://open.spotify.com/episode/6cI9pZHzT9sV1tVwoxncWP?si=S1RgPs_CTYuYQ4D-adzNnA&dl_branch=1
I think of the metrics I mentioned above as proxies rather than as the underlying targets, which is some combination of:
a) Is STV true?
b) Conditional upon STV being true, is it useful?
What my forecasting questions aimed to do is shedding light on a). I agree that academia and citations isn’t the best proxy. They may in some cases have conservatism bias (I think trusting the apparent academic consensus on AI risk in 2014 would’ve been a mistake for early EAs), but are also not immune to falseties/crankery (cf replication crisis). In addition, standards for truth and usefulness are different within EA circles than academia, partially because we are trying to answer different questions.
This is especially an issue as the areas that QRI is likely to interact with (consciousness, psychedelics) seem from the outside to be more prone than average to falseness and motivated cognition, including within academia.
This is what I was trying to get at with “will Luke Muelhauser say statements to the effect that the Symmetry Theory of Valence is substantively true?” because Luke is a non-QRI affiliated person within EA who’s a) respected and b) have thought about concepts adjacent to QRI’s work. Bearing in mind that Luke is very far from a perfect oracle, I would still trust Luke’s judgement on this more than an arbitrarily selected academic in an adjacent field.
I think the actual question I’m interested in is something like “In X year, will a panel of well-respected EAs a) not affiliated with QRI and b) have very different thoughts from each other and c)who have thought about things adjacent to QRI’s work have updated to believing STV to be substantively true” but I was unable to come up with a clean question operationalization in the relatively brief amount of time I gave myself to come up with this.
People are free to counterpropose and make their own questions.
Hi Linch, that’s very well put. I would also add a third possibility (c), which is “is STV false but generative.” — I explore this a little here, with the core thesis summarized in this graphic:
I.e., STV could be false in a metaphysical sense, but insofar as the brain is a harmonic computer (a strong reframe of CSHW), it could be performing harmonic gradient descent. Fully expanded, there would be four cases:
STV true, STHR true
STV true, STHR false
STV false, STHR true
STV false, STHR false
Of course, ‘true and false’ are easier to navigate if we can speak of absolutes; STHR is a model, and ‘all models are wrong; some are useful.’