On the meta-level, I want to think hard about the level of rigor I want to have in research or research-adjacent projects.
I want to say that the target level of rigor I should have is substantially higher than for typical FB or Twitter posts, and way lower than research papers.
But there’s a very wide gulf! I’m not sure exactly what I want to do, but here are some gestures at the thing:
- More rigor/thought/data collection should be put into it than 5-10 minutes typical of a FB/Twitter post, but much less than a hundred or a few hundred hours on papers. - I feel like there are a lot of things that are worth somebody looking into it for a few hours (more rarely, a few dozen), but nowhere near the level of a typical academic paper? - Examples that I think are reflective of what I’m interested in are some of Georgia Ray and Nuno Sempere’s lighter posts, as well as Brian Bi’s older Quora answers on physics. (back when Quora was good) - “research” has the connotation of pushing the boundaries of human knowledge, but by default I’m more interested in pushing the boundaries of my own knowledge? Or at least the boundaries of my “group’s” knowledge. - If the search for truthful things shakes out to have some minor implications on something no other human currently knows, that’s great, but by default I feel like aiming for novelty is too constraining for my purposes. - Forecasting (the type done on Metaculus or Good Judgment Open) feels like a fair bit of this. Rarely do forecasters (even/especially really good ones) discover something nobody already knows; rather than the difficulty comes almost entirely in finding evidence that’s already “out there” somewhere in the world and then weighing the evidence and probabilities accordingly. - I do think more forecasting should be done. But forecasting itself provides very few bits of information (just the final probability distribution on a well-specified problem). Often, people are also interested in your implicit model, the most salient bits of evidence you discovered, etc. This seems like a good thing to communicate. - It’s not clear what the path to impact here is. Probably what I’m interested in is what Stefan Schubert calls “improving elite epistemics,” but I’m really confused on whether/why that’s valuable. - Not everything I or anybody does has to be valuable, but I think I’d be less excited to do medium rigor stuff if there’s no or minimal impact on the world? - It’s also not clear to me how much I should trust my own judgement (especially in out-of-distribution questions, or questions much harder to numerically specify than forecasting). - How do I get feedback? The obvious answer is from other EAs, but I take seriously worries that our community is overly insular. - Academia, in contrast, has a very natural expert feedback mechanism in peer review. But as mentioned earlier, peer review pre-supposes a very initially high level of rigor that I’m usually not excited about achieving for almost all of my ideas. - Also on a more practical level, it might just be very hard for me to achieve paper-worthy novelty and rigor in all but a handful of ideas? - In the few times in the past I reached out to experts (outside the EA community) for feedback, I managed to get fairly useful stuff, but I strongly suspect this is easier for precise well-targeted questions than some of the other things I’m interested in? - Also varies from field to field, for example a while back I was able to get some feedback on questions like water rights, but I couldn’t find public contact information from climate modeling scientist after a modest search (presumably because the latter is much more politicized these days) - If not for pre-existing connections and connections-of-connections, I also suspect it’d be basically impossible to get ahold of infectious disease or biosecurity people to talk to in 2020. - In terms of format, “blog posts” seems the most natural. But I think blog posts could mean anything from “Twitter post with slightly more characters” to “stuff Gwern writes 10,000 words on.” So doesn’t really answer the question of what to do about the time/rigor tradeoff.
Another question that is downstream of what I want to do is branding. Eg, some people have said that I should call myself an “independent researcher,” but this feels kinda pretentious to me? Like when I think “independent research” I think “work of a level of rigor and detail that could be publishable if the authors wanted to conform to publication standards,” but mostly what I’m interested in is lower quality than that? Examples of what I think of as “independent research” are stuff that Elizabeth van Nostrand, Dan Luu, Gwern, and Alexey Guzey sometimes work on (examples below).
(To be clear, by “negative examples” I don’t mean to associate them with negative valence. I think a lot of those work is extremely valuable to have, it’s just that I don’t think most of the things I want to do are sufficiently interesting/important to spend as much time on. Also on a practical level, I’m not yet strong enough to replicate most work on that level).
Cross-posted from Facebook
On the meta-level, I want to think hard about the level of rigor I want to have in research or research-adjacent projects.
I want to say that the target level of rigor I should have is substantially higher than for typical FB or Twitter posts, and way lower than research papers.
But there’s a very wide gulf! I’m not sure exactly what I want to do, but here are some gestures at the thing:
- More rigor/thought/data collection should be put into it than 5-10 minutes typical of a FB/Twitter post, but much less than a hundred or a few hundred hours on papers.
- I feel like there are a lot of things that are worth somebody looking into it for a few hours (more rarely, a few dozen), but nowhere near the level of a typical academic paper?
- Examples that I think are reflective of what I’m interested in are some of Georgia Ray and Nuno Sempere’s lighter posts, as well as Brian Bi’s older Quora answers on physics. (back when Quora was good)
- “research” has the connotation of pushing the boundaries of human knowledge, but by default I’m more interested in pushing the boundaries of my own knowledge? Or at least the boundaries of my “group’s” knowledge.
- If the search for truthful things shakes out to have some minor implications on something no other human currently knows, that’s great, but by default I feel like aiming for novelty is too constraining for my purposes.
- Forecasting (the type done on Metaculus or Good Judgment Open) feels like a fair bit of this. Rarely do forecasters (even/especially really good ones) discover something nobody already knows; rather than the difficulty comes almost entirely in finding evidence that’s already “out there” somewhere in the world and then weighing the evidence and probabilities accordingly.
- I do think more forecasting should be done. But forecasting itself provides very few bits of information (just the final probability distribution on a well-specified problem). Often, people are also interested in your implicit model, the most salient bits of evidence you discovered, etc. This seems like a good thing to communicate.
- It’s not clear what the path to impact here is. Probably what I’m interested in is what Stefan Schubert calls “improving elite epistemics,” but I’m really confused on whether/why that’s valuable.
- Not everything I or anybody does has to be valuable, but I think I’d be less excited to do medium rigor stuff if there’s no or minimal impact on the world?
- It’s also not clear to me how much I should trust my own judgement (especially in out-of-distribution questions, or questions much harder to numerically specify than forecasting).
- How do I get feedback? The obvious answer is from other EAs, but I take seriously worries that our community is overly insular.
- Academia, in contrast, has a very natural expert feedback mechanism in peer review. But as mentioned earlier, peer review pre-supposes a very initially high level of rigor that I’m usually not excited about achieving for almost all of my ideas.
- Also on a more practical level, it might just be very hard for me to achieve paper-worthy novelty and rigor in all but a handful of ideas?
- In the few times in the past I reached out to experts (outside the EA community) for feedback, I managed to get fairly useful stuff, but I strongly suspect this is easier for precise well-targeted questions than some of the other things I’m interested in?
- Also varies from field to field, for example a while back I was able to get some feedback on questions like water rights, but I couldn’t find public contact information from climate modeling scientist after a modest search (presumably because the latter is much more politicized these days)
- If not for pre-existing connections and connections-of-connections, I also suspect it’d be basically impossible to get ahold of infectious disease or biosecurity people to talk to in 2020.
- In terms of format, “blog posts” seems the most natural. But I think blog posts could mean anything from “Twitter post with slightly more characters” to “stuff Gwern writes 10,000 words on.” So doesn’t really answer the question of what to do about the time/rigor tradeoff.
Another question that is downstream of what I want to do is branding. Eg, some people have said that I should call myself an “independent researcher,” but this feels kinda pretentious to me? Like when I think “independent research” I think “work of a level of rigor and detail that could be publishable if the authors wanted to conform to publication standards,” but mostly what I’m interested in is lower quality than that? Examples of what I think of as “independent research” are stuff that Elizabeth van Nostrand, Dan Luu, Gwern, and Alexey Guzey sometimes work on (examples below).
____
Stefan Schubert on elite epistemics: https://twitter.com/StefanFSchubert/status/1248930229755707394
Negative examples (too little rigor):
- Pretty much all my FB posts?
Negative examples (too much rigor/effort):
- almost all academic papers
- many of Gwern’s posts
- eg https://www.gwern.net/Melatonin
- https://davidroodman.com/david/The%20risk%20of%20geomagnetic%20storms%205%20dr.pdf
- https://danluu.com/input-lag/
- https://guzey.com/books/why-we-sleep/
(To be clear, by “negative examples” I don’t mean to associate them with negative valence. I think a lot of those work is extremely valuable to have, it’s just that I don’t think most of the things I want to do are sufficiently interesting/important to spend as much time on. Also on a practical level, I’m not yet strong enough to replicate most work on that level).
Positive examples:
https://brianbi.ca/physics
https://nunosempere.github.io/ea/PastPandemics
https://eukaryotewritesblog.com/2019/05/19/naked-mole-rats-a-case-study-in-biological-weirdness/