JanB

Karma: 984

JanB Apr 30, 2022, 8:00 PM
9 points
0 ∶ 0
on: Big List of Cause Candidates: January 2021–March 2022 update
There has also been this post on cognitive enhancement research:

https://forum.effectivealtruism.org/posts/MojiqNw5MN6WMXETc/cause-profile-cognitive-enhancement-research-1

JanB Apr 9, 2022, 8:40 AM
17 points
0 ∶ 0
on: Why should we care about existential risk?
Here is my take on the value of extinction risk reduction, from some years ago: https://forum.effectivealtruism.org/posts/NfkEqssr7qDazTquW/the-expected-value-of-extinction-risk-reduction-is-positive
This posts also contains links to many other posts related to the topic.
Some other posts, that come to different conclusions:
https://forum.effectivealtruism.org/posts/BY8gXSpGijypbGitT/why-i-prioritize-moral-circle-expansion-over-artificial
https://forum.effectivealtruism.org/posts/RkPK8rWigSAybgGPe/a-longtermist-critique-of-the-expected-value-of-extinction-2
One final thing: Generally, I think people make a distinction between existential risk (roughly: permanent, irreversible, and drastic loss of value of the future) and extinction risk (extinction of humans), where extinction risk is just one type of existential risk.

JanB Mar 10, 2022, 8:44 PM
10 points
0 ∶ 0
on: Samotsvety Nuclear Risk Forecasts — March 2022
Thanks so much for writing this! I expect this will be quite useful for many people.

I actually spent some time this week worrying a bit about a nuclear attack on the UK, bought some preparation stuff, figured out where I would seek shelter or when I’d move to the countryside, and so on. One key thing is that it’s just so hard to know which probability to assign. Is it 1%? Then I should GTFO! Is it 0.001% Then I shouldn’t worry at all.

JanB Mar 2, 2022, 7:14 PM
14 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
Enlightenment at scale (provocative title :-) )
Values and Reflective Processes (?), X-risk (?)
A strong meditation practice promises enticing benefits to the meditator—less suffering, more control over ones attention and awareness, more insight, more equanimity. Brahmavihara practice promises the cultivation of loving-kindness, compassion, and empathetic joy. The world would be a much better place if everybody suffered less, had more equanimity, and felt strong compassion and empathy with other beings. But meditation is hard! Becoming a skilled meditator, and reaping these benefits, requires probably thousands of hours of dedicated practice. Most people will just not put in this amount of effort. But maybe it doesn’t need to be this way. The field of meditation teaching seems underdeveloped, and innovative methods that make use of technology (e.g. neurofeedback) seem largely unexplored. We are interested in supporting scalable solutions that bring the benefits of meditation to many people.
Note:
- I don’t actually know if meditation really has these benefits; this would needed to be established first (there should be quite some research on this by now). It seems plausible to me that meditation can be very beneficial. Several of my friends claim to have experienced significant benefits from meditation, and I think I can also point to tangible benefits in my own life.
- These innovation need not be directly related to meditation; for example, one could imagine development of an extremely safe and non-addictive pharmaceutical substance that would let people experience, say, strong compassion, and thus increase compassion in everyday life (see e.g. the use of MDMA in therapy).

JanB Mar 1, 2022, 10:39 AM
2 points
0 ∶ 0
in reply to: Sam Marks’s comment on: The Future Fund’s Project Ideas Competition
I think Berlin has something like this

JanB Mar 1, 2022, 10:32 AM
16 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
AI alignment prize suggestion: Improve our ability to evaluate (and provide training signal for) fuzzy tasks
Artificial Intelligence
There are many tasks that we want AI systems to do, for which performance cannot be evaluated automatically (and thus training signal provision is hard). If we don’t make progress on our ability to train systems for such tasks, we might end up in a world full of systems that optimise for that which is easy to measure, rather than what we actually want. One example of such a task is the evaluation of free-form text; there is currently no automated method to evaluate free-form text (with respect to criteria such as usefulness or correctness) that matches human evaluation. The Future Fund could offer prizes for work that takes a task for which the gold-standard of evaluation is humans, and demonstrates an automated evaluation method that matches human evaluation very closely (or work that demonstrates an automated evaluation method to be superior to human evaluation).
Note: This is crucially not the same as “training models to perform well on the task in question”. There are a number of technical reasons why what I suggest is easier. Intuitively, evaluating performance is often considerably easier than generating good performance. For example, I can watch a movie and say if it’s good, but I can’t make a good movie.

JanB Mar 1, 2022, 10:19 AM
10 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
AI alignment prize suggestion: Demonstrate a true sandwiching project
Artificial Intelligence
Sandwiching projects are a concrete way for how to make progress on aligning narrowly superhuman models. They “sandwich” the model in between one set of humans which is less capable than it and another set of humans which is more capable than it at the fuzzy task in question, and b) figure out how to help the less-capable set of humans reproduce the judgments of the more-capable set of humans. For example, first fine-tune a coding model to write short functions solving simple puzzles using demonstrations and feedback collected from expert software engineers. Then try to match this performance using some process that can be implemented by people who don’t know how to code and/or couldn’t solve the puzzles themselves.
Importantly, there are many ways to attack a sandwiching project that are slightly cheating. The most challenging version of a sandwiching project would need to make sure that no information whatsoever from the more-capable set of humans is used in the training process. The Future Fund could offer prizes for demonstrations of sandwiching projects on various levels of impressiveness and generality of the employed method.

JanB Mar 1, 2022, 9:53 AM
30 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
AI alignment prize suggestion: Introduce AI Safety concepts into the ML community
Artificial Intelligence
Recently, there have been several papers published at top ML conferences that introduced concepts from the AI safety community into the broader ML community. Such papers often define a problem, explain why it matters, sometimes formalise it, often include extensive experiments to showcase the problem, sometimes include some initial suggestions for remedies. Such papers are useful in several ways: they popularise AI alignment concepts, pave the way for further research, and demonstrate that researchers can do alignment research while also publishing in top venues. A great example would be Optimal Policies Tend To Seek Power, published in NeurIPS. Future Fund could advertise prizes for any paper that gets published in a top ML/NLP/Computer Vision conference (from ML, that would be NeurIPS, ICML, and ICLR) and introduces a key concept of AI alignment.

JanB Mar 1, 2022, 9:39 AM
10 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
Refinement of project idea #22, Prediction Markets
Add: “In particular, we’d like to see prediction platforms that do all of the following three: use real money, are very easy to use, allow very easy creation of markets.

JanB Mar 1, 2022, 9:34 AM
65 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
Highly effective enhancement of productivity, health, and wellbeing for people in high-impact roles
Effective Altruism
When it comes to enhancement of productivity, health, and wellbeing, the EA community does not sufficiently utilise division of labour. Currently, community members need to obtain the relevant knowledge themselves and do related research, e.g. on health issues, themselves. We would like to see dedicated experts on these issues that offer optimal productivity, health, and wellbeing, as a service. As a vision, a person working in a high-impact role could book calls with highly trained nutrition specialists, exercise specialists, sleep specialists, personal coaches, mental trainers, GPs with sufficient time, and so on, increasing their work output by 50% while costing little time. This could involve innovative methods such as ML-enabled optimal experiment design to figure out which interventions work for each individual.
Note: Inspired by conversations with various people. I won’t name them here because I don’t want to ask for permission first, but will share the prize money with them if I win something.

JanB Mar 1, 2022, 9:19 AM
58 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
Reducing gain-of-function research on potentially pandemic pathogens
Biorisk
Lab outbreaks and other lab accidents with infectious pathogens happen regularly. When such accidents happen in labs that work on gain-of-function research (on potentially pandemic pathogens), the outcome could be catastrophic. At the same time, the usefulness of gain-of-function research seems limited; for example, none of the major technological innovations that helped us fight COVID-19 (vaccines, testing, better treatment, infectious disease modelling) was enabled by gain-of-function research. We’d like to see projects that reduce the amount of gain-of-function research done in the world, for example by targeting coordination between journals or funding bodies, or developing safer alternatives to gain-of-function research.
Additional notes:
- There are many stakeholders In the research system (funders, journals, scientists, hosting institutions, hosting countries). I think the concentration of power is strongest in journals: there are only a few really high profile life-science journals(*). Currently, they do publish gain-of-function research. Getting high-profile journals to coordinate against publishing such research would strongly reduce incentives for academic researchers. There is plenty of precedent for collaboration between journals, such as
  - essentially all journals agreed to publish COVID research in Open Access (2016 Statement on Data Sharing in Public Health Emergencies)
  - essentially all journals agree to not publish research with human participants if there was no ethics committee approval for this research
  - essentially all journals agree on requiring authors to state their conflicts of interest
(*) Super roughly:
Tier 1: Science, Nature
Tier 2: Nature Medicine, Cell, (maybe some other Nature family journals)
Tier 3: Some more Nature family journals, Science Advances, PNAS, several others

-> If three publishers (Science, Nature Gropu, Cell Press) coordinated, this would cover all tier 1 and tier 2 journals, and a large part of tier 3 journals.

JanB Mar 1, 2022, 9:16 AM
51 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
Cognitive enhancement research and development (nootropics, devices, …)
Values and Reflective Processes, Economic Growth
Improving people’s ability to think has many positive effects on innovation, reflection, and potentially individual happiness. We’d like to see more rigorous research on nootropics, devices that improve cognitive performance, and similar fields. This could target any aspect of thinking ability—such as long/short term memory, abstract reasoning, creativity—and any stage of the research and development pipeline—from wet lab research or engineering over testing in humans to product development.
Additional notes on cognitive enhancement research:
- Importance:
  - Sign of impact: You already seem to think that AI-based cognitive aids would be good from a longtermist perspective, so you will probably think that non-AI-based cognitive enhancement is also at least positive. (I personally think that’s somewhat likely but not obvious and would love to see more analysis on it).
  - Size of impact: AI-based cognitive enhancement is probably more promising right now. But non-AI-based cognitive enhancement is still pretty promising, there is some precedent (e.g. massive benefits from iodine supplementation), …
- Neglectedness: The research community on many areas of cognitive enhancement research is absolutely tiny. Even very low-hanging fruit doesn’t get picked (e.g. before my team started to work on it, nobody had tried to replicate this 20-year old paper, which found massive IQ increases).
- What funding could achieve: From talking to some people, this seems to be an issue of funding (medical funding bodies don’t usually cover enhancement).There are legions of life scientists/psychologists/clinicians who have the relevant skills for this type of research, so if funding was available, I’d expect a lot more of this reserach would happen

JanB Feb 27, 2022, 4:13 PM
3 points
0 ∶ 0
on: Nuclear attack risk? Implications for personal decision-making
This page, from Rob Wiblin, has been shared on Twitter recently. Contains some advice some minimal version of preparation (e.g. buy potassium iodide tablets): https://nuclearadvice.org/

JanB Feb 26, 2022, 1:10 PM
8 points
0 ∶ 0
in reply to: technicalities’s comment on: Some reflections on testing fit for research
I have a similar knee-jerk reaction whenever I read a post “on research”, so I wrote up my experience with different types of research: https://forum.effectivealtruism.org/posts/pHnMXaKEstJGcKP2m/different-types-of-research-are-different

(I’m not at all trying to imply that Rose should have caveated more in her post.)

Different types of research are different

JanBFeb 26, 2022, 1:06 PM

71 points

1 comment1 min readEA link

JanB 13 Feb 2022 10:50 UTC
3 points
0 ∶ 0
on: Is a career in making AI systems more secure a meaningful way to mitigate the X-risk posed by AGI?
First, check out this post: https://www.lesswrong.com/posts/YDF7XhMThhNfHfim9/ai-safety-needs-great-engineers
Are you talking about adversarial ML/adversarial examples? If so, that is certainly an area that’s relevant to long-term AI safety; e.g. many proposals for aligning AGI include some adversarial training. In general, I’d say many areas of ML have some relevance to safety, and it mostly depends on how you pick your research project within an area.

[Link] Antimicrobial Resistance—Massive new survey paper

JanB31 Jan 2022 21:35 UTC

41 points

2 comments1 min readEA link

JanB 26 Jan 2022 21:07 UTC
6 points
0 ∶ 0
on: Examples of pure altruism towards future generations?
Maybe this? Probably not, though https://www.lesswrong.com/posts/puYfAEJJomeodeSsi/an-observation-of-vavilov-day

JanB 14 Nov 2021 13:33 UTC
6 points
0 ∶ 0
in reply to: Lukas Finnveden’s comment on: Preprint is out! 100,000 lumens to treat seasonal affective disorder
yes, when we did the calculation, it was something like €2 per day (for ~6-8 hours per day). Still very cheap for a depression treatment :-)

JanB 10 Oct 2021 11:32 UTC
5 points
0 ∶ 0
on: Introducing Training for Good (TFG)
Great to see this initiative, it seems like there is probably valuable work to be done in this area. I would make extra sure not to conflate “EA jobs” with “jobs at EA orgs” (not implying that you do conflate them). The latter just don’t have that much capacity in the medium-term.

In a way, it’s easier to offer specific training for skills that are needed by EA orgs, and maybe this is more tractable. But I’d also be very excited about programmes that equip many people with the resources they need to pursue high-impact careers outside of the few main EA orgs (whatever these resources are: skills? personality traits? money? cultural shift in the EA community?).

JanB

Differ­ent types of re­search are different

[Link] An­timicro­bial Re­sis­tance—Mas­sive new sur­vey pa­per

Different types of research are different

[Link] Antimicrobial Resistance—Massive new survey paper