From the bullet list above, it sounds like the author will be the one responsible for publishing and publicising the work.
SoerenMind
Karma: 587
Learned pain as a leading cause of chronic pain
Those definitely help, thanks! Any additional answers are still useful and I don’t want to discourage answers from people who haven’t read the above. For example we may have learned some empirical things since these analyses came out.
I don’t mean to imply that we’ll build a sovereign AI (I doubt it too).
Corrigible is more what I meant. Corrigible but not necessarily limited. Ie minimally intent aligned AIs which won’t kill you but by the strategy stealing assumption can still compete with unaligned AIs.
Re 1) this relates to the strategy stealing assumption: your aligned AI can use whatever strategy unaligned AIs use to maintain and grow their power. Killing the competition is one strategy but there are many others including defensive actions and earning money / resources.
Edit: I implicitly said that it’s okay to have unaligned AIs as long as you have enough aligned ones around. For example we may not need aligned companies if we have (minimally) aligned government+law enforcement.
I agree that it’s not trivial to assume everyone will use aligned AI.
Let’s suppose the goal of alignment research is to make aligned AI equally easy/cheap to build as unaligned AI. I. e. no addition cost. If we then suppose aligned AI also has a nonzero benefit, people are incentivized to use it.
The above seems to be the perspective in this alignment research overview https://www.effectivealtruism.org/articles/paul-christiano-current-work-in-ai-alignment.
More ink could be spilled on whether aligning AI has a nonzero commercial benefit. I feel that efforts like prompting and Instruct GPT are suggestive. But this may not apply to all alignment efforts.
[Question] How much should you optimize for the short-timelines scenario?
Another framing on this: As an academic, if I magically worked more productive hours this month, I could just do the high-priority research I otherwise would’ve done next week/month/year, so I wouldn’t do lower-priority work.
Thanks Aidan, I’ll consider this model when doing any more thinking on this.
It seems to depend on your job. E.g. in academia there’s a practically endless stream of high priority research to do since each field is way too big for one person solve. Doing more work generates more ideas, which generate more work.
CS professor Cal Newport says that if you can do DeepWork TM for 4h / day, you’re hitting the mental speed limit
and:
the next hour worked at the 10h/week mark might have 10x as much impact as the hour worked after the 100h/week mark
Thanks Hauke that’s helpful. Yes, the above would be mainly because you run out of steam at 100h/week. I want to clarify that I assume this effect doesn’t exist. I’m not talking about working 20% less and then relaxing. The 20% of time lost would also go into work, but that work has no benefit for career capital or impact.
Thanks Lukas that’s helpful. Some thoughts on when you’d expect diminishing returns to work: Probably this happens when when you’re in a job at a small-sized org or department where you have a limited amount to do. On the other hand, a sign that there’s lots to do would be if your job requires more than one person (with roughly the same skills as you).
In this case here the career is academia or startup founder.
[Question] If I’m 20% less productive, do I have 20% less expected impact?
Acquire and repurpose new AI startups for AI safety
Artificial intelligence
As ML performance has recently improved there is a new wave of startups coming. Some are composed of top talent, carefully engineered infrastructure, a promising product, well-coordinated teams, with existing workflows and management capacity. All of these are bottlenecks for AI safety R&D.
It should be possible to acquire some appropriate startups and middle-sized companies. Examples include HuggingFace, AI21, Cohere, and smaller, newer startups. The idea is to repurpose the mission of some select companies to align them more closely with socially beneficial and safety-oriented R&D. This is sometimes feasible since their missions are often broad, still in flux, and their product could benefit from improving safety and alignment.
Trying this could have very high information value. If it works, it has enormous potential upside as many new AI startups are being created now that could be acquired in the future. It could potentially more than double the size of the AI alignment R&D.
Paying existing employees to do safety R&D seems easier than paying academics. Academics often like to follow their own ideas but employees are already doing what their superior tells them to. In fact, they may find alignment and safety R&D more motivating than their company’s existing mission. Additionally, some founders may be more willing to sell to a non-profit org with a social-good mission than to Big Tech.
Big tech companies acquire small companies all the time. The reasons for this vary (e.g. killing competition), but overall it suggests that it can be feasible and even profitable.
Caveats:
1) A highly qualified replacement may be needed for the top-level management.
2) Some employees may leave after an acquisition. This seems more likely if the pivot towards safety is a big change to the skills and workflows. Or if the employees don’t like the new mission. It seems possible to partially avoid both of these by acquiring the right companies and steering them towards a mission that is relevant to their existing work. For example, natural language generation startups would usually benefit from fine-tuning their models with alignment techniques.
I’m no expert in this area but I’m told that European think-tanks are often strapped for cash so that may explain why the funders get so much influence (which is promising for the funder of course but it may not generalize to the US).
IIRC Tristan Harris has also made this claim. Maybe his 80k podcast or The Social Dilemma has some clues.
Edit: maybe he just said something like ‘Youtube’s algorithm is trained to send users down rabbit hole’
Re why AI isn’t generating much revenue—have you considered the productivity paradox? It’s historically normal that productivity slows down before steeply increasing when a new general purpose technologies arrives.
See “Why Future Technological Progress Is Consistent with Low Current Productivity Growth” in “Artificial Intelligence and the Modern Productivity Paradox”
Instructions for that: http://www.eccentrictraining.com/6.html
That’s really interesting, thanks! Do you (or someone else) have a sense of how much variation in priorities can be explained by the big 5?
Another contributing factor might be that EAs tend to get especially worried when pain stops them from being able to do their work. That would certainly help explain the abnormally high prevalence of wrist pain from typing among EAs.
(NB this wrist pain happened to me years ago and I did get very worried.)