I was already strongly considering moving to Boston, so this makes me feel lucky :)
Neat! Sadly I can’t interact with the grants.futureoflife.org webpage yet because my “join the community” application is still sitting around.
I think moderated video calls are my favorite format, as boring as that is. I.e. you have a speaker and also a moderator who picks people to ask questions, cuts people off or prompts them to keep talking depending on their judgment, etc.
Another thing I like, if it seems like people are interested in talking about multiple different things after the main talk / QA / discussion, is splitting up the discussion into multiple rooms by topic. I think Discord is a good application for this. Zoom is pretty bad at this but can be cajoled into having the right functionality if you make everyone a co-host, I think Microsoft Teams is fine but other people have problems, and other people think GatherTown is fine but I have problems.
I’m curious about your takes on the value-inverted versions of the repugnant and very-repugnant conclusions. It’s easy to “make sense” of a preference (e.g. for positive experiences) by deciding not to care about it after all, but doing that doesn’t actually resolve the weirdness in our feelings about aggregation.
Once you let go of trying to reduce people to a 1-dimensional value first and then aggregate them second, as you seem to be advocating here in ss. 3⁄4, I don’t see why we should try to hold onto simple rules like “minimize this one simple thing.” If the possibilities we’re allowed to have preferences about are not 1-dimensional aggregations, but are instead the entire self-interacting florescence of life’s future, then our preferences can get correspondingly more interesting. It’s like replacing preferences over the center of mass of a sculpture with preferences about its pose or theme or ornamentation.
Academics choose to work on things when they’re doable, important, interesting, publishable, and fundable. Importance and interestingness seem to be the least bottlenecked parts of that list.
The root of the problem is difficulty in evaluating the quality of work. There’s no public benchmark for AI safety that people really believe in (nor do I think there can be, yet—talk about AI safety is still a pre-paradigmatic problem), so evaluating the quality of work actually requires trusted experts sitting down and thinking hard about a paper—much harder than just checking if it beat the state of the art. This difficulty restricts doability, publishability, and fundability. It also makes un-vetted research even less useful to you than it is in other fields.
Perhaps the solution is the production of a lot more experts, but becoming an expertise on this “weird” problem takes work—work that is not particularly important or publishable, and so working academics aren’t going to take a year or two off to do it. At best we could sponsor outreach events/conferences/symposia aimed at giving academics some information and context to make somewhat better evaluations of the quality of AI safety work.
Thus I think we’re stuck with growing the ranks of experts not slowly per se (we could certainly be growing faster), but at least gradually, and then we have to leverage that network of trust both to evaluate academic AI safety work for fundability / publishability, and also to inform it to improve doability.
That’s a good point. I’m a little worried that coarse-grained metrics like “% unemployment” or “average productivity of labor vs. capital” could fail to track AI progress if AI increases the productivity of labor. But we could pick specific tasks like making a pencil, etc. and ask “how many hours of human labor did it take to make a pencil this year?” This might be hard for diverse task categories like writing a new piece of software though.
What would a plausible capabilities timeline look like, such that we could mark off progress against it?
Rather than replacing jobs in order of the IQ of humans that typically end up doing them (the naive anthropocentric view of “robots getting smarter”), what actually seems to be happening is that AI and robotics develop capabilities for only part of a job at a time, but they do it cheap and fast, and so there’s an incentive for companies/professions to restructure to take advantage of AI. Progressions of jobs eliminated is therefore going to be weird and sometimes ill-defined. So it’s probably better to try to make a timeline of capabilities, rather than a timeline of doable jobs.
Actually, this probably requires brainstorming from people more in-touch with machine learning than me. But for starters, human-level performance on all current quantifiable benchmarks (from Allen Institute’s benchmark of primary-school test questions [easy?] to Mine-RL BASALT [hard?]) would be very impressive.
Scalability, or cost?
When I think of failure to scale, I don’t just think of something with high cost (e.g. transmutation of lead to gold), but something that resists economies of scale.
Level 1 resistance is cost-disease-prone activities that haven’t increased efficiency in step with most of our economy, education being a great example. Individual tutors would greatly increase results for students, but we can’t do it. We can’t do it because it’s too expensive. And it’s too expensive because there’s no economy of scale for tutors—they’re not like solar panels, where increasing production volume lets you make them more cheaply.
Level 2 resistance is adverse network effects—the thing actually becomes harder as you try to add more people. Direct democracy, perhaps? Or maintaining a large computer program? It’s not totally clear what the world would have to be like for these things to be solvable, but it would be pretty wild; imagine if the difficulty of maintaining code scaled sublinearly with size!
Level 3 resistance is when something depends on a limited resource and if you haven’t got it, you’re out of luck. Stradivarius violins, perhaps. Or the element europium used in red-emitting phosphor for CRT tubes. Solutions to these, when possible, probably just look like better technology allowing a workaround.