Wei’s list focused on ethics and decision theory, but I think that it would be most valuable to have more good conceptual analysis of the arguments for why AI safety matters, and particularly the role of concepts like “agency”, “intelligence”, and “goal-directed behaviour”. While it’d be easier to tackle these given some knowledge of machine learning, I don’t think that background is necessary—clarity of thought is probably the most important thing.
This all makes sense, and it does seem that people who are launching big projects might benefit from paid emails as a norm. On the other hand, you seem unusually worried about “spamming” people by sending them things it’s pretty plausible they’d be interested in. It would be fairly easy to put at the top of your email something like “If you’re interested in doing AI forecasting, read on; otherwise feel free to ignore this email” which means the cost is something like ~10 seconds per uninterested recipient, which seems reasonable.
On a meta note, I think I felt less positively towards this post than I otherwise would have, because it felt like a call to action (which I hold to high standards) rather than an exploratory poll—e.g. I read the first few bullet points as rhetorical questions. Seems like it was just a phrasing issue; and as an exploratory poll, I think it’s interesting and I’m glad to have had the issue brought to mind :)
It’s not clear to me that we are in a mess. The only actual example you gave was a spammy corporate newsletter, which seems irrelevant.
This might look as follows: Lots of people write to senior researchers asking for feedback on papers or ideas, yet they’re mostly crackpots or uninteresting, so most stuff is not worth reading. A promising young researcher without many connections would want their feedback (and the senior researcher would want to give it!), but it simply takes too much effort to figure out that the paper is promising, so it never gets read. In fact, expecting this, the junior researcher might not even send it in the first place.
Does this happen much? Have you received feedback from people saying that this has happened to them? I expect personal networks in EA to be pretty good at connecting people—and if a young researcher is promising they can often explain why in a sentence or two (even if it’s just by name-dropping previous positions).
Currently, the signalling problem is solved by things like:
Spending lots of effort crafting interesting-sounding intros which signal that the thing is worth reading, instead of just getting to the point
Burning social capital—adding tags like “[Urgent]” or “[Important]” to the subject line
Does the latter actually happen? I’ve never seen it. Also, why is the former bad? It seems like an even better costly signal than paying money to send emails because it also produces a short description of the work which helps the recipient evaluate it. And very few people have too little time to skim a paragraph-long summary.
I think I enjoyed Diaspora more, and it seems a little more relevant to far-future considerations. What about Permutation City in particular did you like?
Meta: your last link doesn’t seem to point anywhere.
Interesting post. I wanted to write a substantive response, but ran out of energy. However, I have written previously on why I’m skeptical of the relevance of formally defined utility functions to ethics. Here’s one essay about the differences between people’s preferences and the type of “utility” that’s morally valuable. Here’s one about why there’s no good way to ground preferences in the real world. And here’s one attacking the underlying mindset that makes it tempting to model humans are agents with coherent goals.
There are two functions I’m looking for: the “archive/index” function, and the “sequences” function. The former should store as much EA content as possible (including stuff that’s not high-quality enough for us to want to direct newcomers to); it’d ideally also have enough structure to make it easily-browsable. The latter should zoom in on a specific topic or person and showcase their ideas in a way that can be easily read and digested.
https://priority.wiki/ is somewhere in between those two, in a way that seems valuable, but that doesn’t quite fit with the functions I outlined above. It doesn’t seem like it’s aiming to be an exhaustive repository of content. But the individual topic pages also don’t seem well-curated enough that I could just point someone to them and say “read all the stuff on this page to learn about the topic”. The latter might change as more work goes into it, but I’m more hopeful about the EA forum sequences feature for this purpose.
The list of syllabi on EAHub is also interesting, and fits with the sequences function, albeit only on one specific topic (introducing EA).
Are those what you were referring to, or are there other places on EAHub where (object-level) content is collected that I didn’t spot?
I was particularly reminded of this by spending twenty minutes yesterday searching for an EA blog I wanted to cite, which has somehow vanished into the aether. EDIT: never mind, found it.
Collection and curation of EA content from across the internet in a way that’s accessible to newcomers, easily searchable, and will last for decades (at least). Seems like it wouldn’t take that long to do a decent job, doesn’t require uncommon skills, and could be pretty high-value.
I would be open to paying people to do this; message me if interested.
I think Eli was asking whether your whole response was a quote, since the whole thing is in block quote format.
What’s your position on people coming for only part of the workshop? I’d be interested in attending but would rather not miss more than one day of work.
Strong +1 for the kalzumeus blog post, that was very helpful for me.
In general, stocking programmes aim at supporting commercial fisheries.
I’m a little confused by this, since it seems hugely economically inefficient to go to all the effort of raising fish, only to release them and then recapture them. Am I missing something, or is this basically a make-work program for the fishing industry?
Note that your argument here is roughly Ben Pace’s position in this post which we co-wrote. I argued against Ben’s position in the post because I thought it was too extreme, but I agree with both of you that most EAs aren’t going far enough in that direction.
Excellent post, although I think about it using a slightly different framing. How vetting-constrained granters are depends a lot on how high their standards are. In the limit of arbitrarily high standards, all the vetting in the world might not be enough. In the limit of arbitrarily low standards, no vetting is required.
If we find that there’s not enough capability to vet, that suggests that either our standards are correct and we need more vetters, or that our standards are too high and we should lower them. I don’t have much inside information, so this is mostly based on my overall worldview, but I broadly think it’s more the latter: that standards are too high, and that worrying too much about protecting EA’s reputation makes it harder for us to innovate.
I think it would be very valuable to have more granters publicly explaining how they make tradeoffs between potential risks, clear benefits, and low-probability extreme successes; if these explanations exist and I’m just not aware of them, I’d appreciate pointers.
Another startup contacted at least 4 grantmaking organisations. Three of them deferred to the fourth.
One “easy fix” would simply be to encourage grantmakers to defer to each other less. Imagine that only one venture capital fund was allowed in Silicon Valley. I claim that’s one of the worst things you could do for entrepreneurship there.
I agree that all of the things you listed are great. But note that almost all of them look like “convince already-successful people of EA ideas” rather than “talented young EAs doing exceptional things”. For the purposes of this discussion, the main question isn’t when we get the first EA senator, but whether the advice we’re giving to young EAs will make them more likely to become senators or billion-dollar donors or other cool things. And yes, there’s a strong selection bias here because obviously if you’re young, you’ve had less time to do cool things. But I still think your argument weighs only weakly against Vishal’s advocacy of what I’m tempted to call the “Silicon Valley mindset”.
So the empirical question here is something like, if more EAs steer their careers based on a Silicon Valley mindset (as opposed to an EA mindset), will the movement overall be able to do more good? Personally I think that’s true for driven, high-conscientiousness generalists, e.g. the sort of people OpenPhil hires. For other people, I guess what I advocate in the post above is sort of a middle ground between Vishal’s “go for extreme growth” and the more standard EA advice to “go for the most important cause areas”.
Is this not explained by founder effects from Less Wrong?
One other thing that I just noticed: looking at the list of 80k’s 10 priority paths found here, the first 6 (and arguably also #8: China specialist) are all roles for which the majority of existing jobs are within an EA bubble. On one hand, this shows how well the EA community has done in creating important jobs, but it also highlights my concern about us steering people away from conventionally successful careers and engagement with non-EAs.