Building bespoke quantitative models to support decisionmakers in AI and bio. Right now that means: forecasting capabilities gains due to post-training enhancements on top of frontier foundation models, and estimating the annual burden of airborne disease in the US.
Joel Becker
Karma: 1,717
[Link post] Michael Nielsen’s “Notes on Existential Risk from Artificial Superintelligence”
Jeff is also a great presenter
Seconding this… I once saw Jeff give a 30 minute talk, completely unprepared, without using a filler word even once. Easy podcast guest to edit!
Thank you very much for sharing, Chloe.
Ben, Kat, Emerson, and readers of the original post have all noticed that the nature of Ben’s process leads to selection against positive observations about Nonlinear. I encourage readers to notice that the reverse might also be true. Examples of selection against negative information include:
Ben has reason to exclude stories that are less objective or have a less strong evidence base. The above comment is a concrete example of this.
There’s also something related here about the supposed unreliability of Alice as a source: Ben needs to include this to give a complete picture/because other people (in particular the Nonlinear co-founders) have said this. I strongly concur with Ben when he writes that he “found Alice very willing and ready to share primary sources [...] so I don’t believe her to be acting in bad faith.” Personally, my impression is that people are making an incorrect inference about Alice from her characteristics (that are perhaps correlated with source-reliability in a large population, but aren’t logically related, and aren’t relevant in this case).
To the extent that you expect other people to have been silenced (e.g. via anticipated retaliation), you might expect not to hear relevant information from them.
To the extent that you expect Alice and Chloe to have had burnout-style experiences, you might expect not to read clarifications on or news about negative experiences.
Until this post came out, this was true of ~everything in the post.
There is a reason the post was published 1.5 years after the relevant events took place—people involved in the events really do not want to spend further mental effort on this.
Thank you very much for posting!
I think that you should add an edit removing Drew’s name, for this reason if nothing else. (Happy to expand.) Thank you.
Repost from LW:
My understanding (definitely fallible, but I’ve been quite engaged in this case, and am one of the people Ben interviewed) has been that Alice and Chloe are not concerned about this, and in fact that they both wish to insulate Drew from any negative consequences. This seems to me like an informative and important consideration. (It also gives me reason to think that the benefits of gaining more information about this are less likely to be worth the costs.)
Here’s another thing.
That’s a red line in my book, and I will not personally work with Nonlinear in the future because of it, and I recommend their exclusion from any professional communities that wish to keep up the standard of people not being silenced about extremely negative work experiences.
Let’s suppose that Nonlinear have crossed red lines, and that additional information from them won’t change this. (In reality I think that this is up in the air for the next week or so; I won’t allow my limited imagination to diminish the hope.)
Do you not believe in the possibility of rehabilitation in this case?
I haven’t read up on what norms here work well in other high-trust communities. But at least in criminal vs. society settings I would want to be a strong proponent of rehabilitation. It seems pretty plausible to me that, after thinking more about best norms in high-trust communities, I could come to think that “create horrendous work environment” and “create credible fear of severe retaliation” were things that could change (and be monitored) upon rehabilitation, and that it would be good for this to happen after X period of time.
Thank you Ben—please check comment mentions of Drew, too!
Yes, I think that the post does not do enough to make it clear that the central allegations are not about Drew Spartz. Happy to expand.
Agreed. I would have wanted the post itself to make this more clear.
I’m unclear on how this comment speaks to the content of the post, which is compatible with Kat being a courageous, frugal, and dedicated friend and leader.
As one of the people Ben interviewed:
This post closely reflects my understanding of the situation. (EDIT: at this time, before engaging with Nonlinear reply myself.)
Whenever this post touches on something that I can independently corroborate (EDIT: small minority of claims), I believe it to be accurate. Whenever the post communicates something that both Ben and I have heard from Alice and Chloe (EDIT: large majority of claims), it tells their account faithfully.
I appreciate Ben’s emphasis on red lines and the experiences of Alice and Chloe. When he leaves out stories that I think we are both aware of, my guess is that he has done so because these stories aren’t super relevant to the case at hand or aren’t super objective/strongly evidenced. This makes me think more favourably of the rest of his write-up.
To put it more strongly: I would like to make clear that I have never heard any claims of improper conduct by Drew Spartz (in relation to the events discussed in this post or otherwise).
Kat, I am really sorry about the severe emotional difficulty. It makes sense that having this post be public would be an extremely challenging thing to deal with, all the more so if you have decisive contrary evidence. I will be interested in engaging with whatever you present, once you have the opportunity.
I think it is important to say, as one of the people who Ben interviewed: my very strong impression has been that Ben is interested in the truth, and that he is acting in good faith. My guess is that if you have strong, contrary evidence regarding the most important claims, then Ben will engage with this evidence with an open mind and will signal boost if relevant.
God bless your clear thinking and strong stance-taking, Habryka.
If you can share (publicly or privately) strong evidence contradicting “claims [...] that wildly distort the true story” (emphasis mine), I pre-commit to signal boosting.
For what it’s worth, I wouldn’t be surprised if you do have strong counter-evidence to some claims (given the number of claims made, the ease with which things can be lost in translation, your writing this email, etc.). But, as of right now, I would be surprised if my understanding of the important stuff—roughly, the items covered in Ben’s epistemic state and the crossing of red lines—was wildly distorted. I hope that it is.
[EDIT, Nov 13: it sounds like the Nonlinear reply might be in the 100s of pages. This might be the right move from their point of view, but reading 3-figure pages stretches my pre-commitment above further than I would have intended at the time. I’d like to amend the commitment to “engaging with the >=20 pages-equivalent that seems most relevant to me or Nonlinear, or skimming >=50 pages-equivalent.” If people think this is breaking the spirit of my earlier commitment, I’ll seriously consider standing by the literal wording of that commitment (engaging with ~everything). Feel free to message about this.]
Dan Hendrycks’ lecture on “Safety-Capabilities Balance” might be helpful here.
I’ll speak only for myself—I haven’t spoken to everyone else about this. I think Reslab lost momentum for 3 reasons. In descending order of importance:
Uncertainty about the funding environment. We actually heard only positive things from grantmakers. But it still seemed like there might be a significant risk of not receiving funding conditional on (reasonably) successful outcomes. (Also, less confidently, the time delays in grantmaker responses were significant enough to dampen momentum.) In these conditions, it was hard to convince high-opportunity-cost staff + external stakeholders to go full speed ahead.
Not having someone whose main focus was Reslab. Everyone had non-Reslab projects that were more important to them than Reslab.
Reputational damage from FTX. I have the weak sense that advertising to engineering talent outside the EA ecosystem was going to be more difficult after the collapse of FTX.
No problem, thanks for engaging!
I wrote that most of GPT-4’s suggestions were “obvious or unhelpful.” I would include the ones you pointed to in this box. Pilot funding and tiered grants are presumably things you already do implicitly—e.g. by not committing funding for multiple years in one go, or by not giving huge resources to grantees that you don’t think highly of—and where you wouldn’t benefit much from making this more explicit. And mentorship or guidance seems unhelpful because it’s much too time-costly.
I’m guessing that GPT-4 is trying to point to ‘ways to lower the information asymmetry’ characteristic of adverse selection. All three of these methods give money-cheap ways of gaining more information before making money-expensive decisions.
Nonlinear staff were participants on the FTX EA program, which I ran, and where I was in part responsible for participant welfare. Some of the important events took place in this period. This led me to start supporting Alice and Chloe. I have continued to be involved in the case on-and-off since then.