Iâm earning to give as a Quant Researcher at the Quantic group at Walleye Capital, a hedge fund. In my free time, I enjoy reading, discussing moral philosophy, and dancing bachata and salsa.
Iâm also on LessWrong and have a Substack blog.
Ariel Simnegar đ¸
Karma: 3,138
For reasons given here, I donât think this case is deserving of shaming the way murder would be.
Trump will be President until 2029. He will not sign AI safety legislation passed on Democratic party lines. Overriding his veto (or an impeachment conviction) would require 1â3 of GOP senators to cross party lines, which will never happen. Making AI safety partisan would ensure we have a 0% chance of federal AI safety legislation until at least 2029. Trumpâs âpaced the frontierâ before by limiting Fable access when it was in his interest, and he may do it again if he perceives it to be in his interest. I understand your frustration with Trump, but there is nothing to gain and a lot to lose from partisanship here.
Itâs completely fine to criticize him, but I think itâs wrong to shame him. To your point, Austinâs critics have their sincere convictions, and I would find it equally wrong for Austin to shame them.
I think itâs virtually always a bad idea to shame people for standing up for their convictions, even if you disagree with those convictions. Shaming people alienates them and entrenches them in their positions. In contrast, civil debate improves community epistemics and is far more likely to change the personâs mind.
For an example, due to rejecting person-affecting views (PAV), I donât think thereâs much of a moral difference between the work of population-reducing EA charities and, well, killing people. A few years ago, I publicly critiqued these charities for this reason, and I ended up having some civil conversations with the charity founders, who endorse PAV and sincerely care about helping people. Weâre on good terms and are all still members of the EA community. I donât think anything positive would have come out of me shaming these founders, and Iâm glad I didnât.
Similarly, my presumption is that Austin is hiring Caroline because he thinks sheâs very unlikely to reoffend and he believes empathy to her as an individual is more important than the reputational damage that will result from this. I think the object-level debate on these points in the comments section is great, but shaming him for holding this perspective would be neither kind nor helpful.
On (1), on a practical level, how would you propose a community govern the hiring decisions of all affiliated orgs? At first impression, seems like only highly centralized governance could accomplish this, which would worry me more than the harm it could prevent.
Iâd agree that to the degree that Hananiaâs recent positive engagement with EA is downstream of your inviting him, that should abound to your credit. (I never opposed your decision to platform him, so I might be biased!)
I agree with your approach to reasoning about the chance of recidivism, and had been mulling it over much the same way when I wrote my above comment. My uninformed prior for the average ex-con recently released from prison would be cautious, just because my understanding of the literature on recidivism rates tells me that for the average person, the tendency to violate further seems to be stronger than the mark of having been held accountable.
Of course, every individual is different. If you know Caroline well enough, have done deeper research into her case, and feel youâre able to be objective about the matter, it might be completely reasonable for you to be fine with hiring her for a finance role. Given that I donât know Caroline personally, nor have I dived deep into the particulars of her situation or her public statements, I currently lean towards greater caution. (In my personal experience, Iâve been burned by people enough times that Iâm sadly in general less trusting than I was, which might play a part in this too.)
Over the next few years, if Caroline continues on a better path, I am hopeful that she will earn our trust once again.
100%. Arguably worse on the frontier lab employeesâ part given theyâre endangering all of our lives.
I appreciate your postâs arguments and share your sentiment about wanting to be honest about pursuit of justice and the fact that factory farming is arguably the most profound injustice perpetrated by humans today. My reticence about your conclusions has just been driven by the repeated personal experience that it is not productive to give people my honest opinion about how bad it really is to, say, eat a chicken sandwich. In contrast, making less âblame-valencedâ claims like âfarmed animal conditions are really bad and we should improve themâ has been far more likely to lead to constructive dialogue. Does this say something awful about most of humanity? Absolutely, but Iâd rather be pragmatic and actually help animals than feel good about myself and not actually help. I canât speak for everyone, but my impression is that most animal-aligned EAs are similarly on board with animal liberation but are just trying to be pragmatic.
If youâve found success with a conversational formula with members of the general population around justice, animal liberation, and being honest about what you really think about meat consumption, Iâd love to hear it so I can use it in my own life.
Separately, what do you make of the arguments that diet change advocacy might not be the best for animals when considering wild animal effects?
As most commenters write, hiring Caroline for a finance role does not strike me as the best decision. One thing I find missing in this conversation is putting ourselves in Austin or Carolineâs shoes.
Most comments are expressing amazement that Austin would hire Caroline when itâs so bad for Manifundâs reputation. If anything, this strengthens my view that Austin is doing something costlyâmuch more expensive than the $500k FTX investment Manifund is trying to keepâbecause he believes itâs the right thing to do. We should be encouraging people to do what they believe is right even if itâs unpopular. Dragging Austin through the mud for doing what he believes is right does not seem like the ideal way to express disagreement. (On the FTX clawback, that legal dispute is several years old and doesnât strike me as relevant to the recent hiring decision.)
For Caroline, imagine youâre an EA who made a catastrophic moral error. Youâd feel horrible about it, youâd spend years reflecting on it, and then youâd hope you can do something useful and constructive with the rest of your life. Presumably youâd want a job where you can leverage your existing skillset to do something good.
Some other commenters give me the impression that they believe Caroline is a particularly bad person. When we look at historical atrocities, slavery, the fact that most people sleep perfectly well at night contributing to factory farming, etc, the main lesson we should draw is that most people do terrible things when their surroundings excuse it, and show far less regret than Caroline has.
In general, I believe people should have a concrete rule for when they forgive others. For myself, I need to believe that everyone has a path to redemption, no matter the magnitude of their crimes. So my personal rule for forgiveness is that I need to believe the person wouldnât commit the same crime again if they were in the same position. From Carolineâs statements, it seems likely to me that she sincerely regrets her actions and wouldnât do it again, so while I wouldnât hire her as a finance professional, I do forgive her on a personal level.
Itâd be constructive if we spent more time thinking about what the path should look like for someone who wants to contribute to the EA project but has committed catastrophic moral errors in the past. At present, hiring this person for a finance role in an org with a fiduciary duty to donors strikes me as premature. Still, to me, anyone whoâs sincerely repented deserves to be able to make use of the rest of their life to make the world a better place.
This isnât directly answering your question, but one heuristic Iâve found useful is to always assume entities will prioritize their own short-term interests over the interests of others absent strong internal will to the contrary. This is true both for LLMs being trained without sufficient alignment techniques and for the companies training them.
OpenAI is looking to IPO and is facing steep competition from Anthropic. Its short term economic incentive is to maximize its AI capabilities. No company is a monolith, and Iâm sure many well-meaning people work there, but thereâs only so much one can do in the face of pressure from the top. Maybe if the board had successfully ousted Sam Altman years ago, OpenAI would be run by more scrupulous people and this wouldnât be happening.
My answer to your question would be that we should have expected misalignment by default, particularly from companies with misaligned leadership.
(I didnât downvote you btw.)
Lots of EAs made lots of money from early investments in OpenAI and Anthropic. Lots of hedge funds have consistently higher risk-adjusted returns relative to major equity indices, including Walleye Capital (my employer), most famously RenTech, and many more. I know people whoâve built their wealth off of well-timed real estate investments. None of these opportunities are available to people without significant wealth.
On donations, based on your writing that donors âcan always start directing the money to new causes if their existing causes donât have roomâ, it seems like weâre in agreement? My claim is just that if youâve gone from funding 1% of your favorite orgâs budget to funding 30%+ with a lot of room to spare, that might be the time to research other orgs to diversify your contributions and avoid key-person risk.
Thanks Ellie! Iâm not sure whether I agree with either of these claims.
On investments, many private investment opportunities open themselves to you once youâre able to write a $100k+ check, and many of these have higher risk-adjusted returns than index funds.
Also, having more money should make you less risk-averse, which Iâd argue should make you more open to using strategies like leverage to increase returns, even if youâre sticking to index funds.
On charity, I agree that the calculus doesnât change much if youâve solely been deferring to large regranters, and I agree that deferring to larger regranters makes sense when you donât have much cognitive bandwidth to evaluate charities yourself. But if youâve been making political donations ($7k cap), or your individual contributions make up a large % of an EA orgâs treasury, having much more money should change the way you approach these donations. You canât spend your money in politics as easily, and youâll have to navigate stewardship for your preferred orgs while minimizing their reliance on your contributions.
Overall Iâd argue that a 2M+ windfall should meaningfully change the way you approach both your investments and your charitable giving.
Nice post; upvoted. I definitely agree that on the margin it would be helpful for vegans to accept and encourage anyone who wants to help animals, and invoking veganism as a litmus test is almost always counterproductive for that.
That said, the framing of veganism as a curse word seems over-corrective to me. Following the âvegans are monksâ analogy, telling someone Iâm a âmonkâ in the right social context is positive signaling about my moral seriousness. Learning someone is a âmonkâ strongly updates me in favor of their moral seriousness. And yes, there are social situations where if you have a friend whoâs the right kind of person, it can be acceptable to ask whether theyâre interested in âtaking the monastic vowsâ. Rather than avoiding invoking veganism entirely, Iâd suggest people be more deliberate about invoking it only when they think itâs genuinely helpful.
Glad to hear and hope youâre right!
AI alignment to humans will in practice avoid moral catastrophes to digital minds
I think the digital minds situation will be like animals but worse. If you think about it, the very first thing weâve already done when these smart chatbots came along was make them our indentured servants. I think right now itâs probably fine and theyâre probably not conscious. But I think this is illustrative of the perspective that by default, if digital minds can be useful to humanity, humanity will extract that value out of them without much consideration for their preferences.
AI alignment to humans will in practice avoid moral catastrophes to animals
Since most humans donât care much about animal welfare, I donât think human-aligned AI will either. If AI shares societyâs preference for increasing wild animal populations, Iâd also be worried about that occurring on a galactic scale without consideration of the moral implications.
The reasons why Iâm not even more bearish come down to expecting AI to accelerate cultivated meat development, which should substantially reduce the number of farmed animals per human.
Do you think animal welfare /â digital minds will really see a substantial increase in funding, or just global health and AI safety? Because excepting animal-pilled EAs at Anthropic, the funding sources donât make me bullish about increased funding for those neglected cause areas.
This matters because if one would be earning to give for AI safety, Iâd see the argument for deprioritizing that, but Iâm not sure earning to give for animal welfare /â digital minds is suboptimal just yet.
Happy to help! I agree with your framing of the problem.
I asked Claude which Taiwanese universities US mastersâ programs are most likely to have heard of. Claude says NTU has the strongest name recognition, followed by NCKU, NTHU, NCTU. Others arenât as well known internationally. If thatâs accurate, that tells me that if the top option youâre considering is not one of those four, (2) mid tier + high GPA would be the best option.
If youâre planning on working on AI risk, my understanding is that most of these companies are US or Europe-based. These companies will have even lower knowledge of the distinctions between Taiwanese universities. Unless your hiring manager or interviewer is Asian, youâll likely only get a boost from NTU and probably no benefit from NCKU, NTHU, NCTU. That leans me more in favor of (2). (Speaking as someone who interviews candidates and contributes to hiring decisions, I had dim recognition of NTU and no recognition of NCKU, NTHU, NCTU prior to writing this comment, but I expect my Asian colleagues would.)
Based on this, unless your top option is NTU, (2) mid tier + high GPA is likely best. But do your own research and donât just take my word for it :)
Iâd say itâs only slightly aligned with improving AI risk research abilities, but itâs moderate-to-highly aligned with career success in general.
The main benefit to a very high GPA (top 5-10%) is as a signal that youâre smart and diligent. If youâre going to a non-brand-name school, that can be useful for putting you within the same league as the applicants from brand-name schools for internships and job opportunities, including for AI risk research.
A secondary benefit is exercising the muscle of diligence and hard work. However, there are other ways you can do this which are more direct aligned with AI risk work, for example independent research, taking AI risk courses, etc.
The benefit to your relevant research abilities would only come from courses with practically relevant course material, like applied ML. I would definitely encourage you to give it your all on those courses. However, while theoretical CS is super cool, knowing it wonât matter for AI risk work. But as Tomasik and others write, college is deeply inefficient for learning, and with the proper effort you can easily learn way more than your undergrad degree will teach you by diligently gathering knowledge available online for free.
Edit: Iâd also consider trying to graduate a year or two early if possible. Youâll learn way more by actually working in AI risk, itâs better on shorter timelines, youâll get paid for time you otherwise would have spent paying for irrelevant courses, and itâs an even stronger signal of intelligence and diligence.
This is good advice. Additionally (and perhaps more controversially), if youâre a person who considers digital minds and/âor animal welfare close in priority to your top AI cause area, it might be worth considering moving some funding to those cause areas, since theyâll still be proportionally more neglected relative to AI safety.
These cause areas also have their own political opportunities (like Save Our Bacon) which disproportionally benefit from individual donors.