You said you were looking for “when the ideas started gathering people”. I do suspect there’s an interesting counterfactual where in-person gathering wasn’t a major part of the EA movement. I can think of some other movements where in-person gathering is not focal. In any case, I’m not hung up on the distinction, it just seemed worth mentioning.
John_Maxwell
Karma: 3,261
I think the “fulltime job as a scientist” situation could be addressed with an “apply for curation” process, as outlined in the second half of this comment.
Thanks a lot for writing this post!
Personal experience: When I tried a vegan diet, I experienced gradually decreasing energy levels and gradually increasing desire for iron-rich animal products (hamburgers). My energy levels went back to normal when I went ahead and ate the hamburgers.
So, I’m really excited about the potential of nutritional investigations to improve vegan diets!
For bivalvegans, note that some bilvalves are rich in heme iron (heme iron, from animals, is more easily absorbed than the non-heme iron found in plants).
Again, personal experience: I’ve found that if I’m still feeling tired after multiple days of rest, consuming food with significant heme iron usually solves the problem. (Think this might have been true one time even when I didn’t meet the technical criteria for iron deficiency?)
But, just to reinforce the point that too much iron can also be bad:
Donating blood can improve your cardiovascular health. Elevated levels of iron in the blood puts men at increased risk of heart disease. Donating blood takes iron out of your system (It’s gradually replenished by the foods you eat).
Thanks for all your hard work, Megan.
I’m reminded of this post from a few months ago: Does Sam make me want to renounce the actions of the EA community? No. Does your reaction? Absolutely.
And this point from a post Peter Wildeford wrote: “I think criticism of EA may be more discouraging than it is intended to be and we don’t think about this enough.”
In theory, the EA movement isn’t about us as EAs. It’s about doing good for others. But in practice, we’re all humans, and I think it’s human nature to have an expectation of recognition/gratitude when we’ve done an altruistic act. If instead of gratitude, we get a punishment in the form of a bad outcome or sharp words, that feels like a bait & switch.
My hypothesis is that being surrounded by other do-gooders makes the situation worse. You feel like you’re in a recognition deficit, many people around you feel the same way, and no one is injecting gratitude into the ecosystem to resolve the misery spiral. Internal debates exacerbate things, insofar as trying to understand someone else’s perspective depletes the same emotional resource that altruism does.
Anyway, most of that wasn’t very specific to your post—I’m just wondering if emphasizing “other-care” in addition to “self-care” would help us weather ups & downs.
And, thanks to all the EAs reading this for all the good you are doing.
I wonder if a good standard rule for prizes is that you want a marketing budget which is at least 10-20% the size of the prize pool, for buying ads on podcasts ML researchers listen to or subreddits they read or whatever. Another idea is to incentivize people to make submissions publicly, so your contest promotes itself.
Title: Prizes for ML Safety Benchmark Ideas
Author: Joshc, Dan H
Why it’s good: Benchmarks have been a big driver of progress in AI. Benchmarks for ML safety could be a great way to drive progress in AI alignment, and get people to switch from capabilities-ish research to safety-ish research. The structure of the prize looks good: They’re offering a lot of money, there are still over 6 months until the submission deadline, and all they’re asking for is a brief write-up. Thinking up benchmarks also seems like the sort of problem that’s a good fit for a prize. My only gripe with the competition is that it doesn’t seem widely known, hence posting here.
There are hundreds of startup incubators and accelerators—is there a particular reason you like Entrepreneur First?
Interesting points.
I think we had a bunch of good shots of spotting what was going on at FTX before the rest of the world, and I think downplaying Sam’s actual involvement in the community would have harmed that.
I could see this going the other way as well. Maybe EAs would’ve felt more free to criticize FTX if they didn’t see it as associated with EA in the public mind. Also, insofar as FTX was part of the “EA ingroup”, people might’ve been reluctant to criticize them due to tribalism.
I also think that CEA would have very likely approved any request by Sam to be affiliated with the movement, so your safeguard would have I think just differentially made it harder for the higher-integrity people (who CEA sadly tends to want to be associated less with, due to them by necessity also having more controversial beliefs) to actually be affiliated with EA, without helping much with the Sam/FTX case.
Re: controversial beliefs, I think Sam was unusually willing to bite bullets in public even by EA standards—see here.
Presumably any CEA approval process from here on would account for lessons learned from Sam. And any approval process would hopefully get better over time as data comes in about bad actors.
In any case, I expect that paying for audits (or criticism contests, or whatever) is generally a better way to achieve scrutiny of one’s organization than using EA in one’s marketing.
I think it would be terrible if EA updated from the FTX situation by still giving fraudsters a ton of power and influence, but now just don’t publicly associate with them.
I don’t think fraudsters should be given power and influence. I’m not sure how you got that from my comment. My recommendation was made in the spirit of defense-in-depth.
I can see how a business founder trying to conceal their status as an EA might create an adversarial relationship, but that’s not what I suggested.
Put it another way: SBF claimed he was doing good with lots of fanfare, but actually did lots of harm. The next EA billionaire should focus less on claiming they’re doing good, and more on actually doing good for their employees, customers, shareholders, and donation recipients.
Our laws are the end result of literally thousands of years of of experimentation
The distribution of legal cases involving technology over the past 1000 years is very different than the distribution of legal cases involving technology over the past 10 years. “Law isn’t keeping up with tech” is a common observation nowadays.
a literal random change to the status quo
How about we revise to “random viable legislation” or something like that. Any legislation pushed by artists will be in the same reference class as the “thousands of years of of experimentation” you mention (except more recent, and thus better adapted to current reality).
AI is so radically outside the ordinary reference class of risks that it is truly nothing whatsoever like we have ever witnessed or come across before
Either AI will be transformative, in which case this is more or less true, or it won’t be transformative, in which case the regulations matter a lot less.
Suppose that as a result of neo-luddite sentiment, the people hired to oversee AI risks in the government concern themselves only with risks to employment, ignoring what we’d consider to be more pressing concerns.
If we’re involved in current efforts, maybe some of the people hired to oversee AI risks will be EAs. Or maybe we can convert some “neo-luddites” to our point of view.
simply hire right-minded people in the first place
Sounds to me like you’re letting the perfect be the enemy of the good. We don’t have perfect control over what legislation gets passed, including this particular legislation. Odds are decent that the artist lobby succeeds even with our opposition, or that current legislative momentum is better aligned with humanity’s future than any legislative momentum which occurs later. We have to think about the impact of our efforts on the margin, as opposed to thinking of a “President Matthew Barnett” scenario.
On the other hand, I’m quite convinced that, abstractly, it is highly implausible that arbitrarily limiting what data researchers have access to will be positive for alignment.
It could push researchers towards more robust schemes which work with less data.
I want a world where the only way for a company like OpenAI to make ChatGPT commercially useful is to pioneer alignment techniques that will actually work in principle. Throwing data & compute at ChatGPT until it seems aligned, the way OpenAI is doing, seems like a path to ruin.
As an intuition pump, it seems possible to me that a solution for adversarial examples would make GPT work well even when trained on less data. So by making it easy to train GPT on lots of data, we may be letting OpenAI neglect adversarial examples. We want an “alignment overhang” where our alignment techniques are so good that they work even with a small dataset, and become even better when used with a large dataset. (I guess this argument doesn’t work in the specific case of safety problems which only appear with a large dataset, but I’m not sure if there’s anything like that.)
Another note: I’ve had the experience of sharing alignment ideas with OpenAI staff. They responded by saying “what we’re doing seems good enough” / not trying my idea (to my knowledge). Now they’re running into problems which I believe the ideas I shared might’ve solved. I wish they’d focus more on finding a solid approach, and less on throwing data at techniques I view as subpar.
...their regulations will probably not, except by coincidence, be the type of regulations we should try to install.
A priori, I’d expect a randomly formulated AI regulation to be about 50% likely to be an improvement on the status quo, since the status quo wasn’t selected for being good for alignment.
Adopting the wrong AI regulations could lock us into a suboptimal regime that may be difficult or impossible to leave.
I don’t see good arguments supporting this point. I tend to think the opposite—building a coalition to pass a regulation now makes it easier to pass other regulations later.
arbitrary data restrictions risk preventing researchers from having access to good data that might help with alignment
OpenAI claims that ChatGPT is an “alignment advance”. I think this is way off—the approach they’re using just isn’t good enough in principle. Incrementally improving on ChatGPT’s “alignment” the way OpenAI is doing leads to disaster, IMO. You don’t write the code for a nuclear reactor through trial and error.
If an alignment scheme doesn’t work with arbitrary data restrictions, it’s probably not good enough in principle. Even all the data on the internet is “arbitrarily restricted” relative to all the data that exists or could exist. If my alignment scheme fails with public domain data only, why shouldn’t it also fail with all the data on the internet? (And if an alignment scheme works on public domain data only, it should be possible to add in more data to boost performance before using the AI for something mission-critical.)
I think a better argument for your conclusion is that incentivizing researchers to move away from big data approaches might make AI research more accessible and harder to control. Legal restrictions also favor open source, which has the same effect. We don’t want cutting-edge AI to become something that exists on the margins, like using bittorrent for piracy.
I suspect the right compromise is for AI art companies to pay a license fee to any artist in their training data who signs up to be paid. We want to reduce company profits and decrease AI progress incentives without pushing things towards open source a bunch. I’d take the same approach for language models like GPT. I think our interests are actually fairly well-aligned with artists here, in the sense that the approach which allocates profits away from AI companies and towards artists is probably also the best approach for humanity’s long-run future.
Suppose you saw a commercial on TV. At the end of the commercial a voice says “brought to you by Effective Altruism”. The heart-in-lightbulb logo appears on screen for several seconds.
I actually did hear of a case of a rando outside the community grabbing a Facebook page for “Effective Altruism”, gaining a ton of followers, and publishing random dubious stuff.
You can insist EA isn’t a brand all you want, but someone still might use it that way!
I’m not super attached to getting permission from CEA in particular. I just like the idea of EAs starting more companies, and dislike the idea of them often advertising those companies as EA?
Maybe a good thing to point out is that while the survival criterion for nonprofits is donation (i.e. almost by definition nonprofits must achieve the approval of philanthropists), the survival criterion for companies is profitability. Imagine “Superstimulus Slot Machines Inc.” puts the EA logo on the side of their machine and runs a bunch of commercials explaining how all profits go to EA charities. This might be a really profitable business, and become the first thing people think of when they hear “EA”, without any EA outside the company ever signing on. [Note: Please don’t start this company, there are many better business ideas that don’t involve harming people!]
If the process for making this sort of corporate branding decision is fuzzy, it becomes easier for people to tilt it in their favor. So I think an explicit process makes sense, for the same reason it makes sense to preregister data analysis code before data gets collected. If you don’t like the “ask CEA” process, maybe you could suggest an alternative and explain why it’s better?
With recent FTX news, EA has room for more billionaire donors. For any proposed EA cause area, a good standard question to ask is: “Could this be done as a for-profit?” Quoting myself from a few years ago:
There are a few reasons I think for-profit is generally preferable to non-profit when possible:
It’s easier to achieve scale as a for-profit.
For-profit businesses are accountable to their customers. They usually only stay in business if customers are satisfied with the service they provide. Non-profits are accountable to their donors. The impressions of donors correlate imperfectly with the extent to which real needs are being served.
First worlders usually aren’t poor and don’t need charity.
You can donate the money you make to effective charities.
Before anyone jumps on me here: IMO, an important takeaway from the FTX catastrophe is that EA-founded businesses should avoid mentioning EA in their marketing by default. Even if you think you have a good reason to use EA in your marketing, you should still get CEA’s permission first.
Other good ways to not be like SBF include: Have detailed knowledge of what people are buying and accurately communicate it to customers, including your own uncertainty. Pay special attention to identifying and mitigating ways in which your product could do harm (e.g. for a chewy food product, choking is an obvious potential hazard). Do these things beyond what’s required by law, and what’s pragmatic from the point of view of maximizing profit. Be willing to pull the plug on the business proactively if risks seem too high, or things aren’t going in a good direction.
If you aren’t able to achieve healthy profits while respecting such ethical guidelines, that’s an indicator that the business idea isn’t promising and you should find something else. Good business ideas are in “blue oceans” with little competition, plus somewhere you can build a durable competitive moat, meaning you won’t be caught in a race to the bottom.
Anyway, back to the post. Your proposed interventions are “mostly bottlenecked on improving awareness”. In the business world, awareness is achieved through marketing. You could sell a product which helps kids with jaw development, and your marketing could improve awareness of this problem as a side effect.
What product could you sell? Perhaps some sort of kid-themed chewy snacks—a nutty animal cracker? Or maybe a long, thin stick to minimize choking risk (with spiky sides so even if it gets caught in the throat they can still breathe around it?) Maybe the snacks could come in grades of chewing difficulty, so you can put your kid on a step-by-step program of jaw development, gradually ramping them up from the soft foods they’re eating right now.
It’d be very tempting to promote this by saying: “buy this $10 snack and you could save thousands on orthodontics down the road”. That could be an amazing sales pitch. But you want to be very careful in making any kind of health claim. There are lawyers who specialize in navigating FDA regulations around such claims. Maybe you’d want to start with a study measuring your product’s impact on some sort of near-term proxy for jaw development, as a basis for eventually making such a claim.
In terms of the professional treatment side, you could build a directory of dentists who e.g. work with expanders instead of doing extractions. Then create a tool that helps a parent find such a dentist in their area, solicit dentist reviews from parents, advertise your tool, and make money through lead generation.
However, that doesn’t really change my point that usually the reason a new idea seems wacky and strange is because it’s wrong.
I think seeming wacky and strange is mainly a function of difference, not wrongness per se.
I’d argue that the best way to evaluate the merits of a wacky idea is usually to consider it directly. And discussing wacky ideas is what brings them from half-baked to fully-baked.
If you can find a good way to count up the historical reference class of “wacky and strange ideas being explored by highly educated contrarians” and quantify the percentage of such ideas which were verifiable duds, I’d be very interested to see that post. (The “highly educated” part is doing a lot of work here btw—I know there’s a lot of random occult type stuff that never goes anywhere.) I don’t think we’re going to get anywhere talking about biases—my view is that people are biased in the other direction! (Maybe that’s the correct bias to have if you aren’t experienced in the ways of highly educated contrarianism, though.)
Interesting argument!
I’m not fully persuaded, because I think we’re dealing with heterogeneous sub-populations.
Consider the statement “As a non-EA, I believe that EA funders don’t allocate enough capital to funding development econ research”. I don’t think we can conclude from this statement that the opposite is true, and EA funders allocate too much capital to development econ research.
The heterogeneous subpopulations perspective suggests that people who think development econ research is the most promising cause may be self-selecting out of the “dedicated EA” subpopulation. I think criticism can be helpful in mitigating self-selection effects of this kind.
Of course, we can’t conclude that people who self-select out of EA on the basis of some disagreement are taking the correct side of that disagreement. The point is that criticism allows us to hear their perspective even if they’re not heavily involved.
BTW, I thought the outline format was fine for this post. Some individual sentences were choppy, but that was fine after I decided to read less thoroughly because it was a draft.
My sense is if you look at “wacky and strange ideas being explored by highly educated contrarians” as a historical reference class, they’ve been important enough to be worth paying attention to. I would put pre-WWW discussion & exploration of hypermedia in this category, for instance. And the first wiki was a rather wacky and strange thing. I think you could argue that the big ideas underpinning EA (RCTs, veganism, existential risk) were all once wacky and strange. (Existential risk was certainly wacky and strange about 10-15 years ago.)
One extremely under-rated impact of working harder is that you learn more. You have sub-linear short-term impact with increasing work hours because of things like burnout, or even just using up the best opportunities, but long-term you have super-linear impact (as long as you apply good epistemics) because you just complete more operational cycles and try more ideas about how to do the work.
Working more hours could help learning in the sense of helping you collect data faster. But if you want to learn from the data you already have, I’d suggest working fewer hours (or taking a vacation) to facilitate a reflective mindset.
I had a friend who took a stimulant every day, working at a startup, only to wake up many months later and realize the startup wasn’t going anywhere & he’d wasted a lot of time. Beware tunnel vision. Once you’re at 40 hours a week, it’s generally more valuable to work smarter than work harder IMO.
Variant: “EA funds should do small-scale experiments with mechanisms like quadratic voting and prediction markets, that have some story for capturing crowd wisdom while avoiding both low-info voting and single points of failure. Then do blinded evaluation of grants to see which procedure looks best after X years.”
Not necessarily a deliberate strategy though—my model is that EA started out fairly cause-neutral, people had lots of discussions about the best causes, and longtermist causes gradually emerged as the best.
E.g. in 2012 Holden Karnofsky wrote: