What would be the pros and cons of adding a semi-hidden-but-permanent Hot Takes section to the Forum? All of my takes are Hot and due to time constraints I would otherwise not post at all. Some would argue that someone like me should not post Hot Takes at all. Anyway, in true lazy fashion here is ChatGPT on the pros and cons:
Pros:
Encourages diverse perspectives and stimulates debate.
Can attract more engagement and interest from users.
Provides a platform for expressing unconventional or controversial ideas.
Fosters a culture of intellectual curiosity and open discourse within the community.
Cons:
May lead to increased polarization and conflict within the community.
Risk of spreading misinformation or poorly researched opinions.
Could divert attention from more rigorous and evidence-based discussions.
Potential for reputational damage if controversial opinions are associated with the forum.
TL;DR: Someone should probably write a grant to produce a spreadsheet/dataset of past instances where people claimed a new technology would lead to societal catastrophe, with variables such as “multiple people working on the tech believed it was dangerous.”
Slightly longer TL;DR: Some AI risk skeptics are mocking people who believe AI could threaten humanity’s existence, saying that many people in the past predicted doom from some new tech. There is seemingly no dataset which lists and evaluates such past instances of “tech doomers.” It seems somewhat ridiculous* to me that nobody has grant-funded a researcher to put together a dataset with variables such as “multiple people working on the technology thought it could be very bad for society.”
*Low confidence: could totally change my mind
———
I have asked multiple people in the AI safety space if they were aware of any kind of “dataset for past predictions of doom (from new technology)”, but have not encountered such a project. There have been some articles and arguments floating around recently such as “Tech Panics, Generative AI, and the Need for Regulatory Caution”, in which skeptics say we shouldn’t worry about AI x-risk because there are many past cases where people in society made overblown claims that some new technology (e.g., bicycles, electricity) would be disastrous for society.
While I think it’s right to consider the “outside view” on these kinds of things, I think that most of these claims 1) ignore examples of where there were legitimate reasons to fear the technology (e.g., nuclear weapons, maybe synthetic biology?), and 2) imply the current worries about AI are about as baseless as claims like “electricity will destroy society,” whereas I would argue that the claim “AI x-risk is >1%” stands up quite well against most current scrutiny.
(These claims also ignore the anthropic argument/survivor bias—that if they ever were right about doom we wouldn’t be around to observe it—but this is less important.)
I especially would like to see a dataset that tracks things like “were the people warning of the risks also the people who were building the technology?” More generally, some measurement of “analytical rigor” also seems really important, e.g., “could the claims have stood up to an ounce of contemporary scrutiny (i.e., without the benefit of hindsight)?”
Absolutely seems worth spending up to $20K to hire researchers to produce such a spreadsheet within the next two-ish months… this could be a critical time period, where people are more receptive to new arguments/responses…?
Just saw this now, after following a link to another comment.
You have almost given me an idea for a research project. I would run the research honestly and report the facts, but my in-going guess is that survivor bias is a massive factor, contrary to what you say here. And that in most cases, the people who believed it could lead to catastrophe were probably right to be concerned. A lot of people have the Y2K bug mentality, in which they didn’t see any disaster and so concluded that it was all a false-alarm, rather than the reality which is that a lot of people did great work to prevent it.
If I look at the different x-risk scenarios the public is most aware of:
Nuclear annihilation—this is very real. As is nuclear winter.
Climate change. This is almost the poster-child for deniers, but in fact there is as yet no reason to believe that the doom-saying predictions are wrong. Everything is going more or less as the scientists predicted, if anything, it’s worse. We have just underestimated the human capacity to stick our heads in the ground and ignore reality*.
Pandemic. Some people see covid as proof that pandemics are not that bad. But we know that, for all the harm it wrought, covid was far from the worst-case. A bioweapon or a natural pandemic.
AI—the risks are very real. We may be lucky with how it evolves, but if we’re not, it will be the machines who are around to write about what happened (and they will write that it wasn’t that bad …)
Etc.
My unique (for this group) perspective on this is that I’ve worked for years on industrial safety, and I know that there are factories out there which have operated for years without a serious safety incident or accident—and someone working in one of those could reach the conclusion that the risks were exaggerated, while being unaware of cases where entire factories or oil-rigs or nuclear power plants have exploded and caused terrible damage and loss of life.
Before I seriously start working on this (in the event that I find time), could you let me know if you’ve since discovered such a data-base?
*We humans are naturally very good at this, because we all know we’re going to die, and we live our lives trying not to think about this fact or desperately trying to convince ourselves of the existence of some kind of afterlife.
Everything is going more or less as the scientists predicted, if anything, it’s worse.
I’m not that focused on climate science, but my understanding is that this is a bit misleading in your context—that there were some scientists in the (90s/2000s?) who forecasted doom or at least major disaster within a few decades due to feedback loops or other dynamics which never materialized. More broadly, my understanding is that forecasting climate has proven very difficult, even if some broad conclusions (e.g., “the climate is changing,” “humans contribute to climate change”) have held up. Additionally, it seems that many engineers/scientists underestimated the pace of alternative energy technology (e.g., solar).
That aside, I would be excited to see someone work on this project, and I still have not discovered any such database.
I’m not sure. IMHO a major disaster is happening with the climate. Essentially, people have a false belief that there is some kind of set-point, and that after a while the temperature will return to that, but this isn’t the case. Venus is an extreme example of an Earth-like planet with a very different climate. There is nothing in physics or chemistry that says Earth’s temperature could not one day exceed 100 C.
It’s always interesting to ask people how high they think sea-level might rise if all the ice melted. This is an uncontroversial calculation which involves no modelling—just looking at how much ice there is, and how much sea-surface area there is. People tend to think it would be maybe a couple of metres. It would actually be 60 m (200 feet). That will take time, but very little time on a cosmic scale, maybe a couple of thousand years.
Right now, if anything what we’re seeing is worse than the average prediction. The glaciers and ice sheets are melting faster. The temperature is increasing faster. Etc. Feedback loops are starting to be powerful. There’s a real chance that the Gulf Stream will stop or reverse, which would be a disaster for Europe, ironically freezing us as a result of global warming …
Among serious climate scientists, the feeling of doom is palpable. I wouldn’t say they are exaggerating. But we, as a global society, have decided that we’d rather have our oil and gas and steaks than prevent the climate disaster. The US seems likely to elect a president who makes it a point of honour to support climate-damaging technologies, just to piss off the scientists and liberals.
Venus is an extreme example of an Earth-like planet with a very different climate. There is nothing in physics or chemistry that says Earth’s temperature could not one day exceed 100 C. [...] [Regarding ice melting -- ] That will take time, but very little time on a cosmic scale, maybe a couple of thousand years.
I’ll be blunt, remarks like these undermine your credibility. But regardless, I just don’t have any experience or contributions to make on climate change, other than re-emphasizing my general impression that, as a person who cares a lot about existential risk and has talked to various other people who also care a lot about existential risk, there seems to be very strong scientific evidence suggesting that extinction is unlikely.
Anyone else ever feel a strong discordance between emotional response and cognitive worldview when it comes to EA issues?
Like emotionally I’m like “save the animals! All animals deserve love and protection and we should make sure they can all thrive and be happy with autonomy and evolve toward more intelligent species so we can live together in a diverse human animal utopia, yay big tent EA…”
But logically I’m like “AI and/or other exponential technologies are right around the corner and make animal issues completely immaterial. Anything that detracts from progress on that is a distraction and should be completely and deliberately ignored. Optimally we will build an AI or other system that determines maximum utility per unit of matter, possibly including agency as a factor and quite possibly not, so that we can tile the universe with sentient simulations of whatever the answer is.”
OR, a similar discordance between what was just described and the view that we should also co-optimize for agency, diversity of values and experience, fun, decentralization, etc., EVEN IF that means possibly locking in a state of ~99.9999+percent of possible utility unrealized.
Very frustrating, I usually try to push myself toward my rational conclusion of what is best with a wide girth for uncertainty and epistemic humility, but it feels depressing, painful, and self-de-humanizing to do so.
You may know will.i.am as the frontman of The Black Eyed Peas, but his interests beyond music have taken him down a fascinating path at the intersection of creativity and technology. In a recent podcast, he discussed his thoughts on AI and the creative process with host Adam Grant.
Some key points:
Adam notes that the most creative people are often the worst at explaining their ideas, because creativity requires divergent, non-linear thinking while explanation favors convergence and linearity.
The podcast features some impressive live wordplay and freestyling from will.i.am. His verbal creativity is on full display.
Interestingly, will.i.am now hosts a radio show with an AI co-host named Fiona. He shares his hopes about the future of AI in entertainment and creativity.
will.i.am and Adam debate what AI can and can’t do for human creativity. No definitive answers, but a great discussion nonetheless.
I didn’t previously associate will.i.am with the AI scene, but he clearly has an innovative and forward-thinking perspective to share. Worth a listen for anyone interested in the intersection of AI and creativity.
You can now import posts directly from Google docs
Plus, internal links to headers[1] will now be mapped over correctly. To import a doc, make sure it is public or shared with “eaforum.posts@gmail.com″[2], then use the widget on the new/edit post page:
Importing a doc will create a new (permanently saved) version of the post, but will not publish it, so it’s safe to import updates into posts that are already published. You will need to click the “Publish Changes” button to update the live post.
Everything that previously worked on copy-paste[3] will also work when importing, with the addition of internal links to headers (which only work when importing).
There are still a few things that are known not to work:
Nested bullet points (these are working now)
Cropped images get uncropped
Bullet points in footnotes (these will become separate un-bulleted lines)
Blockquotes (there isn’t a direct analog of this in Google docs unfortunately)
There might be other issues that we don’t know about. Please report any bugs or give any other feedback by replying to this quick take, you can also contact us in the usual ways.
Appendix: Version history
There are some minor improvements to the version history editor[4] that come along with this update:
You can load a version into the post editor without updating the live post, previously you could only hard-restore versions
The version that is live[5] on the post is shown in bold
Here’s what it would look like just after you import a Google doc, but before you publish the changes. Note that the latest version isn’t bold, indicating that it is not showing publicly:
Previously the link would take you back to the original doc, now it will take you to the header within the Forum post as you would expect. Internal links to bookmarks (where you link to a specific text selection) are also partially supported, although the link will only go to the paragraph the text selection is in
Sharing with this email address means that anyone can access the contents of your doc if they have the url, because they could go to the new post page and import it. It does mean they can’t access the comments at least
I’m not sure how widespread this knowledge is, but previously the best way to copy from a Google doc was to first “Publish to the web” and then copy-paste from this published version. In particular this handles footnotes and tables, whereas pasting directly from a regular doc doesn’t. The new importing feature should be equal to this publish-to-web copy-pasting, so will handle footnotes, tables, images etc. And then it additionally supports internal links
For most intents and purposes you can think of “live” as meaning “showing publicly”. There is a bit of a sharp corner in this definition, in that the post as a whole can still be a draft.
To spell this out: There can be many different versions of a post body, only one of these is attached to the post, this is the “live” version. This live version is what shows on the non-editing view of the post. Independently of this, the post as a whole can be a draft or published.
There might be other issues that we don’t know about. Please report any bugs or give any other feedback by replying to this quick take, you can also contact us in the usual ways.
2 nitpicks:
The title of the doc is imported as the 1st paragraph of the EA Forum post, instead of being imported as the title.
Blank lines without spacing before and after in the doc are not imported, although I personally think this is a feature! Blank lines without spacing before and after in the footnotes of the doc are imported, but I would rather not have them imported.
I’ll think about how we could handle this one better. It’s tricky because the doc itself as a title, and then people often rewrite the title as a heading inside the doc, so there isn’t an obvious choice for what to use as the title. But it may be true that the heading case is a lot more common so we should make that the default.
That was indeed intended as a feature, because a lot of people use blank lines as a paragraph break. We can add that to footnotes too.
I’ll set a reminder to reply here when we’ve done these.
I have thought this might be quite useful to do. I would guess (people can confirm/correct me) a lot of people have a workflow like:
Edit post in Google doc
Copy into Forum editor, make a few minor tweaks
Realise they want to make larger edits, go back to the Google doc to make these, requiring them to either copy over or merge together the minor tweaks they have made
For this case being able to import/export both ways would be useful. That said it’s much harder to do the other way (we would likely have to build up the Google doc as a series of edits via the api, whereas in our case we can handle the whole post exported as html quite naturally), so I wouldn’t expect us to do this in the near future unfortunately.
Yep images work, and agree that nested bullet points are the biggest remaining issue. I’m planning to fix that in the next week or two.
Edit: Actually I just noticed the cropping issue, images that are cropped in google docs get uncropped when imported. That’s pretty annoying. There is no way to carry over the cropping but we could flag these to make sure you don’t accidentally submit a post with the uncropped images.
It sounds like the Future of Humanity Institute may be permanently shut down.
Background: FHI went on a hiring freeze/pause back in 2021 with the majority of staff leaving (many left with the spin-off of the Centre for the Governance of AI) and moved to other EA organizations. Since then there has been no public communication regarding its future return, until now…
″...Those were heady years. FHI was a unique place—extremely intellectually alive and creative—and remarkable progress was made. FHI was also quite fertile, spawning a number of academic offshoots, nonprofits, and foundations. It helped incubate the AI safety research field, the existential risk and rationalist communities, and the effective altruism movement. Ideas and concepts born within this small research center have since spread far and wide, and many of its alumni have gone on to important positions in other institutions.
Today, there is a much broader base of support for the kind of work that FHI was set up to enable, and it has basically served out its purpose. (The local faculty administrative bureaucracy has also become increasingly stifling.) I think those who were there during its heyday will remember it fondly. I feel privileged to have been a part of it and to have worked with the many remarkable individuals who flocked around it.”
This language suggests that FHI has officially closed. Can anyone at Trajan/Oxford confirm?
Also curious if there is any project in place to conduct a post mortem on the impact FHI has had on the many different fields and movements? I think it’s important to ensure that FHI is remembered as a significant nexus point for many influential ideas and people who may impact the long term.
In other news, Bostrom’s new book “Deep Utopia” is available for pre-order (coming March 27th).
Further evidence: The 80,000 Hours website footer no longer mentions FHI. Until February 2023, the footer contained the following statement:
We’re affiliated with the Future of Humanity Institute and the Global Priorities Institute at the University of Oxford.
By February 21, that statement was replaced with a paragraph simply stating that 80k is part of EV. The references to GPI, CEA and GWWC were also removed:
I like the “we decided to shut down our charity because it wasn’t very effective post” for the obvious reasons, but I wonder once you control for “is rich / comes from a rich family” and “doesn’t have family that depends on them” how that metric gets affected.
I think it is still good praise the decision in general, but unless I know the backgrounds of the people doing it I can’t heap too much praise on them.
If I’m understanding this concern correctly, it’s along the lines of: “they’re not making a financial sacrifice in shutting down, so it’s less praiseworthy than it otherwise would be”.
Just to clarify, charity founders (at least CE ones) take a pay cut to start their charity—they would earn more if working for other EA organizations as employees, and much more if in tech/finance/consulting/careers that typical of people with oxbridge/ivy/etc education levels. The financial sacrifice was already made when starting the charity, and if anything, quitting is actually better for you financially.
I’m confused about where you’re going with this. Why would the founders’ personal financial situation substantially affect how we viewed their decision?
The government’s sentencing memorandum for SBF is here; it is seeking a sentence of 40-50 years.
As typical for DOJ in high-profile cases, it is well-written and well-done. I’m not just saying that because it makes many of the same points I identified in my earlier writeup of SBF’s memorandum. E.g., p. 8 (“doubling down” rather than walking away from the fraud); p. 43 (“paid in full” claim is highly misleading) [page cites to numbers at bottom of page, not to PDF page #].
EA-adjacent material: There’s a snarky reference to SBF’s charitable donations “(for which he still takes credit)” (p. 2) in the intro, and the expected hammering of SBF’s memo for taking credit for attempting to take credit for donations paid with customer money (p. 95). There’s a reference to SBF’s “idiosyncratic . . . beliefs around altruism, utilitarianism, and expected value” (pp. 88-89). This leads to the one surprise theme (for me): the need to incapacitate SBF from committing additional crimes (pp. 87, 90). Per the feds, “the defendant believed and appears still to believe that it is rational and necessary for him to take great risks including imposing those risks on others, if he determines that it will serve what he personally deems a worthy project or goal,” which contributes to his future dangerousness (p. 89).
For predictors: Looking at sentences where the loss was > $100MM and the method was Ponzi/misappropriation/embezzlement, there’s a 20-year, two 30-years, a bunch of 40-years, three 50-years, and three 100+-years (pp. 96-97).
Interesting item: The government has gotten about $3.45MM back from political orgs, and the estate has gotten back ~$280K (pp. 108-09). The proposed forfeiture order lists recipients, and seems to tell us which ones returned monies to the government (Proposed Forfeiture Order, pp. 24-43).
Life Pro Tip: If you are arrested by the feds, do not subsequently write things in Google Docs that you don’t want the feds to bring up at your sentencing. Jotting down the idea that “SBF died for our sins” as some sort of PR idea (p. 88; source here) is particularly ill-advised.
My Take: In Judge Kaplan’s shoes, I would probably sentence at the high end of the government’s proposed range. Where the actual loss will likely be several billion, and the loss would have been even greater under many circumstances, I don’t think a consequence of less than two decades’ actual time in prison would provide adequate general deterrence—even where the balance of other factors was significantly mitigating. That would imply a sentence of ~25 years after a prompt guilty plea. Backsolving, that gets us a sentence of ~35 years without credit for a guilty plea.
But the balance of other factors is aggravating, not mitigating. Stealing from lots of ordinary people is worse than stealing from sophisticated investors. Outright stealing by someone in a fiduciary role is worse than accounting fraud to manipulate stock prices. We also need to adjust upward for SBF’s post-arrest conduct, including trying to hide money from the bankruptcy process, multiple attempts at witness tampering, and perjury on the stand. Stacking those factors would probably take me over 50 years, but like the government I don’t think a likely-death-in-prison sentence is necessary here.
I’d look at pp. 5-12 of the linked sentencing memo for customers, pp. 15-18 for investors/lenders for the government’s statement of the offense conduct. The jury merely utters guilty / not guilty on each count, it does not provide detailed findings of fact. Judge Kaplan heard all the evidence as he presided at trial, and can rely on his own factual findings at sentencing under a more-likely-than-not standard. Of course, that is just a summary; Ellison alone testified for ~3 days.
Basically, SBF & FTX falsely represented that the customer assets were segregated from FTX’s own assets and would not be used by FTX. Yet Alameda regularly took large sums of money out of accounts holding FTX customer funds, and the “allow negative” feature allowed it to borrow ~unlimited money as well. This was not limited to funds made available to customers through the margin lending program.
At various points discussed on pp. 9-11, SBF directed Alameda to “borrow” more money from FTX despite knowing it was underwater at the time. For instance, at one point SBF directed Ellison “to use FTX customer funds to repay loans” (her words), despite knowing that Alameda was $11B in the hole and was already “borrowing” $13B in customer funds (p. 11). About $4.5B more in customer funds was used to pay Alameda’s lenders as a result (p. 11). In short, I don’t think the narrative presented in the linked post is backed up by the trial evidence.
One day some historian of effective altruism will marvel at how easily it transformed itself. —Michael Lewis, Going Infinite
A universal refusal to propagate the human species would be the greatest of conceivable crimes from a Utilitarian point of view —Henry Sidgwick, The Method of Ethics
NunoSempere points out that EA could have been structured in a radically different way, if the “specific cultural mileu” had been different. But I think this can be taken even further. I think that it’s plausible that if a few moments in the history of effective altruism had gone differently, the social makeup—the sort of people that make up the movement—and their axiological worldviews—the sorts of things they value—might have been radically different too.
As someone interested in the history of ideas, I’m fascinated by what our movement has that made it significantly different than the most likely counterfactual movements. Why is effective altruism the way it is? A numberofinterestingbriefhistorieshavebeen written about the history of EA (and longer pieces about more specific things like Moynihan’s excellent X-Risk) but I often feel that there are a lot of questions about the movement’s history, especially regarding tensions that seem to present themselves between the different worldviews that make up EA.
For example,
How much was it the individual “leaders” of EA who brought together different groups of people to create a big-tent EA, as opposed to the communities themselves already being connected? (Toby Ord says that he connected the Oxford GWWC/EA community to the rationality community, but people from both of these “camps” seem to be at Felicifia together in the late 2000s.)
When connecting the history of thought, there’s a tendency to put thinkers after one another in lineages as if they all read and are responding to those who came before them. Parfit lays the ground for longtermism in the the late 20th century in Reasons and Persons and Bostrom continues the work when presenting the idea of x-risk in 2001. Did Bostrom know of and expand upon Parfit’s work, or was Bostrom’s framing independent of that, based on risks discussed by the Extropians, Yudkowsky, SL4, etc? There (maybe) seems to be multiple discovery of early EA ideas in separate creation of the Oxford/GWWC community and GiveWell. Is something like that going on for longtermism/x-risk?
What would EA look like today without Yudkowsky? Bostrom? Karnofsky/Hassenfeld? MacAskill/Ord?
What would EA look like today without Dustin Moskovitz? Or if we had another major donor? (One with different priorities?)
What drove the “longtermist turn?” A shift driven by leaders or by the community?
A few interesting Yudkowsky (not be taken as current opinions, for historical purposes) quotes (see also Extropian Archaeology):
From Eliezer Yudkowsky on the SL4 mailing list, April 30, 2003:
Since the lack of people is a blocker problem, I think I may have to split my attention one more time, hopefully the last, and write something to attract the people we need. My current thought is a book on the underlying theory and specific human practice of rationality, which is something I’d been considering for a while. It has at least three major virtues to recommend it. (1): The Singularity movement is a very precise set of ideas that can be easily and dangerously misinterpreted in any number of emotionally attractive, rationally repugnant directions, and we need something like an introductory course in rationality for new members.
(2): Only a few people seem to have understood the AI papers already online, and the more recent theory is substantially deeper than what is currently online; I have been considering that I need to go back to the basics in order to convey a real understanding of these topics. Furthermore, much of the theory needed to give a consilient description of rationality is also prerequisite to correctly framing the task of building a seed AI.
(3): People of the level SIAI needs are almost certainly already rationalists; this is the book they would be interested in. I don’t think we’ll find the people we need by posting a job opening. Movements often start around books; we don’t have our book yet.
It’s fascinating to me that this is the reason that there’s a “rationality” community around today. (See also) What would EA look like without it? Would it really be any less rational? What does a transhumanisty non-AI-worried EA look like?—I feel like that’s what we might have had without Yudkowsky.
One last thing:
From Eliezer Yudkowsky on the Extropians mailing list, May 12, 2001:
I was there for the presentation and I, literally, felt slightly sick to my stomach. I’d like to endorse “Existential Risks” as being scary and well worth reading.
Feels kinda obnoxious to write a quick take along the lines of “I’m thinking about writing a post on X, does anyone actually give a sh*t? Otherwise I won’t write it.”
I just wanted to check, since I can’t place my finger on why it feels obnoxious but it certainly does.
I’ve loved seeing all the Draft Amnesty posts on the Forum so far! Some really great stuff has been posted (and I’ll highlight that when I write a retrospective)
Posting this quick take as a reminder that people who are considering posting for Draft Amnesty can run a draft past me for quick feedback. Just DM me.
Open Phil claims that campaigns to make more Americans go vegan and vegetarian haven’t been very successful. But does this analysis account for immigration?
If people who already live in the US are shifting their diets, but new immigrants skew omnivore, a simple analysis could easily miss the former shift because immigration is fairly large in the US.
But these advocates haven’t achieved the widespread dietary changes they’ve sought — and that boosters sometimes claim they have. Despite the claims, 6% of Americans aren’t vegan and vegetarianism hasn’t risen fivefold lately: Gallup polls show a constant 5-6% of Americans have identified as vegetarians since 1999 (Gallup found 2% identified as vegans the only time it asked, in 2012). The one credible poll showing vegetarianism doubling in recent years still found only 5-7% of Americans identifying as vegetarian in 2017 — consistent with the stable Gallup numbers.
Although the cited Gallup report doesn’t explicitly distinguish on immigrant status or ethnicity, it does say that “[a]lmost all segments of the U.S. population have similar percentages of vegetarians” while noting a larger difference in marital status.
As a brief example with easyish math, 15M out of 300M = 5%; 15M out of 330M (adding 30M extra meat eaters) only drops it to ~4.5%. Addition of 30M non-v*gan immigrants would mask an 1,500,000 increase in the number of non-immigrant vegetarians (15M/300M = 5% = 16.5M/330M). Without the 30M immigrants, the vegetarian population would have risen from 5% to 16.5M/300M = 5.5%. Given that the assumption that no immigrants are vegetarian is unrealistic, this shows that adding a good number of meat-eaters to the denominator doesn’t move the percentages much at all.
I am wondering whether people view EA vs. cause-specific field-building differently, especially about the Scout Mindset. My general thoughts are:
EA—Focuses on providing knowledge and evidence to facilitate the self-determination of individuals to rationally weigh up the evidence provided to decide on updating beliefs to inform actions wherever they may go. Scout Mindset is intrinsically valuable to provide flexibility and to update beliefs and work on the beliefs that individuals hold.
Field-Building—Focusing on convincing people that this is a cause area worth working on and will have a significant impact; less focus on individual thoughts based on the strength of the arguments and evidence field-builders already possess. Scout Mindset is instrumentally valuable to update and work on the beliefs that field-builders hold.
Argument for Instrumental value:
A more instrumental perspective is that it is much easier to ask someone to understand one thing and act on it rather than understand many things and struggle to act on any, which may be counterfactually more impactful.
Argument for Intrinsic value:
By focusing on the intrinsic value you’re measuring for the internal change process that occurs in EA to see and understand the reason behind different cultural shifts across time with specific emphasis on the potential for value-drift.
The core difference between the two, as I see it, is whether the community builder focuses on promoting the individual or the cause. However, this may be an oversimplification or unfair misrepresentation and I am keen to hear the community’s views.
As part of an AMA I put on X, I was asked for my “top five EA hot takes”. If you’ll excuse the more X-suited tone and spiciness, here they are:
1. OpenAI, Anthropic (and to a lesser extent DeepMind) were the worst cases of Unilateralists Curse of all time. EAs love to discourage enthusiastic newcomers by warning to not do “net negative” unilateralist actions (i.e. don’t start new projects in case they crowd out better, more “well thought through” projects in future, with “more competent” people doing them), but nothing will ever top the monumental unilateralist curse fuck up that was supporting Big AGI in it’s beginnings.
2. AI Safety is nothing without a Pause. Too many EAs are stuck in the pre-GPT-4 paradigm of maxing research, when it’ll all be for nothing unless we get a Pause first. More EAs should switch to Notkilleveryoneism/PauseAI/StopAGI.
3. EA is too elitist. We should be triaging the world’s problems like crazy, and the top 1-2% of people are more than capable of that (most jobs that need doing in EA don’t require top 0.1%).
4. EA is too PR focused—to the point where it actually backfires spectacularly and now there is lots of bad press [big example: SBF’s bad character being known about but not addressed].
5. Despite all it’s flaws, EA is good (and much better than the alternatives in most cases).
Could you explain why you think ‘too much focus being placed on PR’ resulted in bad press?
Perhaps something like: because people were worried about harming SBF’s public reputation they didn’t share their concerns with others, and thus the community as a whole wasn’t able to accurately model his character and act appropriately?
More like, some people did share their concerns, but those they shared them with didn’t do anything about it (because of worrying about bad PR, but also maybe just as a kind of “ends justify the means” thing re his money going to EA. The latter might actually have been the larger effect.).
Ah ok—I guess I would phrase it as ‘not doing anything about concerns because they were too focused on short-term PR’.
I would phrase it this way because, in a world where EA had been more focused on PR, I think we would have been less likely to end up with a situation like SBF (because us having more of a focus on PR would have resulted in doing a better job of PR).
Maybe half the community sees it that way. But not the half with all the money and power it seems. There aren’t (yet) large resources being put into playing the “outside game”. And there hasn’t been anything in the way of EA leadership (OpenPhil, 80k) admitting the error afaik.
Seems pretty dependent on how seriously you take some combination of AI x-risk in general, the likelihood that the naïve scaling hypothesis holding (if it even holds at all), and what the trade-off between empirical/theoretical work on AI Safety is no?
Regarding 2 - Hammers love Nails. EAs as Hammers, love research, so they bias towards seeing the need for more research (after all, it is what smart people do). Conversely, EAs are less likely (personality-wise) to be comfortable with advocacy and protests (smart people don’t do this). It is the wrong type of nail.
Although what you said might be part of the explanation for why many EAs focus on alignment or governance research rather than pause advocacy, I think the bigger part is that many EAs think that pause advocacy isn’t as good as research. See, e.g., some of these posts.
See all my comments and replies on the anti-pause posts. I don’t think any of the anti-pause arguments stand up if you put significant weight on timelines being short and p(doom) high (and viscerally grasp that yes, that means your own life is in danger, and those of your friends and family too, in the short term! It’s no longer just an abstract concern!).
[Question] How should we think about the decision relevance of models estimating p(doom)?
(Epistemic status: confused & dissatisfied by what I’ve seen published, but haven’t spent more than a few hours looking. Question motivated by Open Philanthropy’s AI Worldviews Contest; this comment thread asking how OP updated reminded me of my dissatisfaction. I’ve asked this before on LW but got no response; curious to retry, hence repost)
To illustrate what I mean, switching from p(doom) to timelines:
The recent post AGI Timelines in Governance: Different Strategies for Different Timeframes was useful to me in pushing back against Miles Brundage’s argument that “timeline discourse might be overrated”, by showing how choice of actions (in particular in the AI governance context) really does depend on whether we think that AGI will be developed in ~5-10 years or after that.
A separate takeaway of mine is that decision-relevant estimation “granularity” need not be that fine-grained, and in fact is not relevant beyond simply “before or after ~2030″ (again in the AI governance context).
Finally, that post was useful to me in simply concretely specifying which actions are influenced by timelines estimates.
Question: Is there something like this for p(doom) estimates? More specifically, following the above points as pushback against the strawman(?) that “p(doom) discourse, including rigorous modeling of it, is overrated”:
What concrete high-level actions do most alignment researchers agree are influenced by p(doom) estimates, and would benefit from more rigorous modeling (vs just best guesses, even by top researchers e.g. Paul Christiano’s views)?
What’s the right level of granularity for estimating p(doom) from a decision-relevant perspective? Is it just a single bit (“below or above some threshold X%”) like estimating timelines for AI governance strategy, or OOM (e.g. 0.1% vs 1% vs 10% vs >50%), or something else?
I suppose the easy answer is “the granularity depends on who’s deciding, what decisions need making, in what contexts”, but I’m in the dark as to concrete examples of those parameters (granularity i.e. thresholds, contexts, key actors, decisions)
e.g. reading Joe Carlsmith’s personal update from ~5% to >10% I’m unsure if this changes his recommendations at all, or even his conclusion – he writes that “my main point here, though, isn’t the specific numbers… [but rather that] here is a disturbingly substantive risk that we (or our children) live to see humanity as a whole permanently and involuntarily disempowered by AI systems we’ve lost control over”, which would’ve been true for both 5% and 10%
Or is this whole line of questioning simply misguided or irrelevant?
Some writings I’ve seen gesturing in this direction:
Carl Shulman disagrees, but his comment (while answering my 1st bullet point) isn’t clear in the way the different AI gov strategies for different timelines post is, so I’m still left in the dark – to (simplistically) illustrate with a randomly-chosen example from his reply and making up numbers, I’m looking for statements like “p(doom) < 2% implies we should race for AGI with less concern about catastrophic unintended AI action, p(doom) > 10% implies we definitely shouldn’t, and p(doom) between 2-10% implies reserving this option for last-ditch attempts”, which he doesn’t provide
Froolow’s attempted dissolution of AI risk (which takes Joe Carlsmith’s model and adds parameter uncertainty – inspired by Sandberg et al’s Dissolving the Fermi paradox – to argue that low-risk worlds are more likely than non-systematised intuition alone would suggest)
Froolow’s modeling is useful to me for making concrete recommendations for funders, e.g. (1) “prepare at least 2 strategies for the possibility that we live in one of a high-risk or low-risk world instead of preparing for a middling-ish risk”, (2) “devote significantly more resources to identifying whether we live in a high-risk or low-risk world”, (3) “reallocate resources away from macro-level questions like ‘What is the overall risk of AI catastrophe?’ towards AI risk microdynamics like ‘What is the probability that humanity could stop an AI with access to nontrivial resources from taking over the world?’”, (4) “When funding outreach / explanations of AI Risk, it seems likely it would be more convincing to focus on why this step would be hard than to focus on e.g. the probability that AI will be invented this century (which mostly Non-Experts don’t disagree with)”. I haven’t really seen any other p(doom) model do this, which I find confusing
I’m encouraged by the long-term vision of the MTAIR project “to convert our hypothesis map into a quantitative model that can be used to calculate decision-relevant probability estimates”, so I suppose another easy answer to my question is just “wait for MTAIR”, but I’m wondering if there’s a more useful answer to the “current SOTA” than this. To illustrate, here’s (a notional version of) how MTAIR can help with decision analysis, cribbed from that introduction post:
This question was mainly motivated by my attempt to figure out what to make of people’s widely-varying p(doom) estimates, e.g. in the appendix section of Apart Research’s website, beyond simply “there is no consensus on p(doom)”. I suppose one can argue that rigorous p(doom) modeling helps reduce disagreement on intuition-driven estimates by clarifying cruxes or deconfusing concepts, thereby improving confidence and coordination on what to do, but in practice I’m unsure if this is the case (reading e.g. the public discussion around the p(doom) modeling by Carlsmith, Froolow, etc), so I’m not sure I buy this argument, hence my asking for concrete examples.
It’s possible that investing right and gaming prediction markets[1], something in which EAs have an advantage (this also means that money gained through prediction markets is likely also money that other EAs lost, however), is potentially an easier, more cost-effective, and time-effective way to double your impact when donating to charities.
In addition, perhaps charities should try to multiply the amount of money they have to do good with, using methods like these.
I think acting on the margins is still very underrated. For e.g. I think 5x the amount of advocacy for a Pause on capabilities development of frontier AI models would be great. I also think in 12 months time it would be fine for me to reevaluate this take and say something like ‘ok that’s enough Pause advocacy’.
Basically, you shouldn’t feel ‘locked in’ to any view. And if you’re starting to feel like you’re part of a tribe, then that could be a bad sign you’ve been psychographically locked in.
The main point I make is that NIST may not be well suited to creating measurements for complex, multi-dimensional characteristics of language models—and that some people may be overestimating the capabilities of NIST because they don’t recognize how incomparable the Facial Recognition Vendor Test is to this situation of subjective metrics for GenAI and they don’t realize NIST arguably even botched MNIST (which was actually produced by Yann LeCun by recompiling NIST’s datasets). Moreover, government is slow, while AI is fast. Instead, I argue we should consider an alternative model such as federal funding for private/academic benchmark development (e.g., prize competitions).
I wasn’t sure if this warranted a full post, especially since it feels a bit late; LMK if you think otherwise!
There are some major differences with the type of standards that NIST usually produces. Perhaps the most obvious is that a good AI model can teach itself to pass any standardised test. A typical standard is very precisely defined in order to be reproducible by different testers. But if you make such a clear standard test for an LLM, it would, say, be a series of standard prompts or tasks, which would be the same no matter who typed them in. But in such a case, the model just trains itself on how to answer these prompts, or follows the Volkswagen model of learning how to recognize that it’s being evaluated, and to behave accordingly, which won’t be hard if the testing questions are standard.
So the test tells you literally nothing useful about the model.
I don’t think NIST (or anyone outside the AI community) has experience with the kind of evals that are needed for models, which will need to be designed specifically to be unlearnable. The standards will have to include things like red-teaming in which the model cannot know what specific tests it will be subjected to. But it’s very difficult to write a precise description of such an evaluation which could be applied consistently.
In my view this is a major challenge for model evaluation. As a chemical engineer, I know exactly what it means to say that a machine has passed a particular standard test. And if I’m designing the equipment, I know exactly what standards it has to meet. It’s not at all obvious how this would work for an LLM.
Sure! (I just realized the point about the MNIST dataset problems wasn’t fully explained in my shared memo, but I’ve fixed that now)
Per the assessment section, some of the problems with assuming that FRVT demonstrates NIST’s capabilities for evaluation of LLMs/etc. include:
Facial recognition is a relatively “objective” test—i.e., the answers can be linked to some form of “definitive” answer or correctness metric (e.g., name/identity labels). In contrast, many of the potential metrics of interest with language models (e.g., persuasiveness, knowledge about dangerous capabilities) may not have a “definitive” evaluation method, where following X procedure reliably evaluates a response (and does so in a way that onlookers would look silly to dispute).
The government arguably had some comparative advantage in specific types of facial image data, due to collecting millions of these images with labels. The government doesn’t have a comparative advantage in, e.g., text data.
The government has not at all kept pace with private/academic benchmarks for most other ML capabilities, such as non-face image recognition (e.g., Common Objects in Context) and LLMs (e.g., SuperGLUE).
It’s honestly not even clear to me whether FRVT’s technical quality truly is the “gold standard” in comparison to the other public training/test datasets for facial recognition (e.g., MegaFace); it seems plausible that the value of FRVT is largely just that people can’t easily cheat on it (unlike datasets where the test set is publicly available) because of how the government administers it.
For the MNIST case, I now have the following in my memo:
Even NIST’s efforts with handwriting recognition were of debatable quality: Yann LeCun’s widely-used MNIST is a modification of NIST’s datasets, in part because NIST’s approach used census bureau employees’ handwriting for the training set and high school students’ handwriting for the test set.[1]
Some may argue this assumption was justified at the time because it required that models could “generalize” beyond the training set. However, popular usage appears to have favored MNIST’s approach. Additionally, it is externally unclear that one could effectively generalize from the handwriting of a narrow and potentially unrepresentative segment of society—professional bureaucrats—to high schoolers’, and the assumption that this would be necessary (e.g., due to the inability to get more representative data) seems unrealistic.
I just had a call with a young EA from Oyo State in Nigeria (we were connected through the excellent EA Anywhere), and it was a great reminder of how little I know regarding malaria (and public health in developing countries more generally). In a very simplistic sense: are bednets actually the most cost effective way to fight against malaria?
I’ve read a variety of books on the development economics canon, I’m a big fan of the use of randomized control trials in social science, I remember worm wars and microfinance not being so amazing as people thought and critiques of Tom’s Shoes. I was thrilled when I first read Poor Economics, and it opened my eyes to a whole new world. But I’m a dabbler, not an expert. I haven’t done fieldwork; I’ve merely read popular books. I don’t have advanced coursework in this area.
It was nice to be reminded of how little I actually know, and of how superficial general interest in a field is not the same as detailed knowledge. If I worked professionally in development economics I would probably be hyper aware of the gaps in my knowledge. But as a person who merely dabbles in development as an interest, I’m not often confronted with the areas about which I am completely ignorant, and thus there is something vaguely like a Dunning-Kruger effect. I really enjoyed hearing perspectives from someone that knows a lot more than I do.
QALY/$ for promoting zinc as a common cold intervention
Epistemic status: Fun speculation. I know nothing about public health, and grabbed numbers from the first source I could find for every step of the below. I link to the sources which informed my point estimates.
Here’s my calculation broken down into steps:
Health-related quality of life effect for one year of common cold −0.2
~1.5 million QALY burden per year when aggregated across the US population
This is the average of estimating from the above (1e6) with what I get (2e6) when deriving the US slice of the total DALY burden from global burden of disease data showing 3% global DALYs come from URI
There’s probably a direct estimate out there somewhere
50% probability the right zinc lozenges with proper dosing can prevent >90% of colds. This comes from here, here, and my personal experience of taking zinc lozenges ~10ish occasions.
15% best case adoption scenario, from taking a log-space mean of
Masks 5%
General compliance rate 10-90%
100,000 QALYs/year is my estimate for the expected value of taking some all-or-nothing action to promote zinc lozenges (without the possibility of cheaply confirming whether they work) which successfully changes public knowledge and medical advice to promote our best-guess protocol for taking zinc.
$35 million is my estimate for how much we should be willing to spend to remain competitive with Givewell’s roughly 1 QALY/$71. This assumes a 5 year effect duration. I have no idea how much such a thing would cost but I’d guess at most 1 OOM of value is being left on the table here, so I’m a bit less bullish on Zinc than I was before calculating.
EDIT: I calculated the cost of supplying the lozenges themselves. Going off these price per lozenge, this 5 year USA supply of lozenges costs ~35 million alone. Presumably this doesn’t need to hit the Givewell spending bar, but just US government spending on healthcare.
what I get (2e6) when deriving the US slice of the total DALY burden from global burden of disease data showing 3% global DALYs come from URI
I’m seeing 0.25% globally and 0.31% for the US for URI in the GBD data, ~1 OOM lower (the direct figure for the US is 3.4e5, also ~1 OOM lower). What am I missing?
It feels like this is still a research problem needing larger scale trials. If the claims are true (i.e. the failures to achieve statistically significant results were due to not preparing and consuming lozenges in a particular way, rather than the successes being the anomalies) there are plenty of non-philanthropic entities (governments and employers and media as well as zinc supplement vendors) that would be incentivised to publicise more widely.
Lifeextension cites this https://pubmed.ncbi.nlm.nih.gov/24715076/ claiming “The results showed that when the proper dose of zinc is used within 24 hours of first symptoms, the duration of cold miseries is cut by about 50%” I’d be interested if you do a dig through the citation chain. The lifeextension page has a number of further links.
Looks like they plagiarized from this paper, which found:
Results: Thirteen placebo-controlled comparisons have examined the therapeutic effect of zinc lozenges on common cold episodes of natural origin. Five of the trials used a total daily zinc dose of less than 75 mg and uniformly found no effect. Three trials used zinc acetate in daily doses of over 75 mg, the pooled result indicating a 42% reduction in the duration of colds (95% CI: 35% to 48%). Five trials used zinc salts other than acetate in daily doses of over 75 mg, the pooled result indicating a 20% reduction in the duration of colds (95% CI: 12% to 28%).
What would be the pros and cons of adding a semi-hidden-but-permanent Hot Takes section to the Forum? All of my takes are Hot and due to time constraints I would otherwise not post at all. Some would argue that someone like me should not post Hot Takes at all. Anyway, in true lazy fashion here is ChatGPT on the pros and cons:
Pros:
Encourages diverse perspectives and stimulates debate.
Can attract more engagement and interest from users.
Provides a platform for expressing unconventional or controversial ideas.
Fosters a culture of intellectual curiosity and open discourse within the community.
Cons:
May lead to increased polarization and conflict within the community.
Risk of spreading misinformation or poorly researched opinions.
Could divert attention from more rigorous and evidence-based discussions.
Potential for reputational damage if controversial opinions are associated with the forum.
TL;DR: Someone should probably write a grant to produce a spreadsheet/dataset of past instances where people claimed a new technology would lead to societal catastrophe, with variables such as “multiple people working on the tech believed it was dangerous.”
Slightly longer TL;DR: Some AI risk skeptics are mocking people who believe AI could threaten humanity’s existence, saying that many people in the past predicted doom from some new tech. There is seemingly no dataset which lists and evaluates such past instances of “tech doomers.” It seems somewhat ridiculous* to me that nobody has grant-funded a researcher to put together a dataset with variables such as “multiple people working on the technology thought it could be very bad for society.”
*Low confidence: could totally change my mind
———
I have asked multiple people in the AI safety space if they were aware of any kind of “dataset for past predictions of doom (from new technology)”, but have not encountered such a project. There have been some articles and arguments floating around recently such as “Tech Panics, Generative AI, and the Need for Regulatory Caution”, in which skeptics say we shouldn’t worry about AI x-risk because there are many past cases where people in society made overblown claims that some new technology (e.g., bicycles, electricity) would be disastrous for society.
While I think it’s right to consider the “outside view” on these kinds of things, I think that most of these claims 1) ignore examples of where there were legitimate reasons to fear the technology (e.g., nuclear weapons, maybe synthetic biology?), and 2) imply the current worries about AI are about as baseless as claims like “electricity will destroy society,” whereas I would argue that the claim “AI x-risk is >1%” stands up quite well against most current scrutiny.
(These claims also ignore the anthropic argument/survivor bias—that if they ever were right about doom we wouldn’t be around to observe it—but this is less important.)
I especially would like to see a dataset that tracks things like “were the people warning of the risks also the people who were building the technology?” More generally, some measurement of “analytical rigor” also seems really important, e.g., “could the claims have stood up to an ounce of contemporary scrutiny (i.e., without the benefit of hindsight)?”
Absolutely seems worth spending up to $20K to hire researchers to produce such a spreadsheet within the next two-ish months… this could be a critical time period, where people are more receptive to new arguments/responses…?
Just saw this now, after following a link to another comment.
You have almost given me an idea for a research project. I would run the research honestly and report the facts, but my in-going guess is that survivor bias is a massive factor, contrary to what you say here. And that in most cases, the people who believed it could lead to catastrophe were probably right to be concerned. A lot of people have the Y2K bug mentality, in which they didn’t see any disaster and so concluded that it was all a false-alarm, rather than the reality which is that a lot of people did great work to prevent it.
If I look at the different x-risk scenarios the public is most aware of:
Nuclear annihilation—this is very real. As is nuclear winter.
Climate change. This is almost the poster-child for deniers, but in fact there is as yet no reason to believe that the doom-saying predictions are wrong. Everything is going more or less as the scientists predicted, if anything, it’s worse. We have just underestimated the human capacity to stick our heads in the ground and ignore reality*.
Pandemic. Some people see covid as proof that pandemics are not that bad. But we know that, for all the harm it wrought, covid was far from the worst-case. A bioweapon or a natural pandemic.
AI—the risks are very real. We may be lucky with how it evolves, but if we’re not, it will be the machines who are around to write about what happened (and they will write that it wasn’t that bad …)
Etc.
My unique (for this group) perspective on this is that I’ve worked for years on industrial safety, and I know that there are factories out there which have operated for years without a serious safety incident or accident—and someone working in one of those could reach the conclusion that the risks were exaggerated, while being unaware of cases where entire factories or oil-rigs or nuclear power plants have exploded and caused terrible damage and loss of life.
Before I seriously start working on this (in the event that I find time), could you let me know if you’ve since discovered such a data-base?
*We humans are naturally very good at this, because we all know we’re going to die, and we live our lives trying not to think about this fact or desperately trying to convince ourselves of the existence of some kind of afterlife.
I’m not that focused on climate science, but my understanding is that this is a bit misleading in your context—that there were some scientists in the (90s/2000s?) who forecasted doom or at least major disaster within a few decades due to feedback loops or other dynamics which never materialized. More broadly, my understanding is that forecasting climate has proven very difficult, even if some broad conclusions (e.g., “the climate is changing,” “humans contribute to climate change”) have held up. Additionally, it seems that many engineers/scientists underestimated the pace of alternative energy technology (e.g., solar).
That aside, I would be excited to see someone work on this project, and I still have not discovered any such database.
I’m not sure. IMHO a major disaster is happening with the climate. Essentially, people have a false belief that there is some kind of set-point, and that after a while the temperature will return to that, but this isn’t the case. Venus is an extreme example of an Earth-like planet with a very different climate. There is nothing in physics or chemistry that says Earth’s temperature could not one day exceed 100 C.
It’s always interesting to ask people how high they think sea-level might rise if all the ice melted. This is an uncontroversial calculation which involves no modelling—just looking at how much ice there is, and how much sea-surface area there is. People tend to think it would be maybe a couple of metres. It would actually be 60 m (200 feet). That will take time, but very little time on a cosmic scale, maybe a couple of thousand years.
Right now, if anything what we’re seeing is worse than the average prediction. The glaciers and ice sheets are melting faster. The temperature is increasing faster. Etc. Feedback loops are starting to be powerful. There’s a real chance that the Gulf Stream will stop or reverse, which would be a disaster for Europe, ironically freezing us as a result of global warming …
Among serious climate scientists, the feeling of doom is palpable. I wouldn’t say they are exaggerating. But we, as a global society, have decided that we’d rather have our oil and gas and steaks than prevent the climate disaster. The US seems likely to elect a president who makes it a point of honour to support climate-damaging technologies, just to piss off the scientists and liberals.
I’ll be blunt, remarks like these undermine your credibility. But regardless, I just don’t have any experience or contributions to make on climate change, other than re-emphasizing my general impression that, as a person who cares a lot about existential risk and has talked to various other people who also care a lot about existential risk, there seems to be very strong scientific evidence suggesting that extinction is unlikely.
Anyone else ever feel a strong discordance between emotional response and cognitive worldview when it comes to EA issues?
Like emotionally I’m like “save the animals! All animals deserve love and protection and we should make sure they can all thrive and be happy with autonomy and evolve toward more intelligent species so we can live together in a diverse human animal utopia, yay big tent EA…”
But logically I’m like “AI and/or other exponential technologies are right around the corner and make animal issues completely immaterial. Anything that detracts from progress on that is a distraction and should be completely and deliberately ignored. Optimally we will build an AI or other system that determines maximum utility per unit of matter, possibly including agency as a factor and quite possibly not, so that we can tile the universe with sentient simulations of whatever the answer is.”
OR, a similar discordance between what was just described and the view that we should also co-optimize for agency, diversity of values and experience, fun, decentralization, etc., EVEN IF that means possibly locking in a state of ~99.9999+percent of possible utility unrealized.
Very frustrating, I usually try to push myself toward my rational conclusion of what is best with a wide girth for uncertainty and epistemic humility, but it feels depressing, painful, and self-de-humanizing to do so.
You may know will.i.am as the frontman of The Black Eyed Peas, but his interests beyond music have taken him down a fascinating path at the intersection of creativity and technology. In a recent podcast, he discussed his thoughts on AI and the creative process with host Adam Grant.
Some key points:
Adam notes that the most creative people are often the worst at explaining their ideas, because creativity requires divergent, non-linear thinking while explanation favors convergence and linearity.
The podcast features some impressive live wordplay and freestyling from will.i.am. His verbal creativity is on full display.
Interestingly, will.i.am now hosts a radio show with an AI co-host named Fiona. He shares his hopes about the future of AI in entertainment and creativity.
will.i.am and Adam debate what AI can and can’t do for human creativity. No definitive answers, but a great discussion nonetheless.
I didn’t previously associate will.i.am with the AI scene, but he clearly has an innovative and forward-thinking perspective to share. Worth a listen for anyone interested in the intersection of AI and creativity.
Listen here | Read the transcript here
You can now import posts directly from Google docs
Plus, internal links to headers[1] will now be mapped over correctly. To import a doc, make sure it is public or shared with “eaforum.posts@gmail.com″[2], then use the widget on the new/edit post page:
Importing a doc will create a new (permanently saved) version of the post, but will not publish it, so it’s safe to import updates into posts that are already published. You will need to click the “Publish Changes” button to update the live post.
Everything that previously worked on copy-paste[3] will also work when importing, with the addition of internal links to headers (which only work when importing).
There are still a few things that are known not to work:
Nested bullet points(these are working now)Cropped images get uncropped
Bullet points in footnotes (these will become separate un-bulleted lines)
Blockquotes (there isn’t a direct analog of this in Google docs unfortunately)
There might be other issues that we don’t know about. Please report any bugs or give any other feedback by replying to this quick take, you can also contact us in the usual ways.
Appendix: Version history
There are some minor improvements to the version history editor[4] that come along with this update:
You can load a version into the post editor without updating the live post, previously you could only hard-restore versions
The version that is live[5] on the post is shown in bold
Here’s what it would look like just after you import a Google doc, but before you publish the changes. Note that the latest version isn’t bold, indicating that it is not showing publicly:
Previously the link would take you back to the original doc, now it will take you to the header within the Forum post as you would expect. Internal links to bookmarks (where you link to a specific text selection) are also partially supported, although the link will only go to the paragraph the text selection is in
Sharing with this email address means that anyone can access the contents of your doc if they have the url, because they could go to the new post page and import it. It does mean they can’t access the comments at least
I’m not sure how widespread this knowledge is, but previously the best way to copy from a Google doc was to first “Publish to the web” and then copy-paste from this published version. In particular this handles footnotes and tables, whereas pasting directly from a regular doc doesn’t. The new importing feature should be equal to this publish-to-web copy-pasting, so will handle footnotes, tables, images etc. And then it additionally supports internal links
Accessed via the “Version history” button in the post editor
For most intents and purposes you can think of “live” as meaning “showing publicly”. There is a bit of a sharp corner in this definition, in that the post as a whole can still be a draft.
To spell this out: There can be many different versions of a post body, only one of these is attached to the post, this is the “live” version. This live version is what shows on the non-editing view of the post. Independently of this, the post as a whole can be a draft or published.
Thanks, Will!
2 nitpicks:
The title of the doc is imported as the 1st paragraph of the EA Forum post, instead of being imported as the title.
Blank lines without spacing before and after in the doc are not imported, although I personally think this is a feature! Blank lines without spacing before and after in the footnotes of the doc are imported, but I would rather not have them imported.
Thanks for reporting!
I’ll think about how we could handle this one better. It’s tricky because the doc itself as a title, and then people often rewrite the title as a heading inside the doc, so there isn’t an obvious choice for what to use as the title. But it may be true that the heading case is a lot more common so we should make that the default.
That was indeed intended as a feature, because a lot of people use blank lines as a paragraph break. We can add that to footnotes too.
I’ll set a reminder to reply here when we’ve done these.
Although I’m no expert, maybe next you could try to be able to convert/download posts into google docs? Super cool btw.
I have thought this might be quite useful to do. I would guess (people can confirm/correct me) a lot of people have a workflow like:
Edit post in Google doc
Copy into Forum editor, make a few minor tweaks
Realise they want to make larger edits, go back to the Google doc to make these, requiring them to either copy over or merge together the minor tweaks they have made
For this case being able to import/export both ways would be useful. That said it’s much harder to do the other way (we would likely have to build up the Google doc as a series of edits via the api, whereas in our case we can handle the whole post exported as html quite naturally), so I wouldn’t expect us to do this in the near future unfortunately.
Omg what, this is amazing(though nested bullets not working does seem to make this notably less useful). Does it work for images?
Ok nested bullets should be working now :)
Yep images work, and agree that nested bullet points are the biggest remaining issue. I’m planning to fix that in the next week or two.
Edit: Actually I just noticed the cropping issue, images that are cropped in google docs get uncropped when imported. That’s pretty annoying. There is no way to carry over the cropping but we could flag these to make sure you don’t accidentally submit a post with the uncropped images.
Oh wow actually so happy about this, had definitely been an annoying challenge getting formatting right!
It sounds like the Future of Humanity Institute may be permanently shut down.
Background: FHI went on a hiring freeze/pause back in 2021 with the majority of staff leaving (many left with the spin-off of the Centre for the Governance of AI) and moved to other EA organizations. Since then there has been no public communication regarding its future return, until now…
The Director, Nick Bostrom, updated the bio section on his website with the following commentary [bolding mine]:
This language suggests that FHI has officially closed. Can anyone at Trajan/Oxford confirm?
Also curious if there is any project in place to conduct a post mortem on the impact FHI has had on the many different fields and movements? I think it’s important to ensure that FHI is remembered as a significant nexus point for many influential ideas and people who may impact the long term.
In other news, Bostrom’s new book “Deep Utopia” is available for pre-order (coming March 27th).
Further evidence: The 80,000 Hours website footer no longer mentions FHI. Until February 2023, the footer contained the following statement:
By February 21, that statement was replaced with a paragraph simply stating that 80k is part of EV. The references to GPI, CEA and GWWC were also removed:
Yeah, it looks like the FHI website’s news section hasn’t been updated since 2021. Nor are there any publications since 2021.
I like the “we decided to shut down our charity because it wasn’t very effective post” for the obvious reasons, but I wonder once you control for “is rich / comes from a rich family” and “doesn’t have family that depends on them” how that metric gets affected.
I think it is still good praise the decision in general, but unless I know the backgrounds of the people doing it I can’t heap too much praise on them.
If I’m understanding this concern correctly, it’s along the lines of: “they’re not making a financial sacrifice in shutting down, so it’s less praiseworthy than it otherwise would be”.
Just to clarify, charity founders (at least CE ones) take a pay cut to start their charity—they would earn more if working for other EA organizations as employees, and much more if in tech/finance/consulting/careers that typical of people with oxbridge/ivy/etc education levels. The financial sacrifice was already made when starting the charity, and if anything, quitting is actually better for you financially.
I’m confused about where you’re going with this. Why would the founders’ personal financial situation substantially affect how we viewed their decision?
The government’s sentencing memorandum for SBF is here; it is seeking a sentence of 40-50 years.
As typical for DOJ in high-profile cases, it is well-written and well-done. I’m not just saying that because it makes many of the same points I identified in my earlier writeup of SBF’s memorandum. E.g., p. 8 (“doubling down” rather than walking away from the fraud); p. 43 (“paid in full” claim is highly misleading) [page cites to numbers at bottom of page, not to PDF page #].
EA-adjacent material: There’s a snarky reference to SBF’s charitable donations “(for which he still takes credit)” (p. 2) in the intro, and the expected hammering of SBF’s memo for taking credit for attempting to take credit for donations paid with customer money (p. 95). There’s a reference to SBF’s “idiosyncratic . . . beliefs around altruism, utilitarianism, and expected value” (pp. 88-89). This leads to the one surprise theme (for me): the need to incapacitate SBF from committing additional crimes (pp. 87, 90). Per the feds, “the defendant believed and appears still to believe that it is rational and necessary for him to take great risks including imposing those risks on others, if he determines that it will serve what he personally deems a worthy project or goal,” which contributes to his future dangerousness (p. 89).
For predictors: Looking at sentences where the loss was > $100MM and the method was Ponzi/misappropriation/embezzlement, there’s a 20-year, two 30-years, a bunch of 40-years, three 50-years, and three 100+-years (pp. 96-97).
Interesting item: The government has gotten about $3.45MM back from political orgs, and the estate has gotten back ~$280K (pp. 108-09). The proposed forfeiture order lists recipients, and seems to tell us which ones returned monies to the government (Proposed Forfeiture Order, pp. 24-43).
Life Pro Tip: If you are arrested by the feds, do not subsequently write things in Google Docs that you don’t want the feds to bring up at your sentencing. Jotting down the idea that “SBF died for our sins” as some sort of PR idea (p. 88; source here) is particularly ill-advised.
My Take: In Judge Kaplan’s shoes, I would probably sentence at the high end of the government’s proposed range. Where the actual loss will likely be several billion, and the loss would have been even greater under many circumstances, I don’t think a consequence of less than two decades’ actual time in prison would provide adequate general deterrence—even where the balance of other factors was significantly mitigating. That would imply a sentence of ~25 years after a prompt guilty plea. Backsolving, that gets us a sentence of ~35 years without credit for a guilty plea.
But the balance of other factors is aggravating, not mitigating. Stealing from lots of ordinary people is worse than stealing from sophisticated investors. Outright stealing by someone in a fiduciary role is worse than accounting fraud to manipulate stock prices. We also need to adjust upward for SBF’s post-arrest conduct, including trying to hide money from the bankruptcy process, multiple attempts at witness tampering, and perjury on the stand. Stacking those factors would probably take me over 50 years, but like the government I don’t think a likely-death-in-prison sentence is necessary here.
In case you know this off-hand or it’s easy for you to get or point me in the right direction, do you know how they established SBF’s intent to misuse the billions in customer funds? What I got from Googling this didn’t seem very convincing, but I didn’t read court documents directly. (See also https://forum.effectivealtruism.org/posts/ggkiDAZowmuzqZrnX/is-anyone-else-still-confused-about-what-exactly-happened-at )
I’d look at pp. 5-12 of the linked sentencing memo for customers, pp. 15-18 for investors/lenders for the government’s statement of the offense conduct. The jury merely utters guilty / not guilty on each count, it does not provide detailed findings of fact. Judge Kaplan heard all the evidence as he presided at trial, and can rely on his own factual findings at sentencing under a more-likely-than-not standard. Of course, that is just a summary; Ellison alone testified for ~3 days.
Basically, SBF & FTX falsely represented that the customer assets were segregated from FTX’s own assets and would not be used by FTX. Yet Alameda regularly took large sums of money out of accounts holding FTX customer funds, and the “allow negative” feature allowed it to borrow ~unlimited money as well. This was not limited to funds made available to customers through the margin lending program.
At various points discussed on pp. 9-11, SBF directed Alameda to “borrow” more money from FTX despite knowing it was underwater at the time. For instance, at one point SBF directed Ellison “to use FTX customer funds to repay loans” (her words), despite knowing that Alameda was $11B in the hole and was already “borrowing” $13B in customer funds (p. 11). About $4.5B more in customer funds was used to pay Alameda’s lenders as a result (p. 11). In short, I don’t think the narrative presented in the linked post is backed up by the trial evidence.
Some related thoughts and questions:
NunoSempere points out that EA could have been structured in a radically different way, if the “specific cultural mileu” had been different. But I think this can be taken even further. I think that it’s plausible that if a few moments in the history of effective altruism had gone differently, the social makeup—the sort of people that make up the movement—and their axiological worldviews—the sorts of things they value—might have been radically different too.
As someone interested in the history of ideas, I’m fascinated by what our movement has that made it significantly different than the most likely counterfactual movements. Why is effective altruism the way it is? A number of interesting brief histories have been written about the history of EA (and longer pieces about more specific things like Moynihan’s excellent X-Risk) but I often feel that there are a lot of questions about the movement’s history, especially regarding tensions that seem to present themselves between the different worldviews that make up EA.
For example,
How much was it the individual “leaders” of EA who brought together different groups of people to create a big-tent EA, as opposed to the communities themselves already being connected? (Toby Ord says that he connected the Oxford GWWC/EA community to the rationality community, but people from both of these “camps” seem to be at Felicifia together in the late 2000s.)
When connecting the history of thought, there’s a tendency to put thinkers after one another in lineages as if they all read and are responding to those who came before them. Parfit lays the ground for longtermism in the the late 20th century in Reasons and Persons and Bostrom continues the work when presenting the idea of x-risk in 2001. Did Bostrom know of and expand upon Parfit’s work, or was Bostrom’s framing independent of that, based on risks discussed by the Extropians, Yudkowsky, SL4, etc? There (maybe) seems to be multiple discovery of early EA ideas in separate creation of the Oxford/GWWC community and GiveWell. Is something like that going on for longtermism/x-risk?
What would EA look like today without Yudkowsky? Bostrom? Karnofsky/Hassenfeld? MacAskill/Ord?
What would EA look like today without Dustin Moskovitz? Or if we had another major donor? (One with different priorities?)
What drove the “longtermist turn?” A shift driven by leaders or by the community?
A few interesting Yudkowsky (not be taken as current opinions, for historical purposes) quotes (see also Extropian Archaeology):
It’s fascinating to me that this is the reason that there’s a “rationality” community around today. (See also) What would EA look like without it? Would it really be any less rational? What does a transhumanisty non-AI-worried EA look like?—I feel like that’s what we might have had without Yudkowsky.
One last thing:
Feels kinda obnoxious to write a quick take along the lines of “I’m thinking about writing a post on X, does anyone actually give a sh*t? Otherwise I won’t write it.”
I just wanted to check, since I can’t place my finger on why it feels obnoxious but it certainly does.
I’ve loved seeing all the Draft Amnesty posts on the Forum so far! Some really great stuff has been posted (and I’ll highlight that when I write a retrospective)
Posting this quick take as a reminder that people who are considering posting for Draft Amnesty can run a draft past me for quick feedback. Just DM me.
Open Phil claims that campaigns to make more Americans go vegan and vegetarian haven’t been very successful. But does this analysis account for immigration?
If people who already live in the US are shifting their diets, but new immigrants skew omnivore, a simple analysis could easily miss the former shift because immigration is fairly large in the US.
Source of Open Phil claim at https://www.openphilanthropy.org/research/how-can-we-reduce-demand-for-meat/ :
Although the cited Gallup report doesn’t explicitly distinguish on immigrant status or ethnicity, it does say that “[a]lmost all segments of the U.S. population have similar percentages of vegetarians” while noting a larger difference in marital status.
Even if one assumes that almost no immigrants are vegetarian, the rate of immigration isn’t so high as to really move a low percentage very much. As of 2018, there were ~45M people in the US who were born in another country. [https://www.pewresearch.org/short-reads/2020/08/20/key-findings-about-u-s-immigrants/]
As a brief example with easyish math, 15M out of 300M = 5%; 15M out of 330M (adding 30M extra meat eaters) only drops it to ~4.5%. Addition of 30M non-v*gan immigrants would mask an 1,500,000 increase in the number of non-immigrant vegetarians (15M/300M = 5% = 16.5M/330M). Without the 30M immigrants, the vegetarian population would have risen from 5% to 16.5M/300M = 5.5%. Given that the assumption that no immigrants are vegetarian is unrealistic, this shows that adding a good number of meat-eaters to the denominator doesn’t move the percentages much at all.
I am wondering whether people view EA vs. cause-specific field-building differently, especially about the Scout Mindset. My general thoughts are:
EA—Focuses on providing knowledge and evidence to facilitate the self-determination of individuals to rationally weigh up the evidence provided to decide on updating beliefs to inform actions wherever they may go. Scout Mindset is intrinsically valuable to provide flexibility and to update beliefs and work on the beliefs that individuals hold.
Field-Building—Focusing on convincing people that this is a cause area worth working on and will have a significant impact; less focus on individual thoughts based on the strength of the arguments and evidence field-builders already possess. Scout Mindset is instrumentally valuable to update and work on the beliefs that field-builders hold.
Argument for Instrumental value:
A more instrumental perspective is that it is much easier to ask someone to understand one thing and act on it rather than understand many things and struggle to act on any, which may be counterfactually more impactful.
Argument for Intrinsic value:
By focusing on the intrinsic value you’re measuring for the internal change process that occurs in EA to see and understand the reason behind different cultural shifts across time with specific emphasis on the potential for value-drift.
The core difference between the two, as I see it, is whether the community builder focuses on promoting the individual or the cause. However, this may be an oversimplification or unfair misrepresentation and I am keen to hear the community’s views.
As part of an AMA I put on X, I was asked for my “top five EA hot takes”. If you’ll excuse the more X-suited tone and spiciness, here they are:
1. OpenAI, Anthropic (and to a lesser extent DeepMind) were the worst cases of Unilateralists Curse of all time. EAs love to discourage enthusiastic newcomers by warning to not do “net negative” unilateralist actions (i.e. don’t start new projects in case they crowd out better, more “well thought through” projects in future, with “more competent” people doing them), but nothing will ever top the monumental unilateralist curse fuck up that was supporting Big AGI in it’s beginnings.
2. AI Safety is nothing without a Pause. Too many EAs are stuck in the pre-GPT-4 paradigm of maxing research, when it’ll all be for nothing unless we get a Pause first. More EAs should switch to Notkilleveryoneism/PauseAI/StopAGI.
3. EA is too elitist. We should be triaging the world’s problems like crazy, and the top 1-2% of people are more than capable of that (most jobs that need doing in EA don’t require top 0.1%).
4. EA is too PR focused—to the point where it actually backfires spectacularly and now there is lots of bad press [big example: SBF’s bad character being known about but not addressed].
5. Despite all it’s flaws, EA is good (and much better than the alternatives in most cases).
Could you explain why you think ‘too much focus being placed on PR’ resulted in bad press?
Perhaps something like: because people were worried about harming SBF’s public reputation they didn’t share their concerns with others, and thus the community as a whole wasn’t able to accurately model his character and act appropriately?
More like, some people did share their concerns, but those they shared them with didn’t do anything about it (because of worrying about bad PR, but also maybe just as a kind of “ends justify the means” thing re his money going to EA. The latter might actually have been the larger effect.).
Ah ok—I guess I would phrase it as ‘not doing anything about concerns because they were too focused on short-term PR’.
I would phrase it this way because, in a world where EA had been more focused on PR, I think we would have been less likely to end up with a situation like SBF (because us having more of a focus on PR would have resulted in doing a better job of PR).
Do you think there is tension between 2 and 4 insofar as mechanisms to get a pause done may rely strongly on public support?
These all seem good topics to flesh out further! Is 1 still a “hot take” though? I thought this was pretty much the consensus here at this point?
Maybe half the community sees it that way. But not the half with all the money and power it seems. There aren’t (yet) large resources being put into playing the “outside game”. And there hasn’t been anything in the way of EA leadership (OpenPhil, 80k) admitting the error afaik.
Seems pretty dependent on how seriously you take some combination of AI x-risk in general, the likelihood that the naïve scaling hypothesis holding (if it even holds at all), and what the trade-off between empirical/theoretical work on AI Safety is no?
Regarding 2 - Hammers love Nails. EAs as Hammers, love research, so they bias towards seeing the need for more research (after all, it is what smart people do). Conversely, EAs are less likely (personality-wise) to be comfortable with advocacy and protests (smart people don’t do this). It is the wrong type of nail.
Although what you said might be part of the explanation for why many EAs focus on alignment or governance research rather than pause advocacy, I think the bigger part is that many EAs think that pause advocacy isn’t as good as research. See, e.g., some of these posts.
See all my comments and replies on the anti-pause posts. I don’t think any of the anti-pause arguments stand up if you put significant weight on timelines being short and p(doom) high (and viscerally grasp that yes, that means your own life is in danger, and those of your friends and family too, in the short term! It’s no longer just an abstract concern!).
Yes, my guess is they (like most people!) are motivated by things they’re (1) good at (2) see as high status.
My guess is that many EAs would find protesting cringy and/or awkward!
[Question] How should we think about the decision relevance of models estimating p(doom)?
(Epistemic status: confused & dissatisfied by what I’ve seen published, but haven’t spent more than a few hours looking. Question motivated by Open Philanthropy’s AI Worldviews Contest; this comment thread asking how OP updated reminded me of my dissatisfaction. I’ve asked this before on LW but got no response; curious to retry, hence repost)
To illustrate what I mean, switching from p(doom) to timelines:
The recent post AGI Timelines in Governance: Different Strategies for Different Timeframes was useful to me in pushing back against Miles Brundage’s argument that “timeline discourse might be overrated”, by showing how choice of actions (in particular in the AI governance context) really does depend on whether we think that AGI will be developed in ~5-10 years or after that.
A separate takeaway of mine is that decision-relevant estimation “granularity” need not be that fine-grained, and in fact is not relevant beyond simply “before or after ~2030″ (again in the AI governance context).
Finally, that post was useful to me in simply concretely specifying which actions are influenced by timelines estimates.
Question: Is there something like this for p(doom) estimates? More specifically, following the above points as pushback against the strawman(?) that “p(doom) discourse, including rigorous modeling of it, is overrated”:
What concrete high-level actions do most alignment researchers agree are influenced by p(doom) estimates, and would benefit from more rigorous modeling (vs just best guesses, even by top researchers e.g. Paul Christiano’s views)?
What’s the right level of granularity for estimating p(doom) from a decision-relevant perspective? Is it just a single bit (“below or above some threshold X%”) like estimating timelines for AI governance strategy, or OOM (e.g. 0.1% vs 1% vs 10% vs >50%), or something else?
I suppose the easy answer is “the granularity depends on who’s deciding, what decisions need making, in what contexts”, but I’m in the dark as to concrete examples of those parameters (granularity i.e. thresholds, contexts, key actors, decisions)
e.g. reading Joe Carlsmith’s personal update from ~5% to >10% I’m unsure if this changes his recommendations at all, or even his conclusion – he writes that “my main point here, though, isn’t the specific numbers… [but rather that] here is a disturbingly substantive risk that we (or our children) live to see humanity as a whole permanently and involuntarily disempowered by AI systems we’ve lost control over”, which would’ve been true for both 5% and 10%
Or is this whole line of questioning simply misguided or irrelevant?
Some writings I’ve seen gesturing in this direction:
harsimony’s argument that Precise P(doom) isn’t very important for prioritization or strategy (“identifying exactly where P(doom) lies in the 1%-99% range doesn’t change priorities much”) amounts to the ‘single bit granularity’ answer
Carl Shulman disagrees, but his comment (while answering my 1st bullet point) isn’t clear in the way the different AI gov strategies for different timelines post is, so I’m still left in the dark – to (simplistically) illustrate with a randomly-chosen example from his reply and making up numbers, I’m looking for statements like “p(doom) < 2% implies we should race for AGI with less concern about catastrophic unintended AI action, p(doom) > 10% implies we definitely shouldn’t, and p(doom) between 2-10% implies reserving this option for last-ditch attempts”, which he doesn’t provide
Froolow’s attempted dissolution of AI risk (which takes Joe Carlsmith’s model and adds parameter uncertainty – inspired by Sandberg et al’s Dissolving the Fermi paradox – to argue that low-risk worlds are more likely than non-systematised intuition alone would suggest)
Froolow’s modeling is useful to me for making concrete recommendations for funders, e.g. (1) “prepare at least 2 strategies for the possibility that we live in one of a high-risk or low-risk world instead of preparing for a middling-ish risk”, (2) “devote significantly more resources to identifying whether we live in a high-risk or low-risk world”, (3) “reallocate resources away from macro-level questions like ‘What is the overall risk of AI catastrophe?’ towards AI risk microdynamics like ‘What is the probability that humanity could stop an AI with access to nontrivial resources from taking over the world?’”, (4) “When funding outreach / explanations of AI Risk, it seems likely it would be more convincing to focus on why this step would be hard than to focus on e.g. the probability that AI will be invented this century (which mostly Non-Experts don’t disagree with)”. I haven’t really seen any other p(doom) model do this, which I find confusing
I’m encouraged by the long-term vision of the MTAIR project “to convert our hypothesis map into a quantitative model that can be used to calculate decision-relevant probability estimates”, so I suppose another easy answer to my question is just “wait for MTAIR”, but I’m wondering if there’s a more useful answer to the “current SOTA” than this. To illustrate, here’s (a notional version of) how MTAIR can help with decision analysis, cribbed from that introduction post:
This question was mainly motivated by my attempt to figure out what to make of people’s widely-varying p(doom) estimates, e.g. in the appendix section of Apart Research’s website, beyond simply “there is no consensus on p(doom)”. I suppose one can argue that rigorous p(doom) modeling helps reduce disagreement on intuition-driven estimates by clarifying cruxes or deconfusing concepts, thereby improving confidence and coordination on what to do, but in practice I’m unsure if this is the case (reading e.g. the public discussion around the p(doom) modeling by Carlsmith, Froolow, etc), so I’m not sure I buy this argument, hence my asking for concrete examples.
It’s possible that investing right and gaming prediction markets[1], something in which EAs have an advantage (this also means that money gained through prediction markets is likely also money that other EAs lost, however), is potentially an easier, more cost-effective, and time-effective way to double your impact when donating to charities.
In addition, perhaps charities should try to multiply the amount of money they have to do good with, using methods like these.
I think acting on the margins is still very underrated. For e.g. I think 5x the amount of advocacy for a Pause on capabilities development of frontier AI models would be great. I also think in 12 months time it would be fine for me to reevaluate this take and say something like ‘ok that’s enough Pause advocacy’.
Basically, you shouldn’t feel ‘locked in’ to any view. And if you’re starting to feel like you’re part of a tribe, then that could be a bad sign you’ve been psychographically locked in.
Seeing the drama with the NIST AI Safety Institute and Paul Christiano’s appointment and this article about the difficulty of rigorously/objectively measuring characteristics of generative AI, I figured I’d post my class memo from last October/November.
The main point I make is that NIST may not be well suited to creating measurements for complex, multi-dimensional characteristics of language models—and that some people may be overestimating the capabilities of NIST because they don’t recognize how incomparable the Facial Recognition Vendor Test is to this situation of subjective metrics for GenAI and they don’t realize NIST arguably even botched MNIST (which was actually produced by Yann LeCun by recompiling NIST’s datasets). Moreover, government is slow, while AI is fast. Instead, I argue we should consider an alternative model such as federal funding for private/academic benchmark development (e.g., prize competitions).
I wasn’t sure if this warranted a full post, especially since it feels a bit late; LMK if you think otherwise!
There are some major differences with the type of standards that NIST usually produces. Perhaps the most obvious is that a good AI model can teach itself to pass any standardised test. A typical standard is very precisely defined in order to be reproducible by different testers. But if you make such a clear standard test for an LLM, it would, say, be a series of standard prompts or tasks, which would be the same no matter who typed them in. But in such a case, the model just trains itself on how to answer these prompts, or follows the Volkswagen model of learning how to recognize that it’s being evaluated, and to behave accordingly, which won’t be hard if the testing questions are standard.
So the test tells you literally nothing useful about the model.
I don’t think NIST (or anyone outside the AI community) has experience with the kind of evals that are needed for models, which will need to be designed specifically to be unlearnable. The standards will have to include things like red-teaming in which the model cannot know what specific tests it will be subjected to. But it’s very difficult to write a precise description of such an evaluation which could be applied consistently.
In my view this is a major challenge for model evaluation. As a chemical engineer, I know exactly what it means to say that a machine has passed a particular standard test. And if I’m designing the equipment, I know exactly what standards it has to meet. It’s not at all obvious how this would work for an LLM.
I would be quite interested to hear more about what you’re saying re MNIST and the facial recognition vendor test
Sure! (I just realized the point about the MNIST dataset problems wasn’t fully explained in my shared memo, but I’ve fixed that now)
Per the assessment section, some of the problems with assuming that FRVT demonstrates NIST’s capabilities for evaluation of LLMs/etc. include:
Facial recognition is a relatively “objective” test—i.e., the answers can be linked to some form of “definitive” answer or correctness metric (e.g., name/identity labels). In contrast, many of the potential metrics of interest with language models (e.g., persuasiveness, knowledge about dangerous capabilities) may not have a “definitive” evaluation method, where following X procedure reliably evaluates a response (and does so in a way that onlookers would look silly to dispute).
The government arguably had some comparative advantage in specific types of facial image data, due to collecting millions of these images with labels. The government doesn’t have a comparative advantage in, e.g., text data.
The government has not at all kept pace with private/academic benchmarks for most other ML capabilities, such as non-face image recognition (e.g., Common Objects in Context) and LLMs (e.g., SuperGLUE).
It’s honestly not even clear to me whether FRVT’s technical quality truly is the “gold standard” in comparison to the other public training/test datasets for facial recognition (e.g., MegaFace); it seems plausible that the value of FRVT is largely just that people can’t easily cheat on it (unlike datasets where the test set is publicly available) because of how the government administers it.
For the MNIST case, I now have the following in my memo:
Some may argue this assumption was justified at the time because it required that models could “generalize” beyond the training set. However, popular usage appears to have favored MNIST’s approach. Additionally, it is externally unclear that one could effectively generalize from the handwriting of a narrow and potentially unrepresentative segment of society—professional bureaucrats—to high schoolers’, and the assumption that this would be necessary (e.g., due to the inability to get more representative data) seems unrealistic.
I just had a call with a young EA from Oyo State in Nigeria (we were connected through the excellent EA Anywhere), and it was a great reminder of how little I know regarding malaria (and public health in developing countries more generally). In a very simplistic sense: are bednets actually the most cost effective way to fight against malaria?
I’ve read a variety of books on the development economics canon, I’m a big fan of the use of randomized control trials in social science, I remember worm wars and microfinance not being so amazing as people thought and critiques of Tom’s Shoes. I was thrilled when I first read Poor Economics, and it opened my eyes to a whole new world. But I’m a dabbler, not an expert. I haven’t done fieldwork; I’ve merely read popular books. I don’t have advanced coursework in this area.
It was nice to be reminded of how little I actually know, and of how superficial general interest in a field is not the same as detailed knowledge. If I worked professionally in development economics I would probably be hyper aware of the gaps in my knowledge. But as a person who merely dabbles in development as an interest, I’m not often confronted with the areas about which I am completely ignorant, and thus there is something vaguely like a Dunning-Kruger effect. I really enjoyed hearing perspectives from someone that knows a lot more than I do.
QALY/$ for promoting zinc as a common cold intervention
Epistemic status: Fun speculation. I know nothing about public health, and grabbed numbers from the first source I could find for every step of the below. I link to the sources which informed my point estimates.
Here’s my calculation broken down into steps:
Health-related quality of life effect for one year of common cold −0.2
Common cold prevalence in the USA 1.2/yr
Modally 7 days of symptoms having −0.2
~1.5 million QALY burden per year when aggregated across the US population
This is the average of estimating from the above (1e6) with what I get (2e6) when deriving the US slice of the total DALY burden from global burden of disease data showing 3% global DALYs come from URI
There’s probably a direct estimate out there somewhere
50% probability the right zinc lozenges with proper dosing can prevent >90% of colds. This comes from here, here, and my personal experience of taking zinc lozenges ~10ish occasions.
15% best case adoption scenario, from taking a log-space mean of
Masks 5%
General compliance rate 10-90%
100,000 QALYs/year is my estimate for the expected value of taking some all-or-nothing action to promote zinc lozenges (without the possibility of cheaply confirming whether they work) which successfully changes public knowledge and medical advice to promote our best-guess protocol for taking zinc.
$35 million is my estimate for how much we should be willing to spend to remain competitive with Givewell’s roughly 1 QALY/$71. This assumes a 5 year effect duration. I have no idea how much such a thing would cost but I’d guess at most 1 OOM of value is being left on the table here, so I’m a bit less bullish on Zinc than I was before calculating.
EDIT: I calculated the cost of supplying the lozenges themselves. Going off these price per lozenge, this 5 year USA supply of lozenges costs ~35 million alone. Presumably this doesn’t need to hit the Givewell spending bar, but just US government spending on healthcare.
I’m seeing 0.25% globally and 0.31% for the US for URI in the GBD data, ~1 OOM lower (the direct figure for the US is 3.4e5, also ~1 OOM lower). What am I missing?
It feels like this is still a research problem needing larger scale trials. If the claims are true (i.e. the failures to achieve statistically significant results were due to not preparing and consuming lozenges in a particular way, rather than the successes being the anomalies) there are plenty of non-philanthropic entities (governments and employers and media as well as zinc supplement vendors) that would be incentivised to publicise more widely.
Is there a meta-analysis studying the effect size of this intervention? These seem unrealistically high to me.
Lifeextension cites this https://pubmed.ncbi.nlm.nih.gov/24715076/ claiming “The results showed that when the proper dose of zinc is used within 24 hours of first symptoms, the duration of cold miseries is cut by about 50%” I’d be interested if you do a dig through the citation chain. The lifeextension page has a number of further links.
That citation is retracted?
Here’s the Cochrane withdrawal notice.
Looks like they plagiarized from this paper, which found:
Good catch, thanks.