Marcel D

Karma: 1,910

Marcel D Jun 8, 2023, 1:31 AM
3 points
1 ∶ 0
on: Advice for Undergraduates Interested in US Policy
Epistemic status: writing fast and loose, but based on thoughts I’ve mulled over for a while given personal experience/struggles and discussions with other people. Thus, easy to misinterpret what I’m saying. Take with salt.
On the topic of educational choice, I can’t emphasize enough the importance of having legible hard skills such as language or, perhaps more importantly, quantitative skills. Perhaps the worst mistake I made in college was choosing to double major in both international studies and public policy, rather than adding a second major in econ or computer science. To my freshman self, this seemed to make sense: “I really want to do international security and like public policy, so this should demonstrate my enthusiasm and improve my policy-analysis skills.”
I’m somewhat skeptical that the second belief panned out (relative to majoring in econ), but the first clearly seems to have been misguided. Understanding what I do now about the application process, including how shallow/ineffective the process is at discerning capabilities and interest, how people will just exaggerate/BS their way into positions (through claims in their interviews and cover letters), that some positions will not consider people without STEM backgrounds (even when the skills can sometimes be learned prior to the job), AND how much of the process in some places relies on connections or prestige, it’s really clear that having legible hard skills is crucial. In contrast, you probably get rapidly-diminishing marginal returns with soft science degrees.

Marcel D Jun 7, 2023, 7:38 PM
2 points
0 ∶ 0
in reply to: Matt Beard’s comment on: AI Doom and David Hume: A Defence of Empiricism in AI Safety
Ultimately, I’ve found that the line between empirical and theoretical analysis is often very blurry, and if someone does develop a decent brightline to distinguish the two, it turns out that there are often still plenty of valuable theoretical methods, and some of the empirical methods can be very misleading.
For example, high-fidelity simulations are arguably theoretical under most definitions, but they can be far more accurate than empirical tests.
Overall, I tend to be quite supportive of using whatever empirical evidence we can, especially experimental methods when they are possible, but there are many situations where we cannot do this. (I’ve written more on this here: https://georgetownsecuritystudiesreview.org/2022/11/30/complexity-demands-adaptation-two-proposals-for-facilitating-better-debate-in-international-relations-and-conflict-research/ )

Marcel D Jun 4, 2023, 7:06 PM
2 points
0 ∶ 0
in reply to: Matt Beard’s comment on: AI Doom and David Hume: A Defence of Empiricism in AI Safety
I see. (For others’ reference, those two points are pasted below)
1. All knowledge is derived from impressions of the external world. Our ability to reason is limited, particularly about ideas of cause and effect with limited empirical experience.
2. History shows that societies develop in an emergent process, evolving like an organism into an unknown and unknowable future. History was shaped less by far-seeing individuals informed by reason than by contexts which were far too complex to realize at the time.
Overall, I don’t really know what to make of these. They are fairly vague statements, making them very liable to motte-and-bailey interpretations; they border on deepities, in my reading.
“All knowledge is derived from impressions of the external world” might be true in a trivially obvious sense that you often need at least some iota of external information to develop accurate beliefs or effective actions (although even this might be somewhat untrue with regard to biological instincts). However, it makes no clear claim about how much and what kind of “impressions from the external world” are necessary for “knowledge.”^[1] Insofar as the claim is that forecasts about AI x-risks are not “derived from impressions of the external world,” I think this is completely untrue. In such an interpretation, I question whether the principle even lives up to its own claims: what empirical evidence was this claim derived from?
The second claim suffers from similar problems in my view: I obviously wouldn’t claim that there have always been seers who could just divine the long-run future. However, insofar as it is saying that the future is so “unknowable” that people cannot reason about what actions in front of them are good, I also reject this: it seems obviously untrue with regards to, e.g., fighting Nazi Germany in WW2. Moreover, I would say that even if this has been true, that does not mean it will always be true, especially given the potential for value lock-in from superintelligent AI.
Overall, I agree that it’s important to be humble about our forecasts and that we should be actively searching for more information and methods to improve our accuracy, questioning our biases, etc. But I also don’t trust vague statements that could be interpreted as saying it’s largely hopeless to make decision-informing predictions about what to do in the short term to increase the chance of making the long-run future go well.
1. ^
  A term I generally dislike for its ambiguity and philosophical denotations (which IMO are often dubious at best).

Marcel D Jun 2, 2023, 9:14 PM
3 points
0 ∶ 0
in reply to: Christoph Hartmann 🔸’s comment on: Exponential AI takeoff is a myth
I haven’t looked very hard but the short answer is no, I’m not aware of any posts/articles that specifically address the idea of “methodological overhang” (a phrase I hastily made up and in hindsight realize may not be totally logical) as it relates to AI capabilities.
That being said, I have written about the possibility that our current methods of argumentation and communication could be really suboptimal, here: https://georgetownsecuritystudiesreview.org/2022/11/30/complexity-demands-adaptation-two-proposals-for-facilitating-better-debate-in-international-relations-and-conflict-research/

Marcel D Jun 1, 2023, 9:41 PM
3 points
0 ∶ 0
on: AI Doom and David Hume: A Defence of Empiricism in AI Safety
Is your claim just that people should generally “increase [their] error bars and widen [their] probability distribution”? (I was frustrated by the difficulty of figuring out what this post is actually claiming; it seems like it would benefit from a “I make the following X major claims...” TLDR.)
I probably disagree with your points about empiricism vs. rationalism (on priors that I dislike the way most people approach the two concepts), but I think I agree that most people should substantially widen their “error bars” and be receptive to new information. And it’s for precisely that reason which I feel decently confident in saying “most people whose risk estimates are very low (<0.5%) are significantly overconfident.” You logically cannot have extremely low probability estimates while also believing “there’s a >10% chance that in the future I will justifiably think there is a >5% chance of doom, but right now the evidence tells me the risk is <0.5%.”

Marcel D Jun 1, 2023, 5:52 PM
2 points
0 ∶ 1
on: Exponential AI takeoff is a myth
I think there is plenty of room for debate about what the curve of AI progress/capabilities will look like, and I mostly skimmed the article in about ~5 minutes, but I don’t think your post’s content justified the title (“exponential AI takeoff is a myth”). “Exponential AI takeoff is currently unsupported” or “the common narrative(s) for exponential AI takeoff is based on flawed premises” are plausible conclusions from this post (even if I don’t necessarily agree with them), but I think the original title would require far more compelling arguments to be justified.
(I won’t get too deep into this, but I think it’s plausible that there is significant “methodological overhang”: humans might just struggle to make progress in some fields of research—especially softer sciences and theory-heavy sciences—because principal-agent problems in research plague the accumulation of reliable knowledge through non-experimental methods.)

Marcel D May 30, 2023, 4:48 AM
24 points
7 ∶ 0
on: Harrison D’s Shortform
TL;DR: Someone should probably write a grant to produce a spreadsheet/dataset of past instances where people claimed a new technology would lead to societal catastrophe, with variables such as “multiple people working on the tech believed it was dangerous.”
Slightly longer TL;DR: Some AI risk skeptics are mocking people who believe AI could threaten humanity’s existence, saying that many people in the past predicted doom from some new tech. There is seemingly no dataset which lists and evaluates such past instances of “tech doomers.” It seems somewhat ridiculous* to me that nobody has grant-funded a researcher to put together a dataset with variables such as “multiple people working on the technology thought it could be very bad for society.”
*Low confidence: could totally change my mind
———
I have asked multiple people in the AI safety space if they were aware of any kind of “dataset for past predictions of doom (from new technology)”, but have not encountered such a project. There have been some articles and arguments floating around recently such as “Tech Panics, Generative AI, and the Need for Regulatory Caution”, in which skeptics say we shouldn’t worry about AI x-risk because there are many past cases where people in society made overblown claims that some new technology (e.g., bicycles, electricity) would be disastrous for society.
While I think it’s right to consider the “outside view” on these kinds of things, I think that most of these claims 1) ignore examples of where there were legitimate reasons to fear the technology (e.g., nuclear weapons, maybe synthetic biology?), and 2) imply the current worries about AI are about as baseless as claims like “electricity will destroy society,” whereas I would argue that the claim “AI x-risk is >1%” stands up quite well against most current scrutiny.
(These claims also ignore the anthropic argument/survivor bias—that if they ever were right about doom we wouldn’t be around to observe it—but this is less important.)
I especially would like to see a dataset that tracks things like “were the people warning of the risks also the people who were building the technology?” More generally, some measurement of “analytical rigor” also seems really important, e.g., “could the claims have stood up to an ounce of contemporary scrutiny (i.e., without the benefit of hindsight)?”
Absolutely seems worth spending up to $20K to hire researchers to produce such a spreadsheet within the next two-ish months… this could be a critical time period, where people are more receptive to new arguments/responses…?

Marcel D May 28, 2023, 9:35 PM
2 points
0 ∶ 0
in reply to: Jason’s comment on: OPTIC [Forecasting Comp] — Pilot Postmortem
TBH, I think that the time spent scoring rationales is probably quite manageable: I don’t think it should take longer than 30 person-minutes to decently judge each rationale (e.g., have three judges each spend 10 minutes evaluating each), maybe less? It might be difficult to have results within 1-2 hours if you don’t have that many judges, but probably it should be available by the end of the day.

To be clear, I was thinking that only a small number (no more than three, maybe just two) of the total questions should be “rationale questions.”

But definitely the information value of “do rationale scores correlate with performance” would be interesting! I’m not sure if the literature has ever done this (I don’t think I’ve encountered anything like that, but I haven’t actively searched for it)

Marcel D May 28, 2023, 3:47 PM
2 points
1 ∶ 2
in reply to: bgarfinkel’s comment on: AI strategy career pipeline
I would also strongly recommend having a version of the fellowship that aligns with US university schedules, unlike the current Summer fellowship!

Marcel D May 28, 2023, 3:45 PM
2 points
0 ∶ 1
in reply to: Denis ’s comment on: AI strategy career pipeline
Given the (accusations of) conflicts of interest in OpenAI’s calls for regulation of AI, I would be quite averse to relying on OpenAI for funding for AI governance

Marcel D May 28, 2023, 3:23 PM
6 points
0 ∶ 0
on: AI strategy career pipeline
This situation was somewhat predictable and avoidable, in my view. I’ve lamented the early-career problem in the past but did not get many ideas for how to solve it. My impression has been that many mid-career people in relevant organizations put really high premiums on “mentorship,” to the point that they are dismissive of proposals that don’t provide such mentorship.
There are merits to emphasizing mentorship, but the fact has been that there are major bottlenecks on mentorship capacity and this does little good for people who are struggling to get good internships. The result for me personally was at least ~4 internships that were not very relevant to AI governance, were not paid, and did not provide substantial career benefits (E.g., mentorship).
In summary, people should not let the perfect be the enemy of the good: I would have gladly taken an internship working on AI governance topics, even if I had almost no mentorship (and even if I had little or no compensation). I also think there are ways of substituting this with peer feedback/engagement.
I have multiple ideas for AI governance projects that are not so mentorship-dependent, including one pilot idea that, if it worked, could scale to >15 interns and entry-level researchers with <1 FTE experienced researcher in oversight. But I recognize that the ideas may not all be great (or at least their merits are not very legible). Unfortunately, we don’t seem to have a great ecosystem for sharing and discussing project ideas, at least if you aren’t well connected with people to provide feedback through your job or through HAIST/MAIA or other university groups.

Ultimately, I might recommend that someone aggregate a list of past programs and potential proposals, evaluate the importance of various goals and characteristics (e.g., mentorship, skill development, topic education, networks, credentials/CVs), and identify the key constraints/bottlenecks (e.g., funding vs. mentorship).

Marcel D May 28, 2023, 2:39 PM
2 points
1 ∶ 0
on: Rebooting AI Governance: An AI-Driven Approach to AI Governance
Although this article might interest me, it is also >30 minutes long and it does not appear to have any kind of TL;DR, and the conclusion is too vague for me to sample test the article‘s insights. It is really important to provide such a summary. For example, for an article of this type and length, I’d really like to know things like:
1. What would summarize as the main deficiency in current methods? Or “Many people/readers probably believe X, or just haven’t considered Y.”
2. What does this article argue instead? “I argue Y.”
3. Why does this all matter? “Y is significantly better than X.”

Marcel D May 28, 2023, 2:48 AM
4 points
0 ∶ 0
in reply to: Jason’s comment on: OPTIC [Forecasting Comp] — Pilot Postmortem
I was also going to recommend this, but I’ll just add an implementation idea (which IDK if I fully endorse): you could try to recruit a few superforecasters or subject-matter experts (SMEs) in given field to provide forecasts on the questions at the same time, then have a reciprocal scoring element (I.e., who came closest to the superforecasters’/SMEs’ forecasts). This is basically what was done in the 2022 Existential Risk Persuasion/Forecasting Tournament (XPT), which Philip Tetlock ran (and I participated in). IDK when the study results for that tournament will be out, and maybe it won’t recommend reciprocal scoring, but it definitely seems worth considering.

A separate idea (which again IDK if I fully endorse but was also in the XPT): have people provide dense rationales for a few big forecasts, then you can rate them on the merits of their rationales. (Yes, this involves subjectivity, but it’s not very different from speech and debate tournaments; the bigger problem could be the time required to review the rationales, but even this definitely seems manageable, especially if you provide a clear rubric, as is common in some competitive speech leagues.)

Marcel D May 15, 2023, 3:00 AM
4 points
0 ∶ 0
in reply to: Zach Stein-Perlman’s comment on: Proposal: Bulletin Board for Information Targets (e.g., in AI Policy)
I think the ask-a-question feature on EAF/LW partially fills the need.
I considered this largely insufficient for what I have in mind; I want a centralized hub that I can check (and/or that I expect other people will check), whereas the ask-a-question feature is too decentralized.
I expect most professional researchers don’t have a need for this, since they have a sufficiently strong network that they can ask someone who will know the answer or know who to ask.
Perhaps! I wouldn’t know, I’m mostly just a wannabe amateur who doesn’t have such a luxury. That being said, I would be surprised if it’s really that simple; I’ve heard mixed things about AI Policy research shops’ (e.g., GovAI’s) connections to the DC policy scene—although I have also heard that this might be improving at GovAI.

Marcel D May 14, 2023, 10:39 PM
3 points
0 ∶ 0
on: How much do markets value Open AI?
One particularly confusing fact is that OAI’s valuation appears to have gone from $14 billion in 2021 to $19 billion in 2023. Even ignoring anything about transformative AI, I would have expected that the success of ChatGPT etc. should have resulted in a more than a 35% increase.
I’ve frequently seen people fail to consider the fact that potential competitors (including borderline copycats) can significantly undermine the profitability of one company’s technology, even if the technology could generate substantial revenue. Are you taking this into account?

Marcel D May 10, 2023, 6:31 AM
5 points
1 ∶ 0
in reply to: JoshuaBlake’s comment on: The ‘Wild’ and ‘Wacky’ Claims of Karnofsky’s ‘Most Important Century’
Regarding
Then Karnofsky shouldn’t claim that he was arguing with “as much rigour as possibly”.
See my comment

Marcel D May 10, 2023, 6:29 AM
13 points
3 ∶ 2
on: The ‘Wild’ and ‘Wacky’ Claims of Karnofsky’s ‘Most Important Century’
Holden Karnofsky describes the claims of his “Most Important Century” series as “wild” and “wacky”, but at the same time purports to be in the mindset of “critically examining” such “strange possibilities” with “as much rigour as possible”. This emphasis is mine, but for what is supposedly an important piece of writing in a field that has a big part of its roots in academic analytic philosophy, it is almost ridiculous to suggest that this examination has been carried out with ‘as much rigour as possible’.
(Probably unimportant note: I don’t understand why you don’t link to the specific post(s) you’re referring to; could you please do that? I ask not simply to nitpick, but because I want to acknowledge that perhaps you’re not referring to https://www.cold-takes.com/most-important-century/ , in which case the main comment below may not apply.)
I fear you may have (seriously) mischaracterized/misquoted Holden.^[1] The following is the actual quote from that post:
But part of the mindset I’ve developed through GiveWell and Open Philanthropy is being open to strange possibilities, while critically examining them with as much rigor as possible. And after a lot of investment in examining the above thesis, I think it’s likely enough that the world urgently needs more attention on it.
By writing about it, I’d like to either get more attention on it, or gain more opportunities to be criticized and change my mind.
When I read the quotes from your opening paragraph, I developed the strong impression that Holden wrote something like “This series on ‘The Most Important Century’ is my attempt at critically examining some wild claims with as much rigor as possible.” JoshuaBlake also appears to have developed this impression, given that he commented “Then Karnofsky shouldn’t claim that he was arguing with “as much rigour as possibly”.”
However, this is not what Holden claimed, and I never interpreted The Most Important Century as an attempt to be “as rigorous as possible”—which should be implied given that this is a blog post series, not a series of academic papers.
Ultimately, I would like for OP to clarify the situation (e.g., are you just referring to a quote from a different post?) before I go into more detail.
1. ^
  Caveat: I can’t myself claim to know exactly what Holden intended, all I can say is the way you made me think Holden described his writing is very different from how I actually interpreted Holden’s claims both before and after I read this post.
What links here?
- Marcel D's comment on The ‘Wild’ and ‘Wacky’ Claims of Karnofsky’s ‘Most Important Century’ by carboniferous_umbraculum (May 10, 2023, 6:31 AM; 5 points)

How CISA can Support the Security of Large AI Models Against Theft [Grad School Assignment]

Marcel DMay 3, 2023, 3:36 PM

7 points

0 comments13 min readEA link

Marcel D May 2, 2023, 12:33 AM
2 points
0 ∶ 0
on: Global catastrophic risks law approved in the United States
Just to clarify, I don’t think that the legislation actually explicitly mentions “artificial intelligence,” right? Just “intentional or accidental threats arising from the use and development of emerging technologies.”

Marcel D May 1, 2023, 1:44 PM
4 points
0 ∶ 0
in reply to: freedomandutility’s comment on: What are work practices that you’ve adopted that you now think are underrated?
Learn the ‘proper’ way to type.
Is this just directed at people who still use hunt-and-peck, or is there some new “proper” way to type beyond normal [full hand / home key] typing?

Marcel D

How CISA can Sup­port the Se­cu­rity of Large AI Models Against Theft [Grad School As­sign­ment]

How CISA can Support the Security of Large AI Models Against Theft [Grad School Assignment]