pre-doc at Data Innovation & AI Lab
previously worked in options HFT and tried building a social media startup
founder of Northwestern EA club
Charlie_Guthmann
Karma: 973
Curious why METR. This is less METR specific and more about capabilities benchmarks: Doesn’t frontier capabilities benchmarking help accelerate ai development? I know they also do pure safety but in practice I feel like they have done more to push forth the agentic autmation race.
Tangential to the main post but I just read your shortform comment on less wrong and really agree. From a variety of different anecdotal experiences, I’m getting increasingly paranoid about trusting anyone here regarding ai opinions. How can I trust someone if 50% of their money is in Nvidia and snp 5 delta calls? if we pause AI they stand to lose so much. I don’t think this is a small amount of the community either.
( I have most of my money in snp stock and ~15% specifically in tech stocks so I’m still quite exposed to AI progress but significantly less than a lot of people here and I still find myself rooting for ai bull market sometimes because i’m selfish).
benchmarking model behavior seems increasingly hard to keep track off.
I think there are a bunch of separate layers that are being analyzed, and it’s increasingly complicated the degree to which they are separate.e.g.
level 1 → pre-trained only
level 2 → post-trained
level 3 → with ____ amount of inference (pro, high, low, thinking, etc.)
level 4 → with agentic scaffolding (Claude code, swe agent, codex)
level 5 → context engineering setups inside of your agentic repo (ACE, GEPA, ai scientists)
level 6 → The built digital environment (arguably could be included partially in level 4, stuff like api’s being crafted to be better for llms, workflows being re written to accomplish the same goal in a more verifiable way, ui’s that are more readable by llms).
In some sense you can boil all of this down to cost per run, like ARC, but you will miss important differences in model behavior on a fixed frontier.
https://jeremyberman.substack.com/p/how-i-got-the-highest-score-on-arc-agi-again
if you read J. Berman substack, you will see he uses existing llms to get his scores with an evolutionary scaffolding (hard to place this as being level 4⁄5). While I’m decently bitter lesson pilled, It seems plausible we will see proto-agis popping up that are heavily scaffolded before raw models reach that level (though also plausibly we might just see the really useful, generalizable scaffolds get consumed by the model soon thereafter). The behavior of j bermans system is going to be different than the first raw model to hit that score with no scaffolding and pose different threats at the same level of intelligence.
How politicized will AI get in the next (1,2,5) and what will those trends look like?
I think we are investing as a community more in policy/advocacy/research but the value of these things might be heavily a function of the politicization/toxicity of AI. Not a strong prior but I’d assume that like OMB/Think tanks get to write a really large % of the policy for boring non electorally dominant issues but have much less hard power when the issue at hand is like (healthcare, crime, immigration).
Hi wyatt there is some work on number 1 (although agreed needs a lot more). I keep posting this stack so maybe i should curate it into a sequence.
https://forum.effectivealtruism.org/s/wmqLbtMMraAv5Gyqn
https://forum.effectivealtruism.org/posts/W4vuHbj7Enzdg5g8y/two-tools-for-rethinking-existential-risk-2
https://forum.effectivealtruism.org/posts/zuQeTaqrjveSiSMYo/a-proposed-hierarchy-of-longtermist-concepts
https://forum.effectivealtruism.org/posts/fi3Abht55xHGQ4Pha/longtermist-especially-x-risk-terminology-has-biasing
https://forum.effectivealtruism.org/posts/wqmY98m3yNs6TiKeL/parfit-singer-aliens
https://forum.effectivealtruism.org/posts/zLi3MbMCTtCv9ttyz/formalizing-extinction-risk-reduction-vs-longtermismhttps://static1.squarespace.com/static/58e2a71bf7e0ab3ba886cea3/t/5a8c5ddc24a6947bb63a9bc9/1519148520220/Todd+Miller.evpsych+of+ETI.BioTheory.2017.pdf
https://forum.effectivealtruism.org/posts/mzT2ZQGNce8AywAx3
Very nice, thank you for writing.
It seems plausible that p(annual collapse risk) is in part a function of the N friction as well? I think you may cover some of this here but can’t really remember.
e.g. a society with less non-renewables will have to be more sustainable survive/grow → resulting population is somehow more value aligned → reduced annual collapse risk.
or on the other side
nukes still exist and we can still launch them → we have higher N friction in pre nuclear age in the next society → increased annual collapse risk.
(i have a bad habit of just adding slop onto models and think this isn’t at all something that need be in the scope of original post just a curiousity).
I think monied prediction markets are negative EV. The original reasons the CFTC were not allowing binary event contracts on most things are/were actually good reasons. It’s quite clear that our elected officials can get away with insider trading (and probably to a certain extent market manipulation). My intuition is that in the current admin I expect this behavior to increasingly not be punished and maybe actively encouraged. Importantly, insider trading on the existing financial instruments doesn’t really work. My take here is just that the marginal value of a piece of information is pretty low for traditionally financial markets, so while it might allow a high-tech research shop to beat the market it doesn’t even cover slippage/the lack of complex modeling done when in the hands of a dumb paper congress person. This is not the case for most/all binary event markets, where a single marginal piece of information can credibly flip probabilities from <1 → >99.
Insider trading, which I expect to be much more likely to happen at scale with these markets, is not that bad. It degrades the integrity of the prediction markets themselves and might lower volume but probably not that relevant for EA. However market manipulation could get really really bad in terms of EV (I think significant market manipulation is currently quite unlikely at the current levels of liquidity on these markets). In particular, the less likely something is to happen, the more incentive a prediction market provides for manipulating the market. And like in insider trading, I think market manipulation is going to be much more accessible and effective for generating alpha vs traditional markets.
low stakes example: Trump’s speech writer puts random phrase ( Lock x up, I hate x country, Sperm)medium stakes: Every election market is an assassination market, and a bounty for either candidate to drop out
High stakes: “will x leader/country bomb y”
Of course with the medium and high stakes examples, we are far from the liquidity on these markets to make the incentives provided to be worth seriously considering. But what if we 100x? what if e.g. mamdani could make 50 million dollars by dropping out instead of 500k? What if trump could make 20m by delaying a ceasefire a few days?To some extent we will just have to see how the enforcement of this stuff plays out as that will shape the incentives. It might seem obvious that the stuff above will be easily sorted out by the relevant federal bodies. Again, my intuition is that (1) I don’t actually expect this to be a huge priority right now and (2) the regulatory burden to properly regulate this stuff is ridiculously high (both in terms of just tracking everything that is happening and actually trying to apply the law to decide if various things constitute as “security fraud”).
It’s already hard for the SEC to enforce behavior the NYSE or CME, and that’s with mostly big institutional players who cooperate and record all their communications, etc. I’d have to imagine especially in the short term the vast majority of questionable or illegal behavior on prediction markets will slip through the cracks. The space of potential “securities fraud” is just so ridiculously big and confusing when you have 1000s of random binary event contracts.
Then you put that with the encouragement of gambling and incentivizing the spread of fake news, I just think we should be very very skeptical of going head first into this.
And the alternatives are pretty good! Non monied prediction markets like manifold and metaculus are huge improvements over previous epistemics, and to be fair the monied prediction markets are quite literally crowding them out. It’s not really fair to compare polymarkets accuracy to manifold, when a bunch of people who are on poly would probably use manifold if they made poly illegal (and that being said I’m pretty sure manifold doesn’t stack up to poorly but don’t have up to date stats). And some of the most important binary event contracts were already being traded on CME because they didn’t suffer from the issues I listed above.
Given the public good nature of prediction markets, the government should be quite willing to front $10s-100s of millions to improve their quality. And there are lots of ways to improve the markets without directly providing personal monetary incentives. This could mean improving the site ui/ux, creating a public leaderboard with some sort of recognization/award, improving the question statements/resolution, or even providing prizes that can be donated to a charity of the winners choice.
Yea my original framing was a little confused wrt the “vs” dichotomy you present in paragraph one, good shout. I guess I actually meant a little bit of each, though. My interpretation of the post is basically, (1) in so forth as we need to defeat powerful people or thought patterns we (ea or humans) haven’t proven it (2) it’s somewhat likely we will need to do this to create the world we want.
I.e. Given that future s-risk efforts are probably not going to be successful, current extinction-risk efforts are therefore also less useful.I am saying aligning AI is in the best interests of AI companies
If you define it in a specifically narrow AI Takeover way yes. Making sure it doesn’t allow a dictator to take power or gradually disempowerment scenarios, not really. Or to the extent that ensuring alignment requires slowing down progress.
Anyway mostly in agreement with your points/world, I definitely think we should be focusing on AI right now and I think that our goals and the AI companies/US gov are sufficiently aligned atm that we aren’t swimming up stream, but I resonante with OP that it would allievate some concerns if we actually racked up some hard fought politically unpopular battles before trying to steer the whole future.
It certainly seems possible (>1%) that in the next 2 US admins (current plus next) AI safety becomes so toxic that all the EA -adj ai safety people in the gov get purged and they stop listeing to most ai safety researchers. If this co-occurs with some sort of AI nationalization most of our TOC is cooked.
Many, if not most, longtermists believe we’re living near a hinge of history
Right but this requires believing the future will be better if humans survive. I take ops point as saying she doesn’t agree or is at least skeptical.
and a small group of AI companies adopting research that is in their best interests to use.
I think again, the point of OP is trying to make is we have very little proof of concept of getting people to go against their best interests. And so if doing what’s right isn’t in the ai companies best interest op wouldn’t believe we can get them to do what we think they should.
Yea it’s kinda like what they tell you not to do when building a startup. Every founder wants to build a beautiful, hyperscaling tech-heavy product before they have even confirmed that they have a few single real customers. In this case we are gonna write out our entire plans for the future of the universe before we win a single congressional seat.
Anyway this community isn’t set up to end something like veganism I don’t think. That requires large scale evangelizing and coalition building (unless we can solve it with tech). This movement is investing mostly into research and policy, I.E. we are betting on lots of the most important issues of our time not being politically toxic/salient. I think there is a lot of truth to the notion that most federal policy is written by people in think tanks and OMB—and that as long as it doesn’t piss off the electorate then the policymaker rather than the elected politician effectively gets to write the law.
But for stuff that obviously is in the mainstream overton window, e.g. veganism that is going to require large behavioral changes from ordinary citizens , you need an actual coalition of hard power.
Sure that’s a fair point. I’d guess I hope you would feel at least a little pushed in the direction after this thread that AIs need not take a similar route to humans to automating large amounts of our current work.
“novel idea” means almost nothing to me. A math proof is simply a->b. It doesn’t matter how you figure out a->b. If you can figure it out by reading 16 million papers and clicking them together that still counts. There are many ways to cook an egg.
https://x.com/slow_developer/status/1979157947529023997
I would bet a lot of money you are going to see exactly what I described for math in the next two years. The capabilities literally just exploded. It took us like 20 years to start using the lightbulb but you are expecting results from products that came out in the last few weeks/months.
I can also confidently say because I am working on a project with doctors that the work I described for clinical medicine is being tested and happening right now. It’s exact usefulness remains to be seen but like people are trying exactly what I described, there will be some lag as people need to learn how to use the tools best and then distribute their results.
Again, I don’t think most of this stuff was particularly useful with the tools available to use >1 year ago.
>Would an AI system that can’t learn new ideas from one example or a few examples count as AGI?
https://www.anthropic.com/news/skills
you are going to need to be a lot more precise in your definitions imo otherwise we are going to talk past each other.
i’m fleshing out nunos point a bit. Basically AI have so many systematic advantages with their cost/speed/seemless integration into the digital world that they can afford to be worse than humans at a variety of things and still automate (most/all/some) work. Just as a plane doesn’t need to flap it’s wings. Of course I wasn’t saying I solved automating the economy. I’m just showing you ways in which something lacking some top level human common sense/iq/whatever could replace still.
FWIW I basically disagree with every point you made in the summary. This mostly just comes from using these tools every day and getting utility out of them + seeing how fast they are improving + seeing how many different routes there are to improvement (i was quite skeptical a year ago, not so anymore). But I wanted to keep the argument contained and isolate a point of disagreement.
For example, how can AI automate the labour of scientists, philosophers, and journalists if it can’t understand novel ideas?
The bar is much lower because they are 100x faster and 1000x cheaper than me. They open up a bunch of brute forceable techniques in the same way that you can open up https://projecteuler.net/ solve many of eulers discoveries with little math knowledge but basic python and for loops.
Math → re read every arxiv paper → translate them all into lean → aggregate every open well specificied math problem → use the database of all previous learnings to see if you can chain chunks of previous problems together to solve.
clinical medicine → re-read every RCT ever done and comprehensively rank intervention effectiveness by disease → find cost data where available and rank the cost/qaly of all disease/intervention space
Econometrics → aggregate every natural experiment and instrumental variable ever used in an econometrics paper → think about other use cases for these tools → search if other use cases have available data → reapply the general theory of the original paper with the new data.
This is going to be rambly I don’t have the energy or time to organize my thoughts more atm. tldr is that I think the current uppercase EA movement is broken and not sure it can be fixed. I think there is room for a new uppercase EA movement that is bigger tent, lessed focused on intellectualism, more focused on organizing, and additionally has enough of a political structure that it is transparent who is in charge, by what mechanisms we hold bad behavior accountable, etc. I have been spending increasingly more of my time helping the ethical humanist society organize because I believe while lacking the intellectual core of EA it is more set up with all of the above and it feels easier to shift the intellectual core there than the entire informal structure of EA.
Fundamentally we are a mission with a community not a community with a mission. And that starts from the top (or lack of a clearly defined “top”).
We consistenly overvalue thought leaders and wealthy people and undervalue organizers. Can anyone here name an organizer that they know of just for organizing? I spent a huge amount of my time in college organizing northwestern EA. Of course I don’t regret it because i didn’t do it for myself (mostly) but did I get any status or reputation from my efforts? Not as far as I can tell. Am I wrong to think I’d have more respect if I had never organized but worked at jane street instead of organized + akuna ( a lower tier firm)?
Then after college I stayed in chicago, a city with nearly 1T GDP, with the second most quant traders in the united states, with a history of pushing things forward, and we don’t even have a storefront or church building?
repeating op here, but after a few years of engaging with EA, most people have hit diminishing returns on how new info can help them in their own career, and they will engage more with their own sub community.
How can we keep these people engaged and not just the new people and those whose life mission is cause prio? Build EA churches, develop litury/art/rituals that are independent of finding new intellectual breakthroughs, bond community members. Literally let’s just start by copying the most successful community builders ever and move from there.
Then you have the lack of accountability and transparency. Unless you have money, the best way to gain power in this community seems to me to be moving to SF/DC/oxford and living in a group house. There is no clear pipeline for having large sway over the current orthodoxy of most important cause areas. How would I explain to a 19 year in college how we push forward our ideas? I don’t think it would be fair to call this a pure meritocracy. There is a weird oligopoly of attention that is opaque and could be clarified and altered with a political system or at least by breaking up the location based monopolies.
We continue to basically be an applied utilitarian group that we have mis named (not that big of a deal, but I think we should be bigger tent anyway). Why are we a utilitarian group? Well normative concerns are not logical, so you can’t say merit won out.
Finally there is the bad behavior which we are completely powerless to, because we don’t have any political structure. The very fact that Will continues to hold so much sway and was never formally punished for ftx/twitter/political (if you don’t know what i mean when I say SBF political thing that proves my point even more) is a big part of why there is no trust (edit: i want to clarify that I think will is a good person and didn’t mean this as meaning I specifically don’t trust him rather just the institutions of our community). Currently we have leopold and mechanize, who are now AI accelerationists, who got way they are off the back of our movement, in very small part to the power i gave to the movement by organzing, and I have to watch these people behave in a way I think is bad and I can’t even cast a token vote expressing I would like to see them exhiled or punished.
As angry as people were years ago, WE DIDN’T CHANGE ANYTHING. How can I trust FTX won’t happen again?
Do you have any new thoughts on the probabilities/timelines of when he is going to invoke the insurrection act?
Marcus, ton of respect for your open-mindedness and prediction ability. Sort of parroting Lintz here but if you have the time, I would greatly appreciate if you could give some insight on how to improve the questions.
I understand that questions pertaining to 2028 and maybe even midterms suffer from long term market issues. So maybe we could create a chain of conditional markets? or at least some intermediate steps that we think are proxies and have a reasonable chance of occurring in the next few months?
Additionally, would you say you have updated your views since this comment chain?
Definitely coming in biased because of where my head is at, but I think building back the strength of small groups is a way to combat this and somewhat tractable. I like TT post below.
https://mathstodon.xyz/@tao/115259943398316677
Funnily enough EA has a similar problem (if you consider it a problem). Lack of structure or centralization disproportionately shifts power to the wealthy and already powerful.
I think this is https://www.lesswrong.com/w/coherent-extrapolated-volition this sort of?