Blog at The Good Blog https://thegoodblog.substack.com/
Nathan_Barnard
Karma: 866
This is really useful context!
This looks fantastic, thanks for putting it together!
Thanks for your comment Ceb.
I think my case for more focus on good statistical work when looking at governance is that when doing good statistical work on interventions, we often find very high degrees of variation in effect sizes that are enough to justify the extra work of the intervention. I’m personally very unsure of the effect size of changes in liability law, soft law, and various corporate governance interventions on accident rates.
There’s been lots of great case study/best guess/expert consensus work on these questions which I think is often great and I’m very happy exists, but leaves me with large uncertainties about effect sizes, and so on the current margin, I want more good statistical work.
I think the case for good statistical work on areas like degree of misuse risk and forecasting is stronger because I think that people’s takes on these questions are pretty grounded in quite theoretical arguments (unlike governance interventions which are much more empirically grounded) that I think would benefit a lot from more grounded statistical work.
Thanks for writing this Charlie, this was great
This is great Matt! I think I’d be also be interested in work trying to estimate the effect sizes of this stuff, as well as research on optimal design.
I think this is pretty cool
Strongly there should be more explicit defences of this argument.
One way of doing this in a co-operative way might working on co-operative AI stuff, since it seems to increase the likelihood that misaligned AI goes well, or at least less badly.
Yeah, I think a Bayesian perspective is really helpful here and this reply seems right.
I think overrated-underrated is useful because it’s trying to say whether we should be doing more or less of X on the margin. Often it’s much more useful to know whether something is good on the current margin rather than on average.
There isn’t only one notion of utility—utility in decision theory is different to utility in ethics. Utility in decision theory can indeed be derived from choices over lotteries and is incomparable between individuals (without further assumptions) and is equivalent under positive affine transformation because it’s just representing choices.
Utility in moral philosophy refers to value and typically refers to the value of experiences (as opposed to other conceptions of the good like satisfaction of preferences), is comparable between individuals without further assumptions and isn’t equivalent under positive affine transformation.
An individual’s utility (on either of the definitions) may or may not be changed by the political process.
Consider a new far-right party entering the political sphere. They successfully changed political conversations to be more anti-immigration and have lots of focus on immigrant men committing sexual violence.
A voter exposed to these new political conversations has their choice behaviour changed because they now feel more angry towards immigrants and want to hurt them, rather than because they think that more restrictive immigration policies would make them personally safer, for instance.
This same voter also has utility—in the moral philosophy sense—changed by the new political conversation. Now they feel sadistic pleasure when they hear about immigrants being deported on the news, leading to better subjective experiences when they see immigrants being deported.
I strongly reject the claim that we should imagine voters as exclusively deciding how to vote in terms of the personal benefits they derive in expectation from policies. I think people support capital punishment mostly because it fits with their inbuilt sense of justice rather than because they think it benefits them.
We could (probably) represent this voter as being an expected utility maximiser where they have positive utility from capital punishment, in the decision theory sense. This is a different claim from the claim that a voter expects their subjective experiences to be more positively valenced when there’s capital punishment.
I’m afraid I can’t comment on what ignorance factors do or do not account for under Bayesian regret without rereading the paper, but it’s of course possible that they do account for that disparity between actual and assumed preferences.
My views here are just deferring to gender scholars I respect.
Based
Yes I agree with this—but if this is part of the theory of change then Athena should probably privilege applicants with these different backgrounds and I don’t know if they intend to do this.
I’m sceptical that there are substantial benefits to generating AI safety research ideas from gender diversity. I haven’t read the literature here, but my prior on these types of interventions is that the effect size is small.
I regardless think Athena is good for the same reasons Lewis put forward in his comment—the evidence that women are excluded from male-dominated work environments seems strong and it’s very important that we get as many talented researchers into AI safety as possible. This also seems especially like a problem in the AIS community where anecdotal claims of difficulties from unwanted romantic/sexual advances are common.
I think the intellectual benefits from gender diversity claims haven’t been subjected to sufficient scrutiny because it’s convenient to believe. For this kind of claim, I would need to see high-quality causal inference research to believe it and I haven’t seen this research and the article linked doesn’t cite such research. The linked NatGeo article doesn’t seem to me to bring relevant evidence to bear on the question. I completely buy that having more women in the life sciences leads to better medical treatment for women, but that causal mechanism at work here doesn’t seem like it would apply to AI safety research.
Based
Bentham would be proud
This is fantastic
I think I just don’t have sufficiently precise models to know whether it’s more valuable for people to do implementation or strategy work on the current margin.
I think compared to a year ago implementation work has gone up in value because there appears to be an open policy window and so we want to have shovel-ready policies we think are, all things considered, good. I think we’ve also got a bit more strategic clarity than we had a year or so ago thanks to the strategy writing that Holden, Ajeya and Davidson have done.
On the other hand, I think there’s still a lot of strategic ambiguity and for lots of the most important strategy questions there’s like one report with massive uncertainty that’s been done. For instance, both bioanchors and Davidson’s takeoff speeds report assume we could get TAI by just by scaling up compute. This seems like a pretty big assumption. We have no idea what the scaling laws for robotics are, there are constant references to race dynamics but like one non-empirical paper from 2013 that’s modelled it at the firm level (although there’s another coming out.) The two recent Thorstad papers to come out I think are a pretty strong challenge to longtermism not grounded in digital minds being a big deal.
I think people, especially junior people, should be baised towards work with good feedback loops but I think this is a different axis from strategy vs implementation. Lots of epochs work is stratagy work but also has good feedback loops. The legal priorities project and GPI both do pretty high level work but I think both are great because they’re grounded in academic disciplines. Patiant philanthripy is probably the best example of really high level, purely conceptual, work that is great.
In AI in particualr so high level stuff that I think would be great would be: a book on what good post TAI futures look like, forcasting the growth of the Chinese economy under different political setups, scaling laws for robotics, modelling the elasticity of the semi-conductor supply chain, proposals for transfering ownership capital to the population more broadly, investigating different funding models for AI safety.
I think I mostly disagree with this post.
I think Michael Webb would be an example of someone who did pretty abstract stuff (are ideas, in general, getting harder to find) at a relatively junior level (PhD student) but then because his work was impressive and rigorous became very senior in the British government and DeepMind.
Tamay Besiroglu’s MPhil thesis on ideas getting harder to find in ML I think should be counted as strategy research by a junior person but has been important in feeding into various Epoch papers and the Davidson takeoff speeds model. I expect the Epoch papers and the takeoff model to be very impactful. Tamay is now deputy director of Epoch.
My guess is that it’s easier to do bad strategy research than it is to get really good at a niche but important things, but I think it’s very plausible that it’s the better-expected value decision provided you can make your strategy research legibly impressive, and your strategy research is good research. It seems plausible that doing independent strategy research one isn’t aiming to be published in a journal is particularly bad since it doesn’t provide good career capital, there isn’t good mentorship or feedback and there’s no clear path to impact.
I would guess that economists are unusually well-suited to strategy research because it can often be published in journals which is legibly impressive and so is good career capital, and the type of economics strategy research that one does is either empirical and so has good feedback loops, or is model-based but drawn from economic theory and so much more structured than typical theory would be. I think this latter type of research can clearly be impactful—for instance, Racing to the Precipice is a pure game theory paper but informs much of the conversation of avoiding race dynamics. Economics is also generally respected within government and economists are often hired as economists which is unusual amongst the social sciences.
My training is as an economist and it’s plausible to me that work in political science, law, and political philosophy would also be influential but I have less knowledge of these areas.
I don’t want to overemphasise my disagreement—I think lots of people should become experts in very specific things—but I think this post is mostly an argument against doing bad strategy research that doesn’t gain career capital. I expect doing strategy research at an organization that is experienced at doing good and/or legibly impressive research e.g. in academia mostly solves this problem.
A final point, I think this post underrates the long-run influence of ideas on government and policy. The neoliberals of the 60s and 70s are a well-known example of this, but also Jane Jacob’s influence on US urban planning, the influence of legal and literary theory on the Modern American left, and the importance of Silent Spring for the US environmental movement. Research in information economics has been important in designing healthcare systems, e.g. the mandate to buy healthcare under the ACA. The EUs cap and trade scheme is another idea that came quite directly from pretty abstract research.
This is a different kind of influence to proposing a specific or implementing a specific policy in government—which is a very important kind of influence—but I suspect over the long run is more important (though with weak confidence and I don’t think this is especially cruxy.)
I essentially agree with the basic point of this post—and think it was a great post!
I have some what feel like nitpicks about the specific story that you told that and I’m sort of confused about how much they matter. My guess is that this actually is a counterargument to the point being made in the post and imply that trapped priors are less of a problem than the example used in the post would imply.
I think that the broadly libertarian view and Scandinavian-style social democracy views are much more similar than this post gives them credit for. In particular, they agree on the crucial importance of liberal democracy that prevents elites (in the 19th-century traditional agricultural elites) from using the state to engage in rent-seeking. I remember reading a list of demands of the German Social Democratic party in the 1870s (before it had moderated) that read a list of liberal democratic demands—secret ballot, free speech, expansion of the power of democratically elected Reichstag etc. These two strands of modern liberal thought also agreed on a liberal epistemology that should be used to try to systematically improve society from a broadly utilitarian perspective—the London School of Economics was founded by 4 Fabian Society members to further this aim!
I think this cashes out in the Effective Samaritans and the Libertarian side of EA (although the Libertains side of EA is pretty unusually libertarian) pursuing pretty similar projects when trying to use non-randomista means for development. For instance, my guess is that both would support increasing state capacity in low-income countries to improve the basic nightwatchman functions of the state, reducing corruption, protecting liberties and the integrity of elections, and removing regulation that that represent elite rent-seeking. Of course they’ll be some differences in emphasis—the Effective Samarations might have a particular theory of change around using unions to coordinate labour to push for political change—but these seem relatively minor compared to the core things both agree are important. Byran Caplan and Robin Hanson are genuinely unusually libertarian even amongst broadly free-market economists, but typically both utilitarian-motivated libertarians and social democrats would be interested in building at least a basic welfare state in low-income countries.
I think we actually in practice see this convergence between liberal social democrats and broadly utilitarian libertarians in the broadly unified policy agendas of Ezra Klien’s abundance agenda and lots of EA-Rationalist adjacent Libertarians like a focus on making it easier to build houses in highly productive cities, reducing barrier to immigration to rich countries, increased public funding of R&D and improving state capacity, particularly around extremely ambitious projects like operation warp speed.