1st year PhD student in Agricultural and Resource Economics at Berkeley. Likes animal welfare, development economics and impact evaluation. Past lives at World Bank, IMF, and doing software engineering.
geoffrey
Karma: 735
Would you recommend that I share any such posts with both the authors and the evaluators before making them?
Yes. But zooming back out, I don’t know if these EA Forum posts are necessary.
A practice I saw i4replication (or some other replication lab) is that the editors didn’t provide any “value-added” commentary on any given paper. At least, I didn’t see these in any tweets they did. They link to the evaluation reports + a response from the author and then leave it at that.
Once in a while, there will be a retrospective on how the replications are going as a whole. But I think they refrain from commenting on any paper.
If I had to rationalize why they did that, my guess is that replications are already an opt-in thing with lots of downside. And psychologically, editor commentary has a lot more potential for unpleasantness. Peer review tends to be anonymous so it doesn’t feel as personal because the critics are kept secret. But editor commentary isn’t secret...actually feels personal, and editors tend to have more clout.
Basically, I think the bar for an editor commentary post like this should be even higher than the usual process. And the usual evaluation process already allows for author review and response. So I think a “value-added” post like this should pass a higher bar of diplomacy and insight.
Chiming in here with my outsider impressions on how fair the process seems
@david_reinstein If I were to rank the evaluator reports, evaluation summary, and the EA Forum post in which ones seemed the most fair, I would have ranked the Forum post last. It wasn’t until I clicked through to the evaluation reports that I felt the process wasn’t so cutting.
Let me focus on one very specific framing in the Forum post, since it feels representative. One heading includes the phrase “this meta-analysis is not rigorous enough”. This has a few connotations that you probably didn’t mean. One, this meta-analysis is much worse than others. Two, the claims are questionable. Three, there’s a universally correct level of quality that meta-analyses should reach and anything that falls short of that is inadmissible as evidence.
In reality, it seems this meta-analysis is par for the course in terms of quality. And it was probably more difficult to do so given the heterogeneity in the literature. And the central claim of the meta-analysis doesn’t seem like something either evaluator disputed (though one evaluator was hesitant).
Again, I know that’s not what you meant and there are many caveats throughout the post. But it’s one of a few editorial choices that make the Forum post seem much more critical than the evaluation reports, which is a bit unusual since the Evaluators are the ones who are actually critiquing the paper.
Finally, one piece of context that felt odd not to mention was the fundamental difficulty of finding an expert in both food consumption and meta-analysis. That limits the ability of any reviewer to make a fair evaluation. This is acknowledged at the bottom of the Evaluation Summary. Elsewhere, I’m not sure where it’s said. Without that mentioned, I think it’s easy for a casual reader to leave thinking the two Evaluators are the “most correct”.
Really enjoyed this. Not much public debate in this space as far as I can see. To two of your cruxes:
Is meta-analysis even useful in these contexts, with heterogeneous interventions, outcomes, and analytical approaches?
Will anyone actually do/fund/reward rigorous continued work?
I’ve sometimes wondered if it’d be worth funding a “mega study” like Milkman et al. (2021). They tested 54 different interventions to boost exercise among 61,000 members. Something similar for meat reduction could allow for some clean apples-to-apples comparisons.
I’ve seen the number $2.6 million floating around for how much this intervention costs. Granted, that’s probably on top of convincing the mega-team of researchers to work on the project, which might only happen through the prestige of an academic lab. But it’s also not an astronomical cost. And there’d be still some learning value from a smaller set of interventions and a smaller sample.
This might be a better use of resources than striving for the “ideal” meta-analysis, since that sounds expensive too.
Ooh this is neat.
I like how it neutralizes the certainty-seeking part of me since it’s only me, the difference maker, that has the option of a guaranteed 100% outcome. For the beneficiary, it’s always a gamble.
Agree it’s more about upbringing and messaging. And also relate a lot to this.
But also I think it’s really hard to tell the “cause” of any given problem at an individual level. As recently as a few years ago, I would have put 80% weight on upbringing / messaging (which I agree aren’t the identities themselves but something associated with them). Nowadays I’m more agnostic about it.
I think it’s fine to seek out affinity groups and culturally-relevant advice to some degree. But also, there’s a tradeoff between exploring identities versus applying generic mental health advice. Especially when you get to intersectionality-type stuff like trifectas where the number of things to explore is gets incredibly vast very quickly.
I can speak to two of those three identities (EA and Asian). I think one possibility that took me an unusually long time to consider was that maybe my identities didn’t matter and I’d still feel the same problems if I was the “default person” in society. And I was working through a lot of identities.
It’s a weird way of framing things since we can’t have our identities counterfactually removed. Even if we did, we wouldn’t be the same person. But I think it’s a framework that usually doesn’t get mentioned much in mental health circles , especially on the internet. Partly because it feels invalidating, partly because most people really want contextual advice, and partly because it feels “emotionally dumb and ignorant” to downplay sociological factors.
To do some fake math on this, if we could decompose mental health problems into the triple Venn diagram of Asian-women-EA (which is 6 different things if you count up the intersectionalities!) and include stuff outside that, it’s possible for the Asian-women-EA sources of stress to be maybe only 10-25%.
Basically, part of the challenge of identity is not just figuring out if it matters but also how much. And maybe that amount is ultimately a small thing. Or maybe it’s not as tractable as working on the identity-less portions
Agree the value is high. But practically, there’s two big questions that pop to mind since I work / study around this area:
If aggregating existing datasets, what’s your value-add over what J-PAL, World Bank, IDInsight, Our World in Data, and what numerous un-affliated academics are already doing? (See Best of EconTwitter Substack for “public goods” which are sometimes publicly accessible datasets)
If gaining access to new datasets, what are you offering to LMIC governments in exchange? Even making a single batch of batch of data publicly accessible is a lift. So in practice, they need to see some value, analytically or logistically, to be willing to work with you
This is really good.
What struck me was all the concrete detail. While it is personal, it’s also in service of giving useful lessons to other people. It helps establish how generalizable the career advice would be to other people and it reframes some standard career advice in a way that centers the constraints as a first-order consideration.
I would not have taken the adversity quotient framing seriously otherwise.
The one addition that might help is mentioning whether there were aspects of your career path that felt unusually lucky or aspects of your life circumstances that felt strong relative to others in your situation. Structural barriers can be a subtle thing (like someone getting a decent math education because they went to a decent school in a bad neighborhood). Mostly this helps with generalizability to readers.
Do any of you have heuristics for when to “give up” or “pivot” in a job search? Examples could be aiming lower / differently if no response after 10 applications.
Thankfully this is not something I have to worry about for a long time. But I think it’s useful to have some balance to the usual advice of “just keep trying; job searching takes a long time”. Sometimes a job really is unrealistic for a person’s current profile (let’s operationalize that as 1000 job searching hours would still only result in a 1% chance of getting a certain set of jobs).
Thanks for this. I’m surprised how consistently the studies point in favor of vegan diets being cheaper on the whole (though I’ll caveat none of these are too convincing: the headline RCT is testing a low-fat vegan diet instead of a general vegan diet and the rest are descriptive regressions / modeling exercises).
All that said, I’m wondering if perception of vegan diets being more expensive could be explained by:
Fully plant-based diets are cheaper but various “halfway points” are more expensive.
Meat-eaters mostly get exposure to the “halfway points”. These could be:
The only vegetarian in a meat-eating group who’s stuck buying the heavily marked-up vegetarian option at the steakhouse
The only vegetarian in a meat-eating household who buys groceries and/or does the cooking but can’t economize on legume-based recipes.
The lone transitioning vegetarian who’s goes through a long learning period since they don’t have a plant-based community to learn cooking techniques or new cuisines or hear where to get affordable produce.
There’s some descriptive evidence from “Some vegetarians spend less money on food, others don’t” (Jayson & Lusk 2016) pointing in this direction. The researchers do a neat classification trick where they split vegetarians into partial vegetarians and pure vegetarians. Partial vegetarians are those that identify as vegetarian but still report purchasing / consuming meat products. Spending is highest for partial vegetarians followed by meat-eaters followed by pure vegetarians.
Those results are confounded by demographics. But I still think it points to some things that seem under explored in these studies. Would love to hear of other studies about financial costs for transitioning vegans / social taxes for vegans in meat-eating communities
Agreed, but I’d be careful not to confuse good mentorship with good management. These usually go hand-in-hand. But sometimes a manager is good because they sacrifice some of your career growth for the sake of the company.
I like the archetypes of ruthless versus empathetic managers described here. It’s an arbitrary division and many managers do fulfill both archetypes. But I think it also captures an important dynamic, where managers have to tradeoff between their own career, your career, and the organization as a whole. Mentorship and career development falls into that
Edit: Another distinction I’d add is good manager versus good management. Sometimes it’s the organizational structure that determines whether you’ll get good training. In my experience, larger and stable organizations are better at mentorship for a ton of reasons, such as being able to make multi-year investments in training programs. A scrappy startup, on the other hand, may be a few weeks away from shutting down.
I definitely feel a few of my past managers would have been much better at mentorship if other aspects of the situation were different (more capacity, less short-term deadlines, better higher-up managers, etc.).
Not sure what the right numbers are but I really like the back-of-the-envelope approach you’re taking here. It’s simple and concrete enough that it’s probably going to bounce around in my head for a while
Good point. In a toy model, it’d depend on relative cuts to labor versus non-labor inputs. Now that I think about it, it probably points towards exiting being better in mission-driven fields. People are more attached to their careers so the non-labor resources get cut deeply while all the staff try to hold onto their jobs.
Maybe I’d amend it to… if you’re willing to switch jobs, then you can benefit from increasing marginal returns in some sub-cause areas. Because maybe there’s a sub-cause area where lots of staff are quitting (out of fear the cause area isn’t worth it) while capital investment is about the same.
But I admit that, even if we knew those sub-cause areas existed, it’s not quite as punchy of a reason to stay in the cause area as a whole
Marginal returns to work (probably) go up with funding cuts, not down.
It can be demoralizing when a field you’re working in gets funding cuts. Job security goes down, less stuff is happening in your area, and people may pay you less attention since they believe others are doing more important work. But assuming you have job security and mostly make career decisions on inside views (meaning you’re not updating too heavily on funders de-prioritizing your cause area), then your skills are more valuable than they were previously.
Lots of caveats apply of course. The big one to me seems that some projects need a minimum scale to work. But I also think this idea is a nice psychological counterweight to the career uncertainties that pop up with changes in the funding landscape.
(Inspired by a comment Dean Karlan, a development economist, made on funding cuts to global health)
My immediate hesitation is whether fresh college graduates would be useful enough to hosting organizations to make this program sustainable.
Last I checked, Peace Corps invests 3 months of formal training into each applicant and requires a minimum 2-year commitment in a role (to allow people to grow into competency). But this version of Animal Advocacy Corps has college undergraduates rotate thru multiple organizations for much shorter periods without any training. And I’m not sure how much demand there is for that kind of worker in animal advocacy even if it’s provided for free.
Agreed. I’d extend the claim past ideas and say that EA is very non-elitist (or at least better than most professional fields) at any point of evaluation.
Maybe because of that, it seems more elitist elsewhere.
Like the idea but it might be at odds with the recent AI governance shift. In general, policy folks of all stripes, especially the more senior and insider ones, practice a lot of selective disclosure.
Having done a lot of this advice in my 20s, I’d recommend just getting started with an online training program you find interesting, seems career relevant, and also not too pie-in-the-sky as a near-term plan. Throughout my life, I think there were one or two that felt unusually good or bad all-things-considered. Even then, training programs are short (~6 weeks) and have no stakes if you stop them.
(The exception is if the training somehow includes hands-on training from someone actively trying to progress in one of your desired career paths. Good mentorship is a scarce resource and you should prioritize it above a lot of other things.)
It’s dramatically more important what you do after the online training program. It’s extremely rare that these programs set people up to to do the “impressive project” that hiring managers want from less prestigious candidates. If they did, everyone would be doing them.
As for the program, if you feel like you’re at least passing the course (whatever than means) and it seems promising, then I’d pair that with some informational interviews. You can ask “Hey I’ve been doing X training course and feel like it might be a good career path. Would you be willing to chat about how you got to where you were?”.
That will help you identify directions to take for your “career ladder”, which I put in quotes since it’s really more of a fog-of-war. Unfortunately, it’s usually the things between “Step 1″ and “desired job” where steps are the least clear and the most consequential. So I would save your energy for when you get there.
I’ve done a lot of partially blind hiring processes both within EA and outside it [1]. And as much as I like them (and feel like I’ve benefited from them), I think there’s good reasons why they aren’t done more.
It seems really hard to create a good blind hiring process. Most of the ones I felt good about were constructed with an immense amount of care to balance not rejecting good candidates but still having enough subtlety to distinguish candidates that would pass a final stage. Even then I still felt like there would be exceptional candidates that would have fallen through the cracks because they were bad at the specific skills being tested in the blind stage. I still think the benefits are overall positive but I’m not super confident in that given the risk of mistakenly rejecting good people.
There were always at least two blind stages before the final un-blinded one: the first being a fast filter (either an automated quiz or a short task that could be graded in 1-2 minutes) and the second being a very in-depth task. Granted, this isn’t experimental evidence but it does suggest that one fast blind part isn’t enough.
The second in-depth task seems very time-consuming to maintain. Having graded anonymous work tests for my current role where there was a big “we want to hire overlooked internationals” motivation, I felt like I needed at least 5 minutes per task before I felt even semi-confident what my final feeling was going to be and usually quite a bit more to be sure and fulfill the spirit of a blind evaluation.
Many roles have downside asymmetry. I’ve mostly seen blind tests in highly technical fields where the organization can benefit if the candidate is a superstar but also isolate the damage if they turn out to be highly toxic. With operational roles, the downside is much larger while the benefits are smaller.
Anecdotally, blind hiring these days doesn’t seem guaranteed to increase demographic diversity and may even decrease it a tiny bit. I feel the most confident on this with initiatives to increase women in software engineering via hiring practices. But I’m a lot less confident on country-of-origin. My hunch is it would backfire a bit in the non-profit world, especially in effective animal advocacy building where there seems to be some attempt to build multinational capacity.
- ^
Applied seriously to a software engineer role at a prestigious tech firm and had a final stage interview that far exceeded my abilities, which was painful for everyone involved. Applied on a whim to the Charity Entrepreneurship incubation program and got rejected after the second (third?) stage. Accepted the job offer to my current credentialed non-EA job after two blind stages and a final in-person. Applied seriously to Charity Entrepreneurship Research Training program and got rejected after second stage. Applied as a long shot to a GiveWell Senior Researcher role and got rejected after second stage.
Seth, for what it’s worth, I found your hourly estimates (provided in these forum comments but not something I saw in the evaluator response) on how long the extensions would take to be illuminating. Very rough numbers like this meta-analysis taking 1000 hours for you or a robustness check taking dozens / hundreds of hours more to do properly helps contextualize how reasonable the critiques are.
It’s easy for me (even now while pursuing research, but especially before when I was merely consuming it) to think these changes would take a few days.
It’s also gives me insight into the research production process. How long does it take to do a meta-analysis? How much does rigor cost? How much insight does rigor buy? What insight is possible given current studies? Questions like that help me figure out whether a project is worth pursuing and whether it’s compatible with career incentives or more of a non-promotable task