Eva

Karma: 553

Eva 22 May 2018 0:02 UTC
3 points
0 ∶ 0
in reply to: Michael_S’s comment on: Do Prof Eva Vivalt’s results show ‘evidence-based’ development isn’t all it’s cut out to be?
I agree that it would be important to weigh the costs and benefits—I don’t think it’s exclusively an issue with RCTs, though.

One thing that could help in doing this calculus is a better understanding of when our non-study-informed beliefs are likely to be accurate.

I know at least some researchers are working in this area—Stefano DellaVigna and Devin Pope are looking to follow up their excellent papers on predictions with another one looking at how well people predict results based on differences in context, and Aidan Coville and I also have some work in this area using impact evaluations in development and predictions gathered from policymakers, practitioners, and researchers.

Eva 22 May 2018 0:28 UTC
3 points
0 ∶ 0
in reply to: RomeoStevens’s comment on: Do Prof Eva Vivalt’s results show ‘evidence-based’ development isn’t all it’s cut out to be?
And when groups do work on these issues there is a tendency towards infighting.

Some things that could help:
- Workshops that bring people together. It’s harder to misinterpret someone’s work when they are describing it in front of you, and it’s easier to make fast progress towards a common goal (and to increase the salience of the goal).
- Explicitly recognizing that the community is small and needs nurturing. It’s natural for people to at first be scared that someone else is in their coveted area (career concerns), but overall I think it might be a good thing even on a personal level. It’s such a neglected topic that if people work together and help bring attention to it real progress could be made. In contrast, sometimes you see a subfield where people are so busy tearing down each other’s work that nothing can get published or funded—a much worse equilibrium.
Bringing people together is hugely important to working constructively.

Eva 22 May 2018 10:20 UTC
3 points
0 ∶ 0
on: Do Prof Eva Vivalt’s results show ‘evidence-based’ development isn’t all it’s cut out to be?
This video might also add to the discussion—the closing panel at CSAE this year was largely on methodology, moderated by Hilary Greaves (head of the new Global Priorities Institute at Oxford), with Michael Kremer, Justin Sandefur, Joseph Ssentongo, and myself. Some of the comments from the other panellists still stick with me today.

https://ox.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=ec3f076c-9c71-4462-9b84-a8a100f5a44c

Eva 22 May 2018 23:59 UTC

6 points

0 ∶ 0

Error

The value
  NIL
is not of type
  SIMPLE-STRING
when binding #:USER-ID163

Eva 23 May 2018 0:10 UTC
0 points
0 ∶ 0
in reply to: RomeoStevens’s comment on: Do Prof Eva Vivalt’s results show ‘evidence-based’ development isn’t all it’s cut out to be?
I’ll try to think about this some more. It’s a good question.

Eva 14 Nov 2019 15:05 UTC
8 points
0 ∶ 0
on: Steelmanning the Case Against Unquantifiable Interventions
As a small note, we might get more precise estimates of the effects of a program by predicting magnitudes rather than whether something will replicate (which is what we’re doing with the Social Science Prediction Platform). That said, I think a lot of work needs to be done before we can have trust in predictions, and there will always be a gap between how comfortable we are extrapolating to other things we could study vs. “unquantifiable” interventions.
(There’s an analogy to external validity here, where you can do more if you can assume the study you predict is drawn from the same set as those you have studied, or the same set if weighted in some way. You could in principle make an ordering of how feasible something is to be studied, and regress your ability to predict on that, but that would be incredibly noisy and not practical as things stand, and past some threshold you don’t observe studies anymore and have little to say without making strong assumptions about generalizing past that threshold.)

Eva 2 Jan 2023 22:40 UTC
6 points
0 ∶ 0
in reply to: freedomandutility’s comment on: Prediction Markets for Science
Thanks for mentioning the Social Science Prediction Platform! We had some interest from other sciences as well.

With collaborators, we outlined some other reasons to forecast research results here: https://www.science.org/doi/10.1126/science.aaz1704. In short, forecasts can help to evaluate the novelty of a result (a double-edged sword: very unexpected results are more likely to be suspect), mitigate publication bias against null results / provide an alternative null, and over time help to improve the accuracy of forecasting. There are other reasons, as well, like identifying which treatment to test or which outcome variables to focus on (which might have the highest VoI). In the long run, if forecasts are linked to RCT results, it could also help us say more about those situations for which we don’t have RCTs—but that’s a longer-term goal. If this is an area of interest, I’ve got a podcast episode, EA Global presentation and some other things in this vein… this is probably the most detailed.

I agree that there’s a lot of work in this area and decision makers actively interested in it. I’ll also add that there’s a lot of interest on the researcher side, which is key.

P.S. The SSPP is hiring web developers, if you know anyone who might be a good fit.

Eva 8 Feb 2023 13:50 UTC
42 points
17 ∶ 1
on: Why People Use Burner Accounts: A Commentary on Blacklists, the EA “Inner Circle”, and Funding
I’m not going to deal with the topic of the post, but there’s another reason to not post under a burner account if it can be avoided that I haven’t seen mentioned, which this post indirectly highlights.

When people post under burner accounts, it makes it harder to be confident in the information that the posts contain, because there is ambiguity and it could be the same person repeatedly posting. To give one example (not the only one), if you see X number of burner accounts posting “I observe Y”, then that could mean anywhere from 1 to X observations of Y, and it’s hard to get a sense of the true frequency. This means it undermines the message of those posting, to post under burners, because some of their information will be discounted.

In this post, the poster writes “Therefore, I feel comfortable questioning these grants using burner accounts,” which suggests in fact that they do have multiple burner accounts. I recognize that using the same burner account would, over time, aggregate information that would lead to slightly less anonymity, but again, the tradeoff is that it significantly undermines the signal. I suspect it could lead to a vicious cycle for those posting, if they repeatedly feel like their posts aren’t being taken seriously.

Eva 8 Feb 2023 23:25 UTC
22 points
9 ∶ 1
on: Book Post: The Good It Promises, the Harm It Does: Critical Essays on Effective Altruism
I would really like to read a summary of this book. The reviews posted here (edit: in the original post) do not actually give much insight as to the contents. I’m hoping someone will post a detailed summary on the forum (and, as EAs love self-criticism, fully expect someone will!).

Eva 21 Feb 2023 21:49 UTC
61 points
20 ∶ 1
in reply to: titotal’s comment on: A statement and an apology
I’ve stayed at a (non-EA) professional contact’s house before when they’d invited me to give a talk and later very apologetically realized they didn’t have the budget for a hotel. They likely felt obliged to offer; I felt like it would be awkward to decline. We were both at pains to be extremely, exceedingly, painstakingly polite given the circumstances and turn the formality up a notch.

I agree the org should have paid for a hotel, I’m only mentioning this because if baseline formality is a 5, I would think it would be more normal to kick it up to a 10 under the circumstances. It makes this situation all the more bizarre.

Eva 26 Dec 2023 19:27 UTC
23 points
1 ∶ 0
in reply to: Karthik Tadepalli’s comment on: An update and personal reflections about AidGrade
Thanks. A quick, non-exhaustive list:
- Get feedback early on. Talking to people can save a lot of time
- You should have a very clear idea of why it is needed. Good ideas sound obvious after the fact
- That’s not to say people won’t disagree with you. If your idea takes off you will need to have a thick skin
- A super-easy way to have more impact is to collaborate with others. This doesn’t help for job market papers, where people tend to want to have solo-authored work. But you can get a lot more done collaborating with others and the outputs will be higher-quality, too
- Apart from collaborating with people on the actual project, do what you can to get buy-in from other people who have no relationship to the project. Other people can magnify the impact in big ways and small
- It can take a while before early-career researchers find a good idea. Have more ideas than you would think

Eva 31 Dec 2023 14:11 UTC
10 points
0 ∶ 0
in reply to: Eva’s comment on: An update and personal reflections about AidGrade
There’s another point I don’t quite know how to put but I’ll give it a go.
Despite the comments above about having many ideas and getting feedback early about one’s projects—which both point to having and abandoning ideas quickly—there’s another sense in which actually what one needs is an ability to stick to things. And the good taste to be able to evaluate when to try something else and when to keep going. (This is less about specific projects and more about larger shifts like whether to stay in academia/a certain line of work at all.)
I feel like sometimes people get too much advice to abandon things early. It’s advice that has intuitive appeal (if you can’t pick winners, at least cut your losses early), and it’s good advice in a lot of situations. But my impression is that while there are some people who would do better failing faster, there are also some people who would do better if they were more patient. At least for myself, I started having more success when sticking with things for longer. The longer you stick to a thing, the more expertise you have in it. That may not matter in some fields, but it matters in academia.
Now, obviously, you want to be very selective about what you stick to. That’s where having good taste comes in. But I’d start by looking honestly at yourself and looking at people near you that you see doing well for themselves in your chosen field, and asking which side of the impatient-patient spectrum you fall on compared to them. Some people are too patient. Some people are too impatient. I was too impatient and improved with more patience, and for some people it’s the opposite. Which advice applies the most to you depends on your starting point and field, and of course the outside options.
For econ PhDs, I think it’s worth having a lot of ideas and discarding them quickly especially in grad school because a lot of them are bad at first, but I also think there are people who jump ship from an academic career too early, like when they are on the market or in the first few years after. I suspect this might be generally true in academia where expertise really, really matters and you need to make a long-term investment, but I can’t speak for certain about other academic fields beyond economics. And I’ve definitely met many academics who played it too safe for maximizing impact, and many people who didn’t leave quickly enough. What I’m trying to emphasize is that it’s possible to make mistakes in both directions and you should put effort into figuring out which type of error you personally are more likely to make.