EdoArad

• EdoAradJan 12, 2024, 6:56 AM

  11 points

  0 ∶ 0

  on: Copen­hagen Con­sen­sus Cen­ter’s best in­vest­ment pa­pers for the sus­tain­able de­vel­op­ment goals

  How should we compare their (CCC’s) cost-benefit estimates to GiveWell’s (GW’s) results?

  I quickly spot two differences:

  1. GW’s benefit estimates are in units of doubling of consumption (interventions are compared to direct cash transfers through Give Directly, itself measured in terms of doubling of consumption), whereas CCC’s estimates are in units of value of a statistical life.

  2. CCC’s analyses are for large-scale government operations, so

     1. they don’t account for counterfactual investments,

     2. they encounter diminishing returns (compared to GW’s focus on the marginal dollar).

  Points 1. and 2.a. are both indications that CCC’s cost-benefit ratios are higher than GW-esque’s cost-effectiveness ratios, while 2.b. goes to the other direction.

  From the Tuberculosis paper, regarding value of a statistical life (VSL):

  The GDP growth in this group of countries outpaces the population growth, so that the VSL grows rapidly over time. In constant 2020 US$ values, the benefit of an averted death is US$ 98,700 (2020), US$ 149,800 (2025), US$ 212,000 (2030), US$ 276,300 (2035), US$ 338,100 (2040), US$ 396,800 (2045), and US$ 456,000 (2050).

  In comparison, GW estimates that one can save a life (or have equivalent impact) at about $5000. That’s a factor of ~20. [I think this is misleading, so I’d be interested in a more careful comparison here, directly comparing VSL to doubling of consumption].

  I’m also not sure how exactly to account for the differences in 2.

Prior knowl­edge elic­i­ta­tion: The past, pre­sent, and fu­ture [re­view pa­per 2023]

• EdoAradJan 5, 2024, 9:34 AM

  3 points

  1 ∶ 0

  in reply to: JoshuaBlake’s comment on: How We Plan to Ap­proach Uncer­tainty in Our Cost-Effec­tive­ness Models

  Totally agree with the need for a more balanced and careful analysis!

• EdoAradJan 5, 2024, 9:30 AM

  2 points

  0 ∶ 0

  in reply to: JoshuaBlake’s comment on: How We Plan to Ap­proach Uncer­tainty in Our Cost-Effec­tive­ness Models

  The main point is that access management is more natively associated with the structure of the model in software settings. Say, you are less likely to release a model without its prerequisites.

  But I agree that this could also be messed up in software environments, and that it’s mainly an issue of UI and culture. I guess I generally argue for a modeling environment that is “modeling-first” rather than something like “explainable-results-first”.

• EdoAradJan 3, 2024, 4:26 PM

  40 points

  5 ∶ 1

  on: How We Plan to Ap­proach Uncer­tainty in Our Cost-Effec­tive­ness Models

  I have a very rough draft based on work with @Michael Latowicki and Charity Entrepreneurship (@Filip_Murar and @weeatquince) where we tried various tools for modeling uncertainty for use in their CEAs. Due to personal issues, I’m not sure when I’ll finish it, so here it is (very unpolished and not finished).

  Mainly, we list various tools we found with some comments on them and discuss the pros and cons of spreadsheet vs software solutions. The latter is mostly coherent so I’ll paste it here:

  Spreadsheets vs programming languages

  The tools we found for probabilistic calculation come in two flavours: either the calculation is expressed in a spreadsheet, or it is written as text in some programming language that is customized by a dedicated interpreter or library for this purpose.

  The spreadsheet-based solutions seem to have one salient advantage: they do not scare non-programmers away. This is, we think, a barrier-to-entry advantage, rather than a long-term productivity advantage. The kind of people who build cost-effectiveness model are not incapable of being productive in a simple, probability-dedicated programming language as well as with a probabilistic library within a general purpose language.

  The error-proneness of spreadsheets

  We strongly suspect that using spreadsheets to organize formulae is more error-prone than using programming languages. We are not alone in this suspicion. Audits of spreadsheets generally find that something between a large fraction and nearly all spreadsheets contain errors.

  Some pitfalls that make spreadsheets more error-prone than programming languages:

  • In a spreadsheet, mistyping a cell address does not necessarily result in an error message. By contrast, mistyping a variable name in a programming language typically does.

  • In a spreadsheet, unintentionally leaving a cell empty is equivalent to setting it to zero, which often does not result in an error. In a programming language, using a variable without declaring it does trigger an error.

  • Since spreadsheet formulae typically reference values by cell address, erroneous references are not salient to the reader. If your formula is “power(1+interest_rate, -years_left)”, but it is encoded as “power(1+M7, $A$13)” then you may easily fail to notice that the “13” should actually be “12”.

  • Formulas are not automatically updated to account for added data, for example in sums of columns.

  • Copy-and-pasting formulae often results in erroneous cell references because the software adjusts treats cell addresses as relative by default and the spreadsheet author often fails to consider whether addresses should be relative or absolute.

  There are other causes of errors in spreadsheets, but you get the point. Errors occur in every kind of programming environment, of course, but spreadsheets sport their own additional pitfalls, on top of those that exist in every programming language.

  We are far from certain that writing cost-effectiveness analyses in an ordinary programming language would reduce the error rate compared to spreadsheets—quantitative estimates of the error rate in both spreadsheets and in non-spreadsheet programs find error rates on the same order of magnitude. The mix of problems that are typically approached using these two types of tools is different though, and we have not found an apples-to-apples study of those error rates.

  It is apparently not easy to root out spreadsheet errors. In “Spreadsheet errors: What we know. What we think we can do”, Professor of IT management Ray Panko summarizes his findings:

  Unfortunately, only one approach to error reduction has been demonstrated to be effective. This is code inspection, in which a group of spreadsheet developers checks a spreadsheet cell-by-cell to discover errors. Even this exhausting and expensive process will catch only about 80% of all errors. Other things can be done to reduce errors, but given the tenacity of spreadsheet error in the face of even cell-by-cell code inspection, we should be careful about expecting too much from most error-reducing approach.

• EdoAradJan 3, 2024, 3:39 PM

  2 points

  0 ∶ 0

  in reply to: MHR🔸’s comment on: How We Plan to Ap­proach Uncer­tainty in Our Cost-Effec­tive­ness Models

  [btw, this is a common problem when using spreadsheets rather than when modeling in a software development environment—the software space has a lot of experience in working in (partially-) open source settings]

• EdoAradJan 3, 2024, 3:09 PM

  2 points

  0 ∶ 0

  on: Help me find an­swers for the miss­ing parts in mod­ern ed­u­ca­tion.

  Relevant keyphrase: Positive Education

• EdoAradDec 26, 2023, 4:57 AM

  5 points

  1 ∶ 0

  in reply to: EdoArad’s comment on: new trol­ley problem

  Oh! I see that you’ve also posted Shrimp Neuroanatomy. Great, I’m looking forward future posts on this topic to inform the debate :)

• EdoAradDec 26, 2023, 4:54 AM

  9 points

  2 ∶ 0

  on: new trol­ley problem

  I don’t agree with your take, but many people in EA (and definitely outside it) feel similarly to you so I appreciate you writing this. It’s also a good time to criticize invertebrate welfare, as more attention and funding are directed there.

  That said, I found the tone of this post too confrontational to promote useful discussion so I’ve decided neither to upvote or downvote. I know it’s a tough ask, but it would also be very useful to directly engage with works in this space (say, recent posts on invertebrate welfare or RP’s sequence) and clearly outline the difference in views.

• EdoAradDec 25, 2023, 2:25 PM

  2 points

  1 ∶ 0

  in reply to: MichaelDickens’s comment on: BenMill­wood’s Shortform

  I like this as an example of a case where you wouldn’t want to combine these two different forms of uncertainty

• EdoAradNov 29, 2023, 7:07 AM

  2 points

  0 ∶ 0

  in reply to: Agustín Covarrubias 🔸’s comment on: Re­think Pri­ori­ties’ Cross-Cause Cost-Effec­tive­ness Model: In­tro­duc­tion and Overview

  Yeah. If I understand correctly, everything is resampled rather than cached, so comparing results between two models is only done on aggregate rather than on a sample-by-sample basis

• EdoAradNov 28, 2023, 6:10 PM

  5 points

  0 ∶ 0

  on: Re­think Pri­ori­ties’ Cross-Cause Cost-Effec­tive­ness Model: In­tro­duc­tion and Overview

  If I understand correctly, all the variables are simulated freshly for each model. In particular, that applies to some underlying assumptions that are logically shared or correlated between models (say, sentience probabilities or x-risks).

  I think this may cause some issues when comparing between different causes. At the very least, it seems likely to understate the certainty by which one intervention may be better than another. I think it may also affect the ordering, particularly if we take some kind of risk aversion or other non-linear utility into account.

  To be clear, this would be a problem in any uncertainty-based CEA modeling, and clearly the situation in non-randomized models is usually much worse. It may also be very minor, not sure.

• EdoAradNov 22, 2023, 9:14 AM

  4 points

  0 ∶ 0

  on: Vi­su­als for EA-al­igned Research

  These are beautiful! Thanks for your generous offer to the community, this seems very valuable!

• EdoAradNov 12, 2023, 6:43 AM

  2 points

  1 ∶ 0

  on: Is en­ergy trad­ing use­ful to the en­ergy tran­si­tion?

  I have a very limited understanding of this field, so the following is probably a bit wrong but hopefully leading in the right direction.

  In the US (and maybe some other countries) there is a regulated energy demand & supply prediction market. This is set up at various points in the electric grid, which is connected to many different suppliers of both renewable and non-renwable sources of electricity.

  The coal power plants are required to be able to deliver enough electricity so that there is a low chance of power outage. The bound on that chance is defined by regulations; I’m not sure whether it is measured retroactively (say, % of days without power per household) or directly from the prediction market (say, prepare to supply at least 4 std above the mean demand). Prediction is key here, as plants need to prepare in advance (say, 12 hours, not sure) for some technical reason and are committed to some degree of energy generation. The better the prediction, the less they need to generate to be safe from edge cases.

  The addition of highly-variable/​unpredictable renewable sources of energy to the grid means that they can take some of the load off from coal plants. But again, increasing the predictive accuracy means that the coal plants can generate less excess energy.

  I think there are some estimates of exactly how much near-term CO2 emissions are being reduced based on better predictions. I’m not sure how to think about the long-term effects on the industry.

  Also, there may be other solutions in the longer term. Say, better energy storage mechanisms may replace the need to have highly accurate predictions, as suppliers may save excess energy with less waste.

  ---

  As a career, my main guess is that the potential for E2G here is greater than the direct impact one has. That is, these are generally high-paying roles and I suspect that donating, say, 10% of the income to Clean Air Task Force would be much more effective for the energy transition. If that’s the case, it should be compared to other possible jobs with even higher salaries.

  However, if you are interested in learning more about the industry in general, I think that working in such a company could be a great way for people with strong quantitative background to do so. Generally, when doing any type of data science you are going to learn a lot on the subject matter, and in this case it may even be more so.

  ---

  Again, these are rough guesses. I’d be interested if you or anyone else here would find relevant literature on the topic or write some stronger opinions and analyses.

• EdoAradNov 10, 2023, 5:50 PM

  2 points

  0 ∶ 0

  in reply to: Lorenzo Buonanno🔸’s comment on: 1/​E(X) is not E(1/​X)

  Thanks! Yeah, I totally agree—the topic is surprisingly delicate and nonintuitive, and my examples above are too technical.

  By the way, I’d love it if other people would write posts that make the exact same point but better! (or for different audiences).

• EdoAradNov 10, 2023, 5:43 PM

  2 points

  0 ∶ 0

  in reply to: Stan Pinsent’s comment on: 1/​E(X) is not E(1/​X)

  I think in this thread there are two ways of defining costs:

  1. Michael considers the cost as the total amount spent

  2. Stan suggests a case where the cost is the amount needed to be spent per unit of intervention.

  I think this is the major source of disagreement here, right?

  This discussion resembles the observation that the cost-effectiveness ratio should mostly be used in the margin. That is, in the end we care about something like $\left(\text{total effect}\right)-\left(\text{total cost}\right)$ and when we decide where to spend the next dollar we should compute the derivatives with respect to that extra resource and choose the intervention which maximizes that increased value.

• EdoAradNov 9, 2023, 11:55 AM

  9 points

  4 ∶ 0

  on: 1/​E(X) is not E(1/​X)

  Things to add:

  1. Graphical example

  2. Explicit discussion of GiveWell’s use of cost/​effect

  3. Real-world examples where this causes a significant error

  4. Expand on the lognormal example, discuss cases where the variance is in the same order of magnitude as the expected value

  5. What this says about how we should model uncertainty

1/​E(X) is not E(1/​X)

• EdoAradNov 2, 2023, 7:24 AM

  2 points

  0 ∶ 0

  on: Any­one tried to mon­e­tize a hobby with the in­tent of ETG?

  @Aaron Gertler has some experience fundraising from his MTG stream and may want to chime in :)

Epistemic and Statis­ti­cal Uncer­tainty in CEAs—Draft