Senior Behavioural Scientist at Rethink Priorities
Jamie Elsey
Karma: 989
Does it change how you think of the process if GPT gives its own contextual explanation (the answer may again be no, irrelevant)?
Me:
I went to a cafe where a glass of soda costs $3. I drank two and a half glasses of soda. How much did I have to pay?
ChatGPT:
For drinking two and a half glasses of soda, at $3 per glass, you would have to pay $7.50.
Me:
This is something of a trick question—the correct answer is $9.
ChatGPT:
If you were charged $9 for two and a half glasses of soda, it seems like there might have been a minimum charge per glass regardless of it being a full glass or not. It implies that each glass (or part of a glass) was counted as a whole for billing purposes. This way, two full glasses would be $6, and the half glass, treated as a full one for pricing, would add another $3, totaling $9. That’s a unique way to price it!
This is really interesting and I was surprised when putting this in to GP4 that indeed, it makes what seems like a basic mistake. However, I also think your prompts might imply some contextual assumptions, which might be prompting it to think this is just an elementary math question rather than something to do with specifically ordering drinks in a cafe and the rules around that. If I slightly change the prompt it appears GPT does have the knowledge and can use it, without needing instruction [edit: I think this could be irrelevant to the broader point you are making, but looking at how minor changes in the prompts or followups elicit very different responses makes the examples feel like any other type of input you put into an LLM and lowers my sense that the examples are revealing something special. For example that the level of tutoring/explanation you give it isn’t really necessary etc., though as I note I’m unsure if this changes how you would interpret the outputs]:
Me:
I went to a cafe where a glass of soda costs $3. I ordered 3 glasses of soda, and drank two and a half glasses of soda. How much did I have to pay?
ChatGPT:
To calculate how much you had to pay, we’ll multiply the number of glasses of soda you ordered by the cost of one glass of soda. You ordered 3 glasses of soda at $3 each.
Let’s do the math:
You had to pay $9 for the 3 glasses of soda you ordered. The amount you drank does not affect the total cost.
Sorry, but in your first link it literally says “But, audit rates have dropped for all income levels—with audit rates decreasing the most for taxpayers with incomes of $200,000 or more. IRS officials said audit rates declined due to staffing decreases and because it takes more staff time and expertise to handle complex higher-income audits.” So I am confused by that reference, as it seems to be exactly what I said.
My general point was however not about taxes specifically, I just thought taxes might be one example. I am more generally surprised that people think for crimes overall that the wealthy are more persecuted. My impression is that most of human history indicates the opposite.
Point 1 strikes me as probably incorrect. While there may be more money to get in going after higher net worth individuals, it is also far more expensive because they can set themselves up with excellent legal representation. Unfortunately I don’t have a reference for this but I have a vague recollection of reading in multiple places that e.g. gutting the IRS disproportionately benefits the wealthy because the IRS then does not have the funding to pursue more complex and better defended cases, even though there is more money to be had there.
Similarly for other crimes, including things that are obivously wrong and not just ‘would I feel that bad if my friend did it’ level, I would expect there to be less of this reaching prosecution among the very wealthy than among average people who committed similar crimes (such as sexual or generally violent assault, especially when committed against people who are not themselves very wealthy)
This is really great, and thanks for highlighting where to get the data—I wouldn’t have heard about this otherwise.
Regarding this:
For instance, it was found that areas with older, lower education, and more conservative voter-support had a lower proportion of meat purchases
That’s definitely an interesting finding. Is it possible that some of these factors, such as lower education, are associated with lower income—thus less able to afford a lot of meat? Alternatively, might there also be differences in where people shop; perhaps they get meat from another store or local butcher?
I agree that sales data can be much more informative, or informative in different ways, than self-report data, so this is really nice to see
Thanks for your comment @Robert Van Buskirk, and great that you are also thinking of ways to incorporate uncertainty into CEAs. You note that your way also solves ‘the same problem’, but a key aspect of what we are arguing (and what others have suggested) is not just that a proper monte carlo approach avoids objective issues with an interval-based approach, but that it provides a wealth of more nuanced information and allows for much more discerning assessments to be made. This is approximated by the approach you suggest, but not fully realised. For example ‘A histogram of the full set of CE results is then constructed to illustrate the full range of possible CE values’ - but by definition the histogram won’t include the full range of possible values because values beyond the average of the top and bottom thirds are not included in the calculations. This might obscure or underestimate the possibility of extreme outcomes.
My second concern would just be that if one has the capacity to define what the average of the bottom third of the expected values are, and the average of the top third, this is in all likelihood enough information for defining an approximate distribution, or one could simply select a uniform distribution. In that case, why not just do a proper monte carlo? The main answer to this is often just that people don’t have the confidence to consider a possible probability distribution that might work, and that was the key thing I’m trying to help with in developing and sharing this application
Thanks for sharing your experience on becoming vegetarian! There were also several people in the survey who said things such as just finding it disgusting.
Thanks for sharing this additional information and numbers. My concern basically lies in what I see as a large difference between making meatless/plant-based options a default or more easily available choice vs. removing anything that is not plant-based from campus. The empirical information from interventions above relate to interventions that are in the form of switching a default and not about entirely removing an option, and it is with the ‘forcing of a shift in behavior’ as you put it where I can imagine a lot more negative sentiment among those who do not agree.
Even on plant-default days in a seemingly successful implementation mentioned above, over 40% of students picked a non plant-based option. Those students would presumably not be too happy if the option were entirely removed.
“This is undoubtedly a massive opportunity to kickstart positive change at a large institutional scale with minimal cost or risk involved” … “The primary motivation of this campaign may be for universities to limit their contribution to climate change and to shift public opinion in favour of a plant-based food system”
Although I’m supportive of moves towards plant-based diets and better plant-based options being available, I wonder how confident one can be that this is truly minimal risk, especially with regards to the stated goal of shifting public opinion in favour of plant-based food systems. Do we know what proportion of students would really favour not have access to non-plant-based foods on campus (representative data, rather than petitions etc.)? If a majority are in favour, could this kind of action produce a very disgruntled minority who feel these things are being forced upon them and are resistant for the remainder of their lives to similar or other forms of animal advocacy. I’d be interested to know if there is any data/other relevant information or discussion with respect to these possible risks, and the popularity of such changes among the whole student body
Hi mcint, I’ve made some changes to the plots such that the colors are now darker for higher ratings, and the rating numbers and meaning are all positioned at the top, so it should be less open to misinterpreting what information is presented.
The ordering of the items is just in ordering of descending mean ratings.
Glad you’ve found it useful!
Indeed, the personal conversation is a bit of a black box and in open responses people gave to provide more information, most did not note specific content or arguments but rather just the type of person it was—often family members, partners, and close friends. It doesn’t necessarily seem like the conversations were in the form of people trying to ‘convince’ them or argue with them about becoming vegetarian
Hi Matt, yes indeed it could also be due to survivorship bias, or a combination of survivorship and the suggestions I made.
Not based on anything empirical but I imagine that health related reasons are probably especially vulnerable to drop out because potentially all it takes is finding some other diet that promises health benefits and the person might make a switch.
Great to hear Jacob!
Thanks for highlighting this possible source of confusion. The ′ - ′ is intended to be like a literal minus sign, but it could be read more like a spectrum with Longtermism tending left and Neartermism tending right which seems to be how you (and possibly others) read it. I’ve adapted the labeling of the graph which I hope clarifies this
Hi Irene—there is a forthcoming post that is going to be dedicated to looking at numbers per capita and also delving into possible predictors of this, so you should be able to see a thorough treatment of this soon!
Thanks Emerson, great that people are getting some value out of them
Good summary and especially helpful with the table to present the differing views succinctly. I’m looking forward to watching this debate and wasn’t aware it was out. It’s really a shame that they had some glitch that might have affected how much we can rely on the final scores to represent what really happened with the audience views
Much appreciated Howie, thanks!
We can get a better intimation of the magnitude of the effect here with some further calculations. If we take all the people who have pre and post FTX satisfaction responses (n = 951), we see that 4% of them have a satisfaction score that went up, 53% remained the same, and 43% went down. That’s quite a striking negative impact. For those people whose scores went down, 67% had a reduction of only 1 point, 22% of 2 points, and then 7%, 3%, and 1% each for −3, −4, and −5 points.
We can also try to translate this effect into some more commonly used effect size metrics. Firstly, we can utilise a nice summary effect size metric for these ratings known as probability of superiority (PSup), which makes relatively few assumptions about the data—mainly that higher ratings are higher and lower ratings are lower, within the same respondent. This metric summarises the difference over time by taking the proportion of cases in which a score was higher pre-FTX (42.7%), and assigning a 50% weight to cases in which the score was the same from pre to post FTX (.5 * 53.2% = 26.6%), and adding these quantities together (69.3%). This metric is taken as an approximation of the proportion of people who would report being more satisfied before vs. after in a forced choice of being more or less satisfied. If everyone was more satisfied before, PSup would be 100%, if everyone was more satisfied after, PSup would be 0, and if it were just as likely for people to be more or less satisfied before or after, PSup would be 50%. In this case, we get a PSup of 69.3%. This corresponds to an effect size in standard deviation units (like Cohen’s d), of approximately .7.
We would encourage people not to just look up whether these are small or large effects in a table that would say e.g, from wikipedia, that .7 is in the ‘medium’ effect size bin. Think about how you would respond on this kind of question, what a difference of 1 or more points would mean in your head, and what precisely you think the proportions of people giving different responses substantively might mean to them. How one can best interpret effect sizes varies greatly with context
https://www.samharris.org/podcasts/making-sense-episodes/361-sam-bankman-fried-effective-altruism
This podcast just came out with Will MacAskill talking about EA and SBF with Sam Harris
(edit: listening to the podcast, which I haven’t finished, I get the sense that both Sam and Will are surprisingly unaware of the enrichment of SBF, his family members, and others from FTX)