In terms of understanding the causal effect of talking to journalists, it seems hard to say much in the absence of an RCT.
Someone ought to flip a coin for every interview request, in order to measure (a) the causal effect of accepting an interview on probability of article publication, and (b) the direction of any effects on article accuracy, fairness, and useful critique.
(That was meant as a bit of a joke, but I would honestly be delighted to see a bunch of articles about EA which include sentences like “Person X did not offer any comment because we weren’t assigned to the interview acceptance group in their RCT”. Seems like it sends the right signal to the sort of people we want to attract.)
In any case, until that RCT gets run, maybe it would be worthwhile to compare articles informed by interviews and articles uninformed by interviews side-by-side, and do what we can with the data we have. It’s easy to say “I talked to the journalist and the article was inaccurate”. But claiming that the article ended up worse than it would’ve been in the absence of an interview is harder. (There are also complicating factors: an article with quotes from relevant people may seem more legitimate to readers; no interview might mean no article.)
EA has a pathology of insisting that we defer to data even in situations where sufficient quantities of data can’t be practically collected before a decision is necessary.
And that is extremely relevant to EA’s media problem.
Say it takes 100 datapoints over 10 years to make an informed decision. During that time:
The media ecosystem, the character of the discourse, the institutions (there are now prediction markets involved btw) and the dominant moral worldviews of the audience has completely changed, you no longer need the answer to the question this data answers.
You have already been assassinated for not engaging in a principled and decisive way.
I should assume that I’m talking to someone who has this pathology and needs me to explain what the alternative to “defer to data” even is: Get better at interpreting the data you already have. Seek theories of communication that’re general enough and robust enough that you don’t strictly need to collect further data to validate them. Test them anyway, but you can’t wait for the tests to conclude before deploying.
You make good points, but there’s no boolean that flips when “sufficient quantities of data [are] practically collected”. The right mental model is closer to a multi-armed bandit IMO.
In terms of understanding the causal effect of talking to journalists, it seems hard to say much in the absence of an RCT.
Someone ought to flip a coin for every interview request, in order to measure (a) the causal effect of accepting an interview on probability of article publication, and (b) the direction of any effects on article accuracy, fairness, and useful critique.
(That was meant as a bit of a joke, but I would honestly be delighted to see a bunch of articles about EA which include sentences like “Person X did not offer any comment because we weren’t assigned to the interview acceptance group in their RCT”. Seems like it sends the right signal to the sort of people we want to attract.)
In any case, until that RCT gets run, maybe it would be worthwhile to compare articles informed by interviews and articles uninformed by interviews side-by-side, and do what we can with the data we have. It’s easy to say “I talked to the journalist and the article was inaccurate”. But claiming that the article ended up worse than it would’ve been in the absence of an interview is harder. (There are also complicating factors: an article with quotes from relevant people may seem more legitimate to readers; no interview might mean no article.)
It is a joke, but it’s an appropriate one.
EA has a pathology of insisting that we defer to data even in situations where sufficient quantities of data can’t be practically collected before a decision is necessary.
And that is extremely relevant to EA’s media problem.
Say it takes 100 datapoints over 10 years to make an informed decision. During that time:
The media ecosystem, the character of the discourse, the institutions (there are now prediction markets involved btw) and the dominant moral worldviews of the audience has completely changed, you no longer need the answer to the question this data answers.
You have already been assassinated for not engaging in a principled and decisive way.
I should assume that I’m talking to someone who has this pathology and needs me to explain what the alternative to “defer to data” even is: Get better at interpreting the data you already have. Seek theories of communication that’re general enough and robust enough that you don’t strictly need to collect further data to validate them. Test them anyway, but you can’t wait for the tests to conclude before deploying.
You make good points, but there’s no boolean that flips when “sufficient quantities of data [are] practically collected”. The right mental model is closer to a multi-armed bandit IMO.
That might at least be a good way of establishing a lower bound for EV from talking to journalists.