Thanks very much for posting this; I enjoyed reading it recently, and think it’s good to share to the forum for easier discussion.
For questions that the evaluators do not know how to settle, one plausible option would be to judge overconfident statements as negligent (e.g. “Having a high minimum wage does not reduce employment.”) but allow all sufficiently unconfident statements (e.g. “Minimum wage laws do not seem to substantially reduce employment in most places they are implemented. However, there are many people who disagree with my interpretation of the evidence.”).
I’m not sure your suggestion—that controversial suggestions be caveated—is sufficient, because it is often controversial whether a given statement is controversial! To partisans, disagreement might appear to be in bad faith, to be ‘hate speech’ etc., and hence even the caveated version rejected.
he track record of contemporary truth-checking institutions similarly gives us cause for concern. A salient example is the pandemic, where various social media platforms have taken to either suppressing or labeling as misleading statements that contradict the views of local health authorities or the WHO—even though these authorities were themselves often mistaken.
Similarly, the track record of fact-checkers seems quite poor: they seem to have considerable discretion in what ratings they give statements. As an example, here are two statements that PoliFact recently evaluated by Presidents Trump and Biden:
In both cases that statements, literally interpreted, are making a universal claim: that were no cases of guns, and that there are no possibility of transmission. More colloquially, we might think they both just meant ‘very few cases’ of guns/transmission; I can see arguments for either interpretation. What I cannot see a credible case for is the actual approach they took, which is to interpret Trump’s claim strictly, and hence false because of a small number of anecdotes, but Biden’s in the more colloquial sense, and hence merely guilty of exaggeration.
This issue seems rife for contemporary fact-checkers: through judicious selection of which statements to verify, which experts to consult, and which standards of evidence to apply, they have a huge amount of discretion to make favoured politicians appear truthful and disfavoured ones appear dishonest.
Another example is Ought’s work on truth through debate: they found it very difficult to produce good truth-forcing mechanisms in the face of non-truth-motivated actors.
Re. the particulars of fact-checkers and discretion, I’m in favour of more precise processes for assessing possible meanings of ambiguous statements and then assessing the truth of those possible meanings. I think that this could remove quite a bit of the subjectivity.
In the case of the example you give, I would like to give Biden’s statement a medium penalty, and Trump’s statement a medium-large penalty. The difference is Trump’s use of the word “whatsoever”. This is the opposite of a caveat—it is stressing that the literal meaning rather than the approximate one is intended. To my mind pairs of comparably-bad statements would be:
Not bad:
Guns
“There were very few guns …”
“For the most part, there were no guns …”
Coronavirus
″… are less likely to spread it to you”
″… cannot spread it to you in most cases”
Somewhat bad:
“There were no guns …”
″… cannot spread it to you”
More bad (but still room to be more false):
Guns
“There were no guns whatsoever …”
“There were absolutely no guns …”
Coronavirus
″… absolutely cannot spread it to you”
″… can never spread it to you”
This is not to say that political bias isn’t playing a role in how these organisations are functioning at the moment, but I do think that we can hope to establish more precise standards which reduces the scope for bias to apply.
To my mind these have basically identical meanings: expressing that something is not physically possible. This is actually stronger that simplying saying it hasn’t happened. Consider:
I will not go to Liverpool this year (very likely true)
I will absolutely not go to Liverpool this year (very likely true)
I cannot go to Liverpool this year (false)
So if anything I would expect this analysis to point in the opposite direction.
Some content which didn’t make it into the paper in the end but is relevant for this discussion is a draft protocol for “counting microlies” (the coloured text is the instructions, to be read counterclockwise starting in the top left):
The idea is that one statement which is definitely false seems a much more egregious violation of truthfulness than e.g. four statements each only 75% true.
Raising it to a power >1 is a factor correcting for this. The choice of four is a best guess based on thinking through a few examples and how bad things seemed, but I’m sure it’s not an optimal choice for the parameter.
In general if we’re asking about what has a “poor” track record, it would be good to think about quantification and comparison to alternatives. Note that we’d consider sites like Wikipedia as examples of institutions doing a form of truth evaluation.
Discussions of fact-checking institutions often focus on some concrete case that they got wrong; but they are bound to get some things wrong. The questions are :
What’s the overall track record over all statements (including those that seem easy/obvious)?
How well do they do against alternatives?
Analogously people often point out some particular cases where prediction markets did badly, but advocates of prediction markets just claim that they are at least as accurate over all as alternative prediction mechanisms. And right now many questions humans ask are not controversial (e.g. science questions, local questions). But AI currently says false things about these questions! So there’s lots of room for improvement without even touching the controversial stuff (though eventually one wants some relatively graceful handling of controversy).
Thanks very much for posting this; I enjoyed reading it recently, and think it’s good to share to the forum for easier discussion.
I’m not sure your suggestion—that controversial suggestions be caveated—is sufficient, because it is often controversial whether a given statement is controversial! To partisans, disagreement might appear to be in bad faith, to be ‘hate speech’ etc., and hence even the caveated version rejected.
he track record of contemporary truth-checking institutions similarly gives us cause for concern. A salient example is the pandemic, where various social media platforms have taken to either suppressing or labeling as misleading statements that contradict the views of local health authorities or the WHO—even though these authorities were themselves often mistaken.
Similarly, the track record of fact-checkers seems quite poor: they seem to have considerable discretion in what ratings they give statements. As an example, here are two statements that PoliFact recently evaluated by Presidents Trump and Biden:
Trump: “There were no guns whatsoever” at the Capitol riot on Jan. 6.
Rated FALSE
Biden: People who are vaccinated for the coronavirus “cannot spread it to you.”
Rated HALF TRUE
In both cases that statements, literally interpreted, are making a universal claim: that were no cases of guns, and that there are no possibility of transmission. More colloquially, we might think they both just meant ‘very few cases’ of guns/transmission; I can see arguments for either interpretation. What I cannot see a credible case for is the actual approach they took, which is to interpret Trump’s claim strictly, and hence false because of a small number of anecdotes, but Biden’s in the more colloquial sense, and hence merely guilty of exaggeration.
This issue seems rife for contemporary fact-checkers: through judicious selection of which statements to verify, which experts to consult, and which standards of evidence to apply, they have a huge amount of discretion to make favoured politicians appear truthful and disfavoured ones appear dishonest.
Another example is Ought’s work on truth through debate: they found it very difficult to produce good truth-forcing mechanisms in the face of non-truth-motivated actors.
Re. the particulars of fact-checkers and discretion, I’m in favour of more precise processes for assessing possible meanings of ambiguous statements and then assessing the truth of those possible meanings. I think that this could remove quite a bit of the subjectivity.
In the case of the example you give, I would like to give Biden’s statement a medium penalty, and Trump’s statement a medium-large penalty. The difference is Trump’s use of the word “whatsoever”. This is the opposite of a caveat—it is stressing that the literal meaning rather than the approximate one is intended. To my mind pairs of comparably-bad statements would be:
Not bad:
Guns
“There were very few guns …”
“For the most part, there were no guns …”
Coronavirus
″… are less likely to spread it to you”
″… cannot spread it to you in most cases”
Somewhat bad:
“There were no guns …”
″… cannot spread it to you”
More bad (but still room to be more false):
Guns
“There were no guns whatsoever …”
“There were absolutely no guns …”
Coronavirus
″… absolutely cannot spread it to you”
″… can never spread it to you”
This is not to say that political bias isn’t playing a role in how these organisations are functioning at the moment, but I do think that we can hope to establish more precise standards which reduces the scope for bias to apply.
What distinction are you drawing between
cannot spread it to you
and
can never spread it to you?
To my mind these have basically identical meanings: expressing that something is not physically possible. This is actually stronger that simplying saying it hasn’t happened. Consider:
I will not go to Liverpool this year (very likely true)
I will absolutely not go to Liverpool this year (very likely true)
I cannot go to Liverpool this year (false)
So if anything I would expect this analysis to point in the opposite direction.
The distinction I’m drawing is that “cannot spread it to you” is ambiguous between whether it’s shorthand for:
Cannot (in any circumstances) spread it to you
Cannot (as a rule of thumb) spread it to you
Whereas I think that “can never spread it to you” or “absolutely cannot spread it to you” are harder to interpret as being shortenings of 2.
Some content which didn’t make it into the paper in the end but is relevant for this discussion is a draft protocol for “counting microlies” (the coloured text is the instructions, to be read counterclockwise starting in the top left):
(Unimportant: Why is falsity raised to the fourth power?)
The idea is that one statement which is definitely false seems a much more egregious violation of truthfulness than e.g. four statements each only 75% true.
Raising it to a power >1 is a factor correcting for this. The choice of four is a best guess based on thinking through a few examples and how bad things seemed, but I’m sure it’s not an optimal choice for the parameter.
In general if we’re asking about what has a “poor” track record, it would be good to think about quantification and comparison to alternatives. Note that we’d consider sites like Wikipedia as examples of institutions doing a form of truth evaluation.
Discussions of fact-checking institutions often focus on some concrete case that they got wrong; but they are bound to get some things wrong. The questions are :
What’s the overall track record over all statements (including those that seem easy/obvious)?
How well do they do against alternatives?
Analogously people often point out some particular cases where prediction markets did badly, but advocates of prediction markets just claim that they are at least as accurate over all as alternative prediction mechanisms. And right now many questions humans ask are not controversial (e.g. science questions, local questions). But AI currently says false things about these questions! So there’s lots of room for improvement without even touching the controversial stuff (though eventually one wants some relatively graceful handling of controversy).
(Thanks to Owain for most of these points.)