[Link] “Evaluating Arguments One Step at a Time” (Ought)

Dis­clo­sure: I do con­tract work for Ought.


https://​​ought.org/​​up­dates/​​2020-01-11-ar­gu­ments (a)

From the in­tro­duc­tion:


We’re study­ing fac­tored cog­ni­tion: un­der what con­di­tions can a group of peo­ple ac­com­plish com­plex cog­ni­tive tasks if each per­son only has min­i­mal con­text?
In a re­cent ex­per­i­ment, we fo­cused on di­vid­ing up the task of eval­u­at­ing ar­gu­ments. We cre­ated short, struc­tured ar­gu­ments for claims about movie re­views. We then tried to dis­t­in­guish valid from in­valid ar­gu­ments by show­ing each par­ti­ci­pant only one step of the ar­gu­ment, not the re­view or the other steps.
In this ex­per­i­ment, we found that:
1. Fac­tored eval­u­a­tion of ar­gu­ments can dis­t­in­guish some valid from in­valid ar­gu­ments by iden­ti­fy­ing im­plau­si­ble steps in ar­gu­ments for false claims.
2. How­ever, ex­per­i­ment par­ti­ci­pants dis­agreed a lot about whether steps were valid or in­valid. This method is there­fore brit­tle in its cur­rent form, even for ar­gu­ments which only have 1–5 steps.
3. More di­verse ar­gu­ment and ev­i­dence types (be­sides di­rect quotes from the text), larger trees, and differ­ent par­ti­ci­pant guidelines should im­prove re­sults.
In this tech­ni­cal progress up­date, we de­scribe these find­ings in depth.
No comments.