I regularly simplify my evaluations into pros and cons lists and find them surprisingly good. Open Phil’s format essentially boils down into description, pros, cons, risks.
Check out kialo. It allows really nice nested pro con lists built of claims and supporting claims +discussion around those claims. It’s not perfect but we use it for keeping track of logic for our early stage evals / for taking a step back on evals we have gotten too in the weeds with
Do you have an example of an Open Phil report that boils down to that format? This would be an update for me. I tried my best to randomly sample reports from their website (by randomly clicking around), and the three I ended up looking at didn’t seem to follow that structure at all:
Ah, yes. I didn’t end up clicking on any of those, but I agree, that does update me a bit.
I think the biggest crux that I have is that I expect that it would be almost impossible for one person to only write the “risk” section of that article, and a separate person to only write the “case” section of the article, since they are usually framed in contrast to one another. Presenting things in the form of pro/con is very different from generating things in the form of pro/con.
My impression is you have in mind something different than what was intended in the proposal.
What I imagined was ‘priming’ the argument-mappers with prompts like
Imagine this projects fails. How?
Imagine this project works, but has some unintended bad consequences. What they are?
What would be a strong reason not to associate this project with the EA movement?
(and the opposites). When writing their texts the two people would be communicating and looking at the arguments from both sides.
The hope is this would produce more complete argument map. One way to think about it, is each person is ‘responsible’ for the pro/con section, trying to make sure it captures as much important considerations as possible.
It seems quite natural for people to think about arguments in this way, with “sides” (sometimes even single authors expose complex arguments in the “dialogue” way).
There are possible benefits—related to why ‘debate’ style is used in justice
It levels the playing field in interesting ways (when compared to public debate on the forum). In the public debate, what “counts” is not just arguments, but also discussion and social skills, status of participants, moods and emotions of the audience, and similar factor. The proposed format would mean both the positives and negatives have “advocates” ideally of “similar debate strength” (anonymous volunteer). This is very different from a public forum discussion, where all kinds of “elephant in the brain” biases may influence participants and bias judgements.
It removes some of the social costs and pains associated with project discussions. Idea authors may get discouraged by negative feedback, downvotes/karma, or similar.
Also, just looking at how discussions on the forum look now, it seems in practice it is easy for people to look at things from positive or negative perspectives: certainly I have seen arguments structured like (several different ways how something fails + why is it too costly if it succeeded + speculation what harm it may cause anyway).
Overall: in my words, I’m not sure whether your view is ‘in the space of argument-mapping, noting in the vicinity of debate, will work—at least when done by humans and applied to real problems’. Or ‘there are options in this space which are bad’ - where I agree something like bullet-pointed lists of positives and negatives where the people writing them would not communicate seems bad.
I regularly simplify my evaluations into pros and cons lists and find them surprisingly good. Open Phil’s format essentially boils down into description, pros, cons, risks.
Check out kialo. It allows really nice nested pro con lists built of claims and supporting claims +discussion around those claims. It’s not perfect but we use it for keeping track of logic for our early stage evals / for taking a step back on evals we have gotten too in the weeds with
Do you have an example of an Open Phil report that boils down to that format? This would be an update for me. I tried my best to randomly sample reports from their website (by randomly clicking around), and the three I ended up looking at didn’t seem to follow that structure at all:
https://www.openphilanthropy.org/blog/long-term-significance-reducing-global-catastrophic-risks
https://www.openphilanthropy.org/blog/breakthrough-fundamental-science
https://www.openphilanthropy.org/research/cause-reports/scientific-research/mechanisms-aging
I imagined Alex was talking about the grant reports, which are normally built around “case for the grant” and “risks”. Example: https://www.openphilanthropy.org/giving/grants/georgetown-university-center-security-and-emerging-technology
Ah, yes. I didn’t end up clicking on any of those, but I agree, that does update me a bit.
I think the biggest crux that I have is that I expect that it would be almost impossible for one person to only write the “risk” section of that article, and a separate person to only write the “case” section of the article, since they are usually framed in contrast to one another. Presenting things in the form of pro/con is very different from generating things in the form of pro/con.
My impression is you have in mind something different than what was intended in the proposal.
What I imagined was ‘priming’ the argument-mappers with prompts like
Imagine this projects fails. How?
Imagine this project works, but has some unintended bad consequences. What they are?
What would be a strong reason not to associate this project with the EA movement?
(and the opposites). When writing their texts the two people would be communicating and looking at the arguments from both sides.
The hope is this would produce more complete argument map. One way to think about it, is each person is ‘responsible’ for the pro/con section, trying to make sure it captures as much important considerations as possible.
It seems quite natural for people to think about arguments in this way, with “sides” (sometimes even single authors expose complex arguments in the “dialogue” way).
There are possible benefits—related to why ‘debate’ style is used in justice
It levels the playing field in interesting ways (when compared to public debate on the forum). In the public debate, what “counts” is not just arguments, but also discussion and social skills, status of participants, moods and emotions of the audience, and similar factor. The proposed format would mean both the positives and negatives have “advocates” ideally of “similar debate strength” (anonymous volunteer). This is very different from a public forum discussion, where all kinds of “elephant in the brain” biases may influence participants and bias judgements.
It removes some of the social costs and pains associated with project discussions. Idea authors may get discouraged by negative feedback, downvotes/karma, or similar.
Also, just looking at how discussions on the forum look now, it seems in practice it is easy for people to look at things from positive or negative perspectives: certainly I have seen arguments structured like (several different ways how something fails + why is it too costly if it succeeded + speculation what harm it may cause anyway).
Overall: in my words, I’m not sure whether your view is ‘in the space of argument-mapping, noting in the vicinity of debate, will work—at least when done by humans and applied to real problems’. Or ‘there are options in this space which are bad’ - where I agree something like bullet-pointed lists of positives and negatives where the people writing them would not communicate seems bad.