Really happy to see this get some attention. I think this is where the biggest potential value add of EA lies. Very very few groups are prepared to do work on methodological issues. Those that do seem to generally get bogged down in object level implementation details quickly (See: the output of METRICS for example.) Method work is hard, connecting people and resources to advance it is neglected.
And when groups do work on these issues there is a tendency towards infighting.
Some things that could help:
Workshops that bring people together. It’s harder to misinterpret someone’s work when they are describing it in front of you, and it’s easier to make fast progress towards a common goal (and to increase the salience of the goal).
Explicitly recognizing that the community is small and needs nurturing. It’s natural for people to at first be scared that someone else is in their coveted area (career concerns), but overall I think it might be a good thing even on a personal level. It’s such a neglected topic that if people work together and help bring attention to it real progress could be made. In contrast, sometimes you see a subfield where people are so busy tearing down each other’s work that nothing can get published or funded—a much worse equilibrium.
Bringing people together is hugely important to working constructively.
when groups do work on these issues there is a tendency towards infighting.
Do you think this is a side effect of the-one-true-ontology issues?
Do you happen to know which conferences research results in this area tend to get presented at or which journals they tend to get published in? Could be useful to bootstrap from those networks. I’ve been tracing some citation chains from highly cited stat papers, but it’s very low signal to noise for meta-research vs esoteric statistical methods.
Really happy to see this get some attention. I think this is where the biggest potential value add of EA lies. Very very few groups are prepared to do work on methodological issues. Those that do seem to generally get bogged down in object level implementation details quickly (See: the output of METRICS for example.) Method work is hard, connecting people and resources to advance it is neglected.
And when groups do work on these issues there is a tendency towards infighting.
Some things that could help:
Workshops that bring people together. It’s harder to misinterpret someone’s work when they are describing it in front of you, and it’s easier to make fast progress towards a common goal (and to increase the salience of the goal).
Explicitly recognizing that the community is small and needs nurturing. It’s natural for people to at first be scared that someone else is in their coveted area (career concerns), but overall I think it might be a good thing even on a personal level. It’s such a neglected topic that if people work together and help bring attention to it real progress could be made. In contrast, sometimes you see a subfield where people are so busy tearing down each other’s work that nothing can get published or funded—a much worse equilibrium.
Bringing people together is hugely important to working constructively.
Do you think this is a side effect of the-one-true-ontology issues?
Do you happen to know which conferences research results in this area tend to get presented at or which journals they tend to get published in? Could be useful to bootstrap from those networks. I’ve been tracing some citation chains from highly cited stat papers, but it’s very low signal to noise for meta-research vs esoteric statistical methods.
I’ll try to think about this some more. It’s a good question.