I also conduct research on the generalizability issue, but from a different perspective. In my view, any attempt to measure effect heterogeneity (and by extension, research generalizability) is scale dependent. It is very difficult to tease apart genuine effect heterogeneity from the appearance of heterogeneity due to using an inappropriate scale to measure the effects.
In order to to get around this, I have constructed a new scale for measuring effects, which I believe is more natural than the alternative measures. My work on this is available on arXiv at https://arxiv.org/abs/1610.00069 . The paper has been accepted for publication at the journal Epidemiologic Methods, and I plan to post a full explanation of the idea here and on Less Wrong when it is published (presumably, this will be a couple of weeks from now).
I would very much appreciate feedback on this work, and as always, I operate according to Crocker’s Rules.
I think counterfactual outcome state transition parameters is a bad name in that it doesn’t help people identify where and why they should use it, nor does it communicate all that well what it really is. I’d want to thesaurus each of the key terms in order to search for something punchier. You might object that essentially ‘marketing’ an esoteric statistics concept seems perverse, but papers with memorable titles do in fact outperform according to the data AFAIK. Sucks but what can you do?
I bother to go into this because this research area seems important enough to warrant attention and I worry it won’t get it.
Thank you! I will think about whether I can come up with a catchier name for future publications (and about whether the benefits outweight the costs of rebranding).
If anyone has suggestions for a better name (for an effect measure that intuitively measures the probability that the exposure switches a person’s outcome state), please let me know!