This is excellent. It will definitely inform the way I work with RCTs from now on.
I would just like to quibble your use of the word “discount”. In most of the post you use it synonymously with “multiplier” (ie. a 60% discount to account for publication bias means you would multiply the experimental result by 0.6). However in your final worked example under External Validity you apply “discounts” of 20%, 20% and 5% for necessary conditions, special care effects and general equilibrium effects respectively, with the calculation 100/(100+20+20+5)*100% = 74%. This alternative definition took me some time to get my head around.
I’d also expect to see the 20%, 20%, 5% combined as multipliers, since these effects act independently: 1/(1.2*1.2*1.05) *100% = 66% (although I realise the final result isn’t too different in this case).
Thanks Stan! This is really helpful- agreed with you that they should be combined as multipliers rather than added together (I’ve now edited accordingly). I’m still mulling a bit over whether using the word ‘discount’ or ‘adjustment’ or something else might help improve clarity.
This is excellent. It will definitely inform the way I work with RCTs from now on.
I would just like to quibble your use of the word “discount”. In most of the post you use it synonymously with “multiplier” (ie. a 60% discount to account for publication bias means you would multiply the experimental result by 0.6). However in your final worked example under External Validity you apply “discounts” of 20%, 20% and 5% for necessary conditions, special care effects and general equilibrium effects respectively, with the calculation 100/(100+20+20+5)*100% = 74%. This alternative definition took me some time to get my head around.
I’d also expect to see the 20%, 20%, 5% combined as multipliers, since these effects act independently: 1/(1.2*1.2*1.05) *100% = 66% (although I realise the final result isn’t too different in this case).
Thanks Stan! This is really helpful- agreed with you that they should be combined as multipliers rather than added together (I’ve now edited accordingly). I’m still mulling a bit over whether using the word ‘discount’ or ‘adjustment’ or something else might help improve clarity.