I have been feeling the siren song of agent-based models recently (I think it seems a natural move in a lot of cases, because we are actually modelling agents), but your criticisms of them reminded me that they often don’t pay for their complexity in better predictions. It seems quite a general and useful point, and perhaps could be extracted to a standalone post, if you had the time and inclination.
I know it wasn’t a major area of focus for you, but do you have a vague impression of when randomisation might be a big win purely by reducing costs of evaluation? One particular case where it might be useful is funders where disbursement is bottlenecked by evaluation capacity. Do you have any pointers for useful places to start research on the idea?
do you have a vague impression of when randomisation might be a big win purely by reducing costs of evaluation?
Not really I’m afraid. I’d expect that due to the risk of inadvertent negative impacts and large improvements from weeding out obviously suboptimal options a pure lottery will rarely be a good idea. How much effort to expend beyond weeding out clearly suboptimal options to me likely seems to depend on contextual information specific to the use case. I’m not sure how much there is to be said in general except for platitudes along the lines of “invest time into explicit evaluation until the marginal value of information has diminished sufficiently”.
on the greaterwrong version of the EA forum, there’s an automatically generated TOC. So that’s an option for people who would strongly prefer a TOC
I have been feeling the siren song of agent-based models recently (I think it seems a natural move in a lot of cases, because we are actually modelling agents), but your criticisms of them reminded me that they often don’t pay for their complexity in better predictions. It seems quite a general and useful point, and perhaps could be extracted to a standalone post, if you had the time and inclination.
I know it wasn’t a major area of focus for you, but do you have a vague impression of when randomisation might be a big win purely by reducing costs of evaluation? One particular case where it might be useful is funders where disbursement is bottlenecked by evaluation capacity. Do you have any pointers for useful places to start research on the idea?
Not really I’m afraid. I’d expect that due to the risk of inadvertent negative impacts and large improvements from weeding out obviously suboptimal options a pure lottery will rarely be a good idea. How much effort to expend beyond weeding out clearly suboptimal options to me likely seems to depend on contextual information specific to the use case. I’m not sure how much there is to be said in general except for platitudes along the lines of “invest time into explicit evaluation until the marginal value of information has diminished sufficiently”.