When we eventually told the cash arm participants that we had given other households assets of the same value, most said they would have preferred the assets, “We don’t have good products to buy here”. We had also originally planned to work in 2 countries but ended up working in just 1, freeing up enough budget to pay for cash.
I’m intuitively drawn to cash transfer arms, but “just ask the participants what they would want” also sounds very compelling for basically the same reasons. Ideally you could do that both before and after (“would you recommend other families take the cash or the asset?”)
Have you done or seen systematic analysis along these lines? How do you feel about that idea?
Asking about the comparison to cash also seems like a reasonable way to do the comparison even if you were running both arms (i.e. you could ask both groups whether they’d prefer $X or asset Y, and get some correction for biases to prefer/disprefer the option they actually received).
Maybe direct comparison surveys also give you a bit more hope of handling timing issues, depending on how biased you think participants are by recency effects. If you give someone an asset that pays off over multiple years, I do expect their “cash vs asset” answers to change over time. But still people can easily imagine getting the cash now and so if nothing else it seems like a strong sanity check if you ask asset-recipients in 2 years and confirm they prefer the asset.
At a very basic intuitive level, hearing “participants indicated strong preference for receiving our assets to receiving twice as much cash” feels more persuasive than comparing some measured outcome between the two groups (at least for this kind of asset transfer program where it seems reasonable to defer to participants about what they need/want)
I really like the idea of asking people what assets they would like. We did do a version of this to determine what products to offer, using qualitative interviews where people ranked ~30 products in order of preference. This caused us to add more chickens and only offer maize inputs to people who already grew maize. But participants had to choose from a narrow list of products (those with RCT evidence that we could procure), I’d love have given them freedom to suggest anything.
We did also consider letting households determine which products they received within a fixed budget (rather than every household getting the same thing) but the logistics got too difficult. Interestingly, people had zero interest in deworming pills, oral hydration salts or Vitamin A supplements as they not were aware of needing them—I could see tensions arising between households not valuing these kinds of products and donors wanting to give them based on cost-effectiveness models. This “what do you want” approach might work best with products that recipients already have reasonably accurate mental models of, or that can be easily and accurately explained.
At a very basic intuitive level, hearing “participants indicated strong preference for receiving our assets to receiving twice as much cash” feels more persuasive than comparing some measured outcome between the two groups (at least for this kind of asset transfer program where it seems reasonable to defer to participants about what they need/want)
Very interesting suggestion: we did try something like this but didn’t consider it as an outcome measure and so didn’t put proper thought/resources into it. We asked people, “How much would you be willing to pay for product X?”, with the goal of saying something like “Participants valued our $120 bundle at $200″ but unfortunately the question generally caused confusion: participants would think we were asking them to pay for the product they’d received for free and either understandably got upset or just tried lowballing us with their answer, expecting it to be a negotiation.
If we had thought of it in advance, perhaps this would have worked as a way to generate real value estimates:
We randomise participants into groups
The first group is offered either our bundle (worth $120) or $120 cash
If >50% take the bundle, we then adjust our cash offer upwards and offer it to another group (or the opposite if <50% take the bundle)
We repeat this process of adjusting our price offer until we have ~50% of participants accepting our cash offer: that equilibrium price is then the “value” of the bundle
I’m not sure what the sample size implications would be but a big advantage would be the timeline: this could be done in a few weeks, not years
I bet proper economists have a way to do this but it’s interesting to brainstorm on
I can see a few issues with this:
We still have to assume people “know what’s best for them”: that’s a really patronising statement, but as above, people need reliable mental models to make these decisions, and won’t have that for novel products
We need to have donors who will accept this measure: the data only really matters when it informs decisions
I’m very open to other thoughts here, I really like the concept of a cash benchmark and would love to find a way to resurrect the idea.
I’m intuitively drawn to cash transfer arms, but “just ask the participants what they would want” also sounds very compelling for basically the same reasons. Ideally you could do that both before and after (“would you recommend other families take the cash or the asset?”)
Have you done or seen systematic analysis along these lines? How do you feel about that idea?
Asking about the comparison to cash also seems like a reasonable way to do the comparison even if you were running both arms (i.e. you could ask both groups whether they’d prefer $X or asset Y, and get some correction for biases to prefer/disprefer the option they actually received).
Maybe direct comparison surveys also give you a bit more hope of handling timing issues, depending on how biased you think participants are by recency effects. If you give someone an asset that pays off over multiple years, I do expect their “cash vs asset” answers to change over time. But still people can easily imagine getting the cash now and so if nothing else it seems like a strong sanity check if you ask asset-recipients in 2 years and confirm they prefer the asset.
At a very basic intuitive level, hearing “participants indicated strong preference for receiving our assets to receiving twice as much cash” feels more persuasive than comparing some measured outcome between the two groups (at least for this kind of asset transfer program where it seems reasonable to defer to participants about what they need/want)
I really like the idea of asking people what assets they would like. We did do a version of this to determine what products to offer, using qualitative interviews where people ranked ~30 products in order of preference. This caused us to add more chickens and only offer maize inputs to people who already grew maize. But participants had to choose from a narrow list of products (those with RCT evidence that we could procure), I’d love have given them freedom to suggest anything.
We did also consider letting households determine which products they received within a fixed budget (rather than every household getting the same thing) but the logistics got too difficult. Interestingly, people had zero interest in deworming pills, oral hydration salts or Vitamin A supplements as they not were aware of needing them—I could see tensions arising between households not valuing these kinds of products and donors wanting to give them based on cost-effectiveness models. This “what do you want” approach might work best with products that recipients already have reasonably accurate mental models of, or that can be easily and accurately explained.
Very interesting suggestion: we did try something like this but didn’t consider it as an outcome measure and so didn’t put proper thought/resources into it. We asked people, “How much would you be willing to pay for product X?”, with the goal of saying something like “Participants valued our $120 bundle at $200″ but unfortunately the question generally caused confusion: participants would think we were asking them to pay for the product they’d received for free and either understandably got upset or just tried lowballing us with their answer, expecting it to be a negotiation.
If we had thought of it in advance, perhaps this would have worked as a way to generate real value estimates:
We randomise participants into groups
The first group is offered either our bundle (worth $120) or $120 cash
If >50% take the bundle, we then adjust our cash offer upwards and offer it to another group (or the opposite if <50% take the bundle)
We repeat this process of adjusting our price offer until we have ~50% of participants accepting our cash offer: that equilibrium price is then the “value” of the bundle
I’m not sure what the sample size implications would be but a big advantage would be the timeline: this could be done in a few weeks, not years
I bet proper economists have a way to do this but it’s interesting to brainstorm on
I can see a few issues with this:
We still have to assume people “know what’s best for them”: that’s a really patronising statement, but as above, people need reliable mental models to make these decisions, and won’t have that for novel products
We need to have donors who will accept this measure: the data only really matters when it informs decisions
I’m very open to other thoughts here, I really like the concept of a cash benchmark and would love to find a way to resurrect the idea.