I spent about two hours looking at this in further depth and made an initial stab at modeling out the impact. I estimate an effectiveness of $200/hr (95% interval: $50/hr to $511/hr), not taking into account the value of donating the salary earned from working at Wave.
Some places where I notice we disagree or I am confused:
1.) I disagree with you here (footnote 1) that there is a 50% chance of failure (or success). I think the chance of failure could be significantly higher. From https://en.wikipedia.org/wiki/M-Pesa: M-Pesa expanded to Kenya (>10M subscribers), Tanzania (5M), South Africa (100K in a year, 1M in five years), India (???), Mozambique (???), and Lesotho (???).
Based on this, I model that a Kenya-level success (>10M subscribers) thus looks like it would have a less than 1⁄10 chance and a South Africa-level success (1-10M subscribers) looks like it would have a ~3/10 chance. However, I think this success figure could be lower due to diminishing marginal returns since M-Pesa has already plucked low hanging fruit. It’s possible better technology could increase this chance. I’d have to know more specifically about what problems M-Pesa runs into and how these are addressed.
-
2.) I think your estimate that getting M-Pesa a year earlier is only 66x worse than getting a $288 transfer from GiveDirectly is an overestimate because I expect future roll-outs will take place in countries with higher base consumption. However, as you point out, that estimate is also already an underestimate due to misunderstanding the study. I don’t know how to correct for this either way, so I used the 66x number literally in my calculation.
-
3.) I disagree with you here (footnote 1a) that marginal ETG donations are at GiveDirectly levels of cost-effectiveness. I expect them to continue at AMF levels (or greater) for at least a few more years, for reasons OpenPhil mentioned and Carl mentioned. I did an AMF-adjustment in my model for this reason.
-
4.) I really don’t know how many staff years it would take to either complete a roll-out or know that it’s not going to happen and this is an important part of the model. I currently guess 5-10 full-time staff for 2-5 years, or 10-50 total staff years. This does not count field agents or other hired locals. I couldn’t find any information on M-Pesa’s total staff count anywhere. I note that Wave has at least 44 staff (from counting faces on the about page), but I don’t know if they’re all full-time or all focused on expanding cash transfers.
-
5.) I’m confused about why GiveDirectly is stated to be 5x more cost-effective than AMF when from GiveWell’s cost-effectiveness estimate, GiveDirectly has a median of $7702 per life saved, ranging from $2200 to $16000, excluding outliers. AMF has a median of $3282 per life saved, ranging from $2200 to $4800, excluding outliers. Together, this implies a comparison centered around 2.35x but ranging from 1x to 3.3x. Maybe I misread the sheet—I haven’t invested that much time in making sure I fully understand it yet.
Re 1, it’s worth noting that M-Pesa was administered by very different teams in different countries. In Kenya it was allowed to operate mostly as a startup with limited oversight from Safaricom (or anyone else), whereas in other countries (to varying degrees) the people running M-Pesa were constrained by stricter management from the telecom’s country head. This means that there are obvious ways in which M-Pesa was executed less well in other countries. For instance, Wave integrates with M-Pesa in both Kenya and Tanzania, and despite running on exactly the same technology, the Tanzanian system’s uptime is substantially worse. Similarly, the quality of their agents and their customer support staff in Tanzania is noticeably lower.
Since Wave isn’t hamstrung by oversight from a stodgy and risk-averse telecom, I think you should give less weight to examples from countries other than Kenya as a base rate.
1) … I think the chance of failure could be significantly higher.
Possibly, but they are already starting to operate in the country in question, and my understanding is that’s been going pretty well. My impression is that they’re much more competent than Safaricom. My inside view is much higher than 50%, and getting down to 50% was a discount from there.
2) … I expect future roll-outs will take place in countries with higher base consumption
I’m confused. I was trying to talk about the counterfactual for a specific very poor country if Wave were not working there. So if future mobile money rollouts by other organizations happen first in countries with higher base consumption then that increases the counterfactual impact of Wave choosing to come into a country with very low consumption.
3) … I expect them to continue at AMF levels (or greater) for at least a few more years
4) … I really don’t know how many staff years it would take
That, combined with estimating marginal impact, makes this pretty awkward. I figure something like 40 person years?
5) … I’m confused about why GiveDirectly is stated to be 5x more cost-effective than AMF
This comes from cell F31 of the “Results” tab. I haven’t put time into understanding how that’s calculated, but it looked like the relevant bottom line number.
1) … I think the chance of failure could be significantly higher.
Possibly, but they are already starting to operate in the country in question, and my understanding is that’s been going pretty well. My impression is that they’re much more competent than Safaricom. My inside view is much higher than 50%, and getting down to 50% was a discount from there.
Sounds like you definitely have inside info that I don’t have, so for now I’d have to rely on my outside view, but I can work to acquire that inside info if I look into this more.
-
2) … I expect future roll-outs will take place in countries with higher base consumption
I’m confused. I was trying to talk about the counterfactual for a specific very poor country if Wave were not working there. So if future mobile money rollouts by other organizations happen first in countries with higher base consumption then that increases the counterfactual impact of Wave choosing to come into a country with very low consumption.
I don’t know what country Wave is looking at or how they are doing what they do because you have inside info that I don’t have. If it has consumption comparable to Kenya than my point is invalid. I just was concerned that it wouldn’t.
-
3) … I expect them to continue at AMF levels (or greater) for at least a few more years
Cool. Sounds like this isn’t a disagreement between us then.
-
4) … I really don’t know how many staff years it would take
That, combined with estimating marginal impact, makes this pretty awkward. I figure something like 40 person years?
Agreed that it is pretty awkward to estimate. I modified my model to use some of your inputs—such as a 40% chance of 1-10M subscribers and a 10% chance of >10M subscribers and 40 person years—and it comes out to $383/hr (95%: $145/hr to $834/hr). The new mean is still in my old 95% interval which is about the best I can hope for with this level of uncertainty.
-
5) … I’m confused about why GiveDirectly is stated to be 5x more cost-effective than AMF
This comes from cell F31 of the “Results” tab. I haven’t put time into understanding how that’s calculated, but it looked like the relevant bottom line number.
Oh, I see that now. I suppose this is a question for GiveWell and not you. I’ll ask them.
Sounds like you definitely have inside info that I don’t have, so for now I’d have to rely on my outside view, but I can work to acquire that inside info if I look into this more.
If you’re interested in working for Wave, or are advising other people on whether it’s a good idea for them, I could imagine they’d be quite interested in talking to you!
if it has consumption comparable to Kenya than my point is invalid. I just was concerned that it wouldn’t.
I spent about two hours looking at this in further depth and made an initial stab at modeling out the impact. I estimate an effectiveness of $200/hr (95% interval: $50/hr to $511/hr), not taking into account the value of donating the salary earned from working at Wave.
Some places where I notice we disagree or I am confused:
1.) I disagree with you here (footnote 1) that there is a 50% chance of failure (or success). I think the chance of failure could be significantly higher. From https://en.wikipedia.org/wiki/M-Pesa: M-Pesa expanded to Kenya (>10M subscribers), Tanzania (5M), South Africa (100K in a year, 1M in five years), India (???), Mozambique (???), and Lesotho (???).
Also, a 2016 Vodaphone press-release suggested M-Pesa seems to have 25M customers worldwide after 10 years of effort.
Based on this, I model that a Kenya-level success (>10M subscribers) thus looks like it would have a less than 1⁄10 chance and a South Africa-level success (1-10M subscribers) looks like it would have a ~3/10 chance. However, I think this success figure could be lower due to diminishing marginal returns since M-Pesa has already plucked low hanging fruit. It’s possible better technology could increase this chance. I’d have to know more specifically about what problems M-Pesa runs into and how these are addressed.
-
2.) I think your estimate that getting M-Pesa a year earlier is only 66x worse than getting a $288 transfer from GiveDirectly is an overestimate because I expect future roll-outs will take place in countries with higher base consumption. However, as you point out, that estimate is also already an underestimate due to misunderstanding the study. I don’t know how to correct for this either way, so I used the 66x number literally in my calculation.
-
3.) I disagree with you here (footnote 1a) that marginal ETG donations are at GiveDirectly levels of cost-effectiveness. I expect them to continue at AMF levels (or greater) for at least a few more years, for reasons OpenPhil mentioned and Carl mentioned. I did an AMF-adjustment in my model for this reason.
-
4.) I really don’t know how many staff years it would take to either complete a roll-out or know that it’s not going to happen and this is an important part of the model. I currently guess 5-10 full-time staff for 2-5 years, or 10-50 total staff years. This does not count field agents or other hired locals. I couldn’t find any information on M-Pesa’s total staff count anywhere. I note that Wave has at least 44 staff (from counting faces on the about page), but I don’t know if they’re all full-time or all focused on expanding cash transfers.
-
5.) I’m confused about why GiveDirectly is stated to be 5x more cost-effective than AMF when from GiveWell’s cost-effectiveness estimate, GiveDirectly has a median of $7702 per life saved, ranging from $2200 to $16000, excluding outliers. AMF has a median of $3282 per life saved, ranging from $2200 to $4800, excluding outliers. Together, this implies a comparison centered around 2.35x but ranging from 1x to 3.3x. Maybe I misread the sheet—I haven’t invested that much time in making sure I fully understand it yet.
Re 1, it’s worth noting that M-Pesa was administered by very different teams in different countries. In Kenya it was allowed to operate mostly as a startup with limited oversight from Safaricom (or anyone else), whereas in other countries (to varying degrees) the people running M-Pesa were constrained by stricter management from the telecom’s country head. This means that there are obvious ways in which M-Pesa was executed less well in other countries. For instance, Wave integrates with M-Pesa in both Kenya and Tanzania, and despite running on exactly the same technology, the Tanzanian system’s uptime is substantially worse. Similarly, the quality of their agents and their customer support staff in Tanzania is noticeably lower.
Since Wave isn’t hamstrung by oversight from a stodgy and risk-averse telecom, I think you should give less weight to examples from countries other than Kenya as a base rate.
Possibly, but they are already starting to operate in the country in question, and my understanding is that’s been going pretty well. My impression is that they’re much more competent than Safaricom. My inside view is much higher than 50%, and getting down to 50% was a discount from there.
I’m confused. I was trying to talk about the counterfactual for a specific very poor country if Wave were not working there. So if future mobile money rollouts by other organizations happen first in countries with higher base consumption then that increases the counterfactual impact of Wave choosing to come into a country with very low consumption.
See http://www.jefftk.com/p/leaving-google-joining-wave#fb-835897806972_835943804792
That, combined with estimating marginal impact, makes this pretty awkward. I figure something like 40 person years?
This comes from cell F31 of the “Results” tab. I haven’t put time into understanding how that’s calculated, but it looked like the relevant bottom line number.
Thanks Jeff!
-
Sounds like you definitely have inside info that I don’t have, so for now I’d have to rely on my outside view, but I can work to acquire that inside info if I look into this more.
-
I don’t know what country Wave is looking at or how they are doing what they do because you have inside info that I don’t have. If it has consumption comparable to Kenya than my point is invalid. I just was concerned that it wouldn’t.
-
Cool. Sounds like this isn’t a disagreement between us then.
-
Agreed that it is pretty awkward to estimate. I modified my model to use some of your inputs—such as a 40% chance of 1-10M subscribers and a 10% chance of >10M subscribers and 40 person years—and it comes out to $383/hr (95%: $145/hr to $834/hr). The new mean is still in my old 95% interval which is about the best I can hope for with this level of uncertainty.
-
Oh, I see that now. I suppose this is a question for GiveWell and not you. I’ll ask them.
If you’re interested in working for Wave, or are advising other people on whether it’s a good idea for them, I could imagine they’d be quite interested in talking to you!
It’s poorer than Kenya.
That sounds pretty awesome, who do you think would be a good person to reach out to when I’m ready?
Ben Kuhn maybe?