We’re seeking feedback on our cost-effectiveness model and scaling plan
The cost-effectiveness you mentioned is incredibly strong, which made me suspicious. “$5 per income doubling” is high, indeed.
I’ve worked in software for most of my professional life. Going through this more, I’m fairly skeptical of the inputs to your model.
Good web applications are a ton of work, even if they’re reusing AI in some way. I have a hard time picturing how much you could really get for $2M, in most settings. (Unless perhaps the founding team is working for a huge pay gap or something, but then this would change the effective cost-effectiveness).
I don’t see much discussion of marketing/distribution expenses. I’d expect these to be high.
The AI space is rapidly changing. This model takes advantage of recent developments, but doesn’t seem to assume there will be huge changes in the future. If there are, the math changes a lot. I’d expect a mix of [better competition], [the tool quickly becoming obsolete], and [the employment landscape changes so much that the income doubling becomes suspicious].
You mention the results of academic studies, but my impression is that you don’t yet have scientific results of people using your specific app. I’d be very skeptical for how much you can generalize the studies. I’d naively expect it would be difficult to motivate users to actually spend much time on the app.
In the startup world, business models in the very early stages of development are treated with tremendous suspicion. I think we have incredibly large uncertainty bounds (with lots more probability of failure), until we see some more serious use.
Overall, this write-up reminds me of a lot of what I hear by early entrepreneurs. I like the enthusiasm, but think that it’s a fair bit overoptimistic.
All that said, it’s still very possible it’s still a good opportunity. Often in the early stages, people would expect a lot of experimentation and change to the specific product.
Thanks Ozzie! I agree that it’s a bit hard to take $5 per income doubling at face value. I was tempted to revise some of my figures (eg reduce the 1⁄4 probability of ‘success’) to come up with a more ‘realistic’ figure (though arguably this undermines the very process of cost efficacy analysis if we post-hoc change the inputs to reach an output we want/find plausible). I fully agree that the uncertainty bars are large. Addressing your points in more detail: 1. It’s true, David & Edward have both made massive contributions to the coding for free. That said, both have reported that use of AI has consistently helped them achieve goals faster & easier than they had thought. 2. We were budgeting $3 per user for distribution costs (row 41 on first sheet of spreadsheet) which is roughly what my organisation’s paper-based intervention currently costs. But it obviously depends a lot on whether word-of-mouth helps it take off, or if we need to continue promoting it widely. 3. Agreed that the AI landscape is highly uncertain; for this reason we are only modelling out 5 years and not assuming any benefits post that period (perhaps we should reduce this to 3 years). More broadly, I think if/when we do reach AGI, it will require a massive rethink of everything—including most global health & development initiatives—and most cost-efficacy analyses will go out the window. Whether returns to education would increase or decrease in a post-AGI world is a fascinating question. 4. Agreed, we have not yet built a Hindi version of the app; once we have this will be a top priority to measure learning gains. The effect sizes are derived from both others’ studies of comparable EdTech and my organisation’s paper-based ALfA program. As written above, I have similar doubts about whether we can hold the user’s attention with the app—will require significant work. Fair call to be skeptical until we get some results. If/when we do manage to build the app and pilot it, will post the results here. Thanks, Tom
I went back-and-forth on this topic with Claude. I was hoping that it would re-derive my points, but getting it to provide decent criticism took a bit more time than I was expecting.
That said, I think with a few prompts (like asking it what it thought of those specific points), it was able to be useful.
Happy to see genuine attempts at this area.
The cost-effectiveness you mentioned is incredibly strong, which made me suspicious. “$5 per income doubling” is high, indeed.
I’ve worked in software for most of my professional life. Going through this more, I’m fairly skeptical of the inputs to your model.
Good web applications are a ton of work, even if they’re reusing AI in some way. I have a hard time picturing how much you could really get for $2M, in most settings. (Unless perhaps the founding team is working for a huge pay gap or something, but then this would change the effective cost-effectiveness).
I don’t see much discussion of marketing/distribution expenses. I’d expect these to be high.
The AI space is rapidly changing. This model takes advantage of recent developments, but doesn’t seem to assume there will be huge changes in the future. If there are, the math changes a lot. I’d expect a mix of [better competition], [the tool quickly becoming obsolete], and [the employment landscape changes so much that the income doubling becomes suspicious].
You mention the results of academic studies, but my impression is that you don’t yet have scientific results of people using your specific app. I’d be very skeptical for how much you can generalize the studies. I’d naively expect it would be difficult to motivate users to actually spend much time on the app.
In the startup world, business models in the very early stages of development are treated with tremendous suspicion. I think we have incredibly large uncertainty bounds (with lots more probability of failure), until we see some more serious use.
Overall, this write-up reminds me of a lot of what I hear by early entrepreneurs. I like the enthusiasm, but think that it’s a fair bit overoptimistic.
All that said, it’s still very possible it’s still a good opportunity. Often in the early stages, people would expect a lot of experimentation and change to the specific product.
Thanks Ozzie!
I agree that it’s a bit hard to take $5 per income doubling at face value. I was tempted to revise some of my figures (eg reduce the 1⁄4 probability of ‘success’) to come up with a more ‘realistic’ figure (though arguably this undermines the very process of cost efficacy analysis if we post-hoc change the inputs to reach an output we want/find plausible). I fully agree that the uncertainty bars are large.
Addressing your points in more detail:
1. It’s true, David & Edward have both made massive contributions to the coding for free. That said, both have reported that use of AI has consistently helped them achieve goals faster & easier than they had thought.
2. We were budgeting $3 per user for distribution costs (row 41 on first sheet of spreadsheet) which is roughly what my organisation’s paper-based intervention currently costs. But it obviously depends a lot on whether word-of-mouth helps it take off, or if we need to continue promoting it widely.
3. Agreed that the AI landscape is highly uncertain; for this reason we are only modelling out 5 years and not assuming any benefits post that period (perhaps we should reduce this to 3 years). More broadly, I think if/when we do reach AGI, it will require a massive rethink of everything—including most global health & development initiatives—and most cost-efficacy analyses will go out the window. Whether returns to education would increase or decrease in a post-AGI world is a fascinating question.
4. Agreed, we have not yet built a Hindi version of the app; once we have this will be a top priority to measure learning gains. The effect sizes are derived from both others’ studies of comparable EdTech and my organisation’s paper-based ALfA program. As written above, I have similar doubts about whether we can hold the user’s attention with the app—will require significant work.
Fair call to be skeptical until we get some results. If/when we do manage to build the app and pilot it, will post the results here.
Thanks,
Tom
I went back-and-forth on this topic with Claude. I was hoping that it would re-derive my points, but getting it to provide decent criticism took a bit more time than I was expecting.
That said, I think with a few prompts (like asking it what it thought of those specific points), it was able to be useful.
https://claude.ai/share/00cbbfad-6d97-4ad8-9831-5af231d36912