Hello Sanjay, thanks both for writing this up and actually having a go at building something! We did discuss this a few months ago but I can’t remember all the details of what we discussed.
First, is there a link to the bot so people can see it or use it? I can’t see one.
Second, my main question for you -sorry if I asked this before—is: what is the retention for the app? When people ask me about mental health tech, my main worry is not whether it might work if people used it, but whether people do want to use it, given the general rule that people try apps once or twice and then give up on them. If you build something people want to keep using and can provide that service cheaply, this would very likely be highly cost-effective.
I’m not sure it’s that useful to create a cost-effectiveness model based on the hypothetical scenario where people use the chatbot: the real challenge is to get people to use it. It’s a bit like me pitching a business to venture capitalists saying “if this works, it’ll be the next facebook”, to which they would say “sure, now tell us why you think it will be the next facebook”.
Third, I notice your worst-cast scenario is the effect lasts 0.5 years, but I’d expect using a chatbot to only make me feel better for a few minutes or hours, so unless people are using it many times, I’d expect the impact to be slight. Quick maths: a 1 point increase on a 0-10 happiness scale for 1 day is 0.003 happiness life-years.
Thank you very much for taking the time to have a look at this.
(1) For links to the bot, I recommend having a look at the end of Appendix 1a, where I provide links to the bot, but also explain that people who aren’t feeling low tend not to behave like real users, so it might be easier to look at one of the videos/recordings that we’ve made, which show some fictional conversations which are more realistic.
(2) Re retention, we have deliberately avoided measuring this, because we haven’t thought through whether that would count as being creepy with users’ data. We’ve also inherited some caution from my Samaritans experience, where we worry about “dependency” (i.e. people reusing the service so often that it almost becomes an addiction). So we have deliberately not tried to encourage reuse, nor measured how often it happens. We do however know that at least some users mention that they will bookmark the site and come back and reuse it. Given the lack of data, the model is pretty cautious in its assumptions—only 1.5% of users are assumed to reuse the site; everyone else is assumed to use it only once. Also, those users are not assumed to have a better experience, which is also conservative.
I believe your comments about hypotheticals and “this will be the next facebook” are based on a misunderstanding. This model is not based on the “hypothetical” scenario of people using the bot, it’s based on the scenario of people using the bot *in the same way the previous 10,000+ users have used the bot*. Thus far we have sourced users through a combination of free and paid-for Google ads, and, as described in Appendix 4a, the assumptions in the model are based on this past experience, adjusted for our expectations of how this will change in the future. The model gives no credit to the other ways that we might source users in the future (e.g. maybe we will aim for better retention, maybe we will source users from other referrals) -- those would be hypothetical scenarios, and since I had no data to base those off, I didn’t model them.
(3) I see that there is some confusion about the model, so I’ve added some links in the model to appendix 4a, so that it’s easier for people viewing the model to know where to look to find the explanations.
To respond to the specific points, the worst case scenario does *not* assume that the effect lasts 0.5 years. The worst case scenario assumes that the effect lasts a fraction of day (i.e. a matter of hours) for exactly 99.9% of users. For the remaining 0.1% of users, they are assumed to like it enough to reuse it for about a couple of weeks and then lose interest.
I very much appreciate you taking the time to have a look and provide comments. So sorry for the misunderstandings, let’s hope I’ve now made the model clear enough that future readers are able to follow it better.
Hello Sanjay, thanks both for writing this up and actually having a go at building something! We did discuss this a few months ago but I can’t remember all the details of what we discussed.
First, is there a link to the bot so people can see it or use it? I can’t see one.
Second, my main question for you -sorry if I asked this before—is: what is the retention for the app? When people ask me about mental health tech, my main worry is not whether it might work if people used it, but whether people do want to use it, given the general rule that people try apps once or twice and then give up on them. If you build something people want to keep using and can provide that service cheaply, this would very likely be highly cost-effective.
I’m not sure it’s that useful to create a cost-effectiveness model based on the hypothetical scenario where people use the chatbot: the real challenge is to get people to use it. It’s a bit like me pitching a business to venture capitalists saying “if this works, it’ll be the next facebook”, to which they would say “sure, now tell us why you think it will be the next facebook”.
Third, I notice your worst-cast scenario is the effect lasts 0.5 years, but I’d expect using a chatbot to only make me feel better for a few minutes or hours, so unless people are using it many times, I’d expect the impact to be slight. Quick maths: a 1 point increase on a 0-10 happiness scale for 1 day is 0.003 happiness life-years.
Thank you very much for taking the time to have a look at this.
(1) For links to the bot, I recommend having a look at the end of Appendix 1a, where I provide links to the bot, but also explain that people who aren’t feeling low tend not to behave like real users, so it might be easier to look at one of the videos/recordings that we’ve made, which show some fictional conversations which are more realistic.
(2) Re retention, we have deliberately avoided measuring this, because we haven’t thought through whether that would count as being creepy with users’ data. We’ve also inherited some caution from my Samaritans experience, where we worry about “dependency” (i.e. people reusing the service so often that it almost becomes an addiction). So we have deliberately not tried to encourage reuse, nor measured how often it happens. We do however know that at least some users mention that they will bookmark the site and come back and reuse it. Given the lack of data, the model is pretty cautious in its assumptions—only 1.5% of users are assumed to reuse the site; everyone else is assumed to use it only once. Also, those users are not assumed to have a better experience, which is also conservative.
I believe your comments about hypotheticals and “this will be the next facebook” are based on a misunderstanding. This model is not based on the “hypothetical” scenario of people using the bot, it’s based on the scenario of people using the bot *in the same way the previous 10,000+ users have used the bot*. Thus far we have sourced users through a combination of free and paid-for Google ads, and, as described in Appendix 4a, the assumptions in the model are based on this past experience, adjusted for our expectations of how this will change in the future. The model gives no credit to the other ways that we might source users in the future (e.g. maybe we will aim for better retention, maybe we will source users from other referrals) -- those would be hypothetical scenarios, and since I had no data to base those off, I didn’t model them.
(3) I see that there is some confusion about the model, so I’ve added some links in the model to appendix 4a, so that it’s easier for people viewing the model to know where to look to find the explanations.
To respond to the specific points, the worst case scenario does *not* assume that the effect lasts 0.5 years. The worst case scenario assumes that the effect lasts a fraction of day (i.e. a matter of hours) for exactly 99.9% of users. For the remaining 0.1% of users, they are assumed to like it enough to reuse it for about a couple of weeks and then lose interest.
I very much appreciate you taking the time to have a look and provide comments. So sorry for the misunderstandings, let’s hope I’ve now made the model clear enough that future readers are able to follow it better.