This is super interesting stuff thanks for posting!
The first thing that jumped out at me was that you are reading and analysing people’s messages that come through the chatbot. I’m sure they consented (as much as this is possible to truly consent with the level of education, and the cash incentive) and its all anonymised but it still seems weird.
I have so many ethical questions about this. None of them I think necessarily mean something like this isn’t worth trying, but I think it’s worth discussing. Here’s just a couple off the top of my head
What do you do if they have a conversation about harming themselves or others? Do you react and do something about it or do you leave it be? Would people then be aware that what they type could illicit some kind of external response?
When they ask something like “”What business has quick profits” for which there is obviously no good answer, what does the bot do? I hope it doesn’t try and give business advice. When I asked Claude sonnet, the first answer it gave was
“Poultry farming is one of the most cited options. It has high demand for eggs and chicken meat, with startup capital of around 1–2.5 million RWF and potential returns in 2–3 months if well managed.”
In many rural contexts this might be a decent idea, but without proper disease treatment, housing, protection from theft etc. this advice could be a huge liability.
Also Who are you that you answer me?” is pretty haunting. I concur.
I think there’s a huge amount to be gained potentially by trying chatbots in these settings, but its a bit of an ethical minefield and its a new fronteir for sure.
Hi NickLaing, you raise excellent points—this does raise ethical issues and it’s something that we have thought a lot about and have put systems in place to address. We do gather consent from recipients when they are initially enrolled, and we ensure that data is anonymised when it is analysed. We have built additional railguards into the chatbots (we tested exactly the types of questions you raised to check how the chatbot responded and then ensured that any questions that raised a red flag receive responses that link them to our call centre). Our safeguarding teams also monitor messages for anything problematic and address these with the appropriate level of engagement depending on the issues that surface. Our approach to safeguarding and guardrails is evolving as we learn—if you have additional suggestions of things we should be thinking about, we’d welcome these suggestions.
This is super interesting stuff thanks for posting!
The first thing that jumped out at me was that you are reading and analysing people’s messages that come through the chatbot. I’m sure they consented (as much as this is possible to truly consent with the level of education, and the cash incentive) and its all anonymised but it still seems weird.
I have so many ethical questions about this. None of them I think necessarily mean something like this isn’t worth trying, but I think it’s worth discussing. Here’s just a couple off the top of my head
What do you do if they have a conversation about harming themselves or others? Do you react and do something about it or do you leave it be? Would people then be aware that what they type could illicit some kind of external response?
When they ask something like “”What business has quick profits” for which there is obviously no good answer, what does the bot do? I hope it doesn’t try and give business advice. When I asked Claude sonnet, the first answer it gave was
“Poultry farming is one of the most cited options. It has high demand for eggs and chicken meat, with startup capital of around 1–2.5 million RWF and potential returns in 2–3 months if well managed.”
In many rural contexts this might be a decent idea, but without proper disease treatment, housing, protection from theft etc. this advice could be a huge liability.
Also Who are you that you answer me?” is pretty haunting. I concur.
I think there’s a huge amount to be gained potentially by trying chatbots in these settings, but its a bit of an ethical minefield and its a new fronteir for sure.
Hi NickLaing, you raise excellent points—this does raise ethical issues and it’s something that we have thought a lot about and have put systems in place to address. We do gather consent from recipients when they are initially enrolled, and we ensure that data is anonymised when it is analysed. We have built additional railguards into the chatbots (we tested exactly the types of questions you raised to check how the chatbot responded and then ensured that any questions that raised a red flag receive responses that link them to our call centre). Our safeguarding teams also monitor messages for anything problematic and address these with the appropriate level of engagement depending on the issues that surface. Our approach to safeguarding and guardrails is evolving as we learn—if you have additional suggestions of things we should be thinking about, we’d welcome these suggestions.