This was interesting to read, especially the point about the marginal cost of automating additional tasks falling as the underlying infrastructure gets better. One thing I was wondering about is how the maintenance side is developing as you add more automations.
At 11% you can probably still understand each automation fairly well, but if the aim is to automate a much larger share of program operations, some of the work presumably shifts from doing the task itself to keeping a growing set of automations working as forms, Drive structures, naming conventions, staff roles etc. change. I don’t know if you are already measuring this, but something like human minutes per run or per program round, including review, exceptions and repairs, might be useful alongside the percentage of tasks automated. It could help distinguish automations that really reduce operational load from ones that mostly move that load somewhere less visible.
This is an interesting point, and no, we’re not measuring that exactly. The main measure we use is how much we get done with the same resources: either more output for the same time (like the referrals example, roughly 3-4x for about the same time spent), or being able to take on additional work at all (like the Talent Directory Automated Matching, we have something exciting coming up at HIP that’s a mix of both).
On maintenance: if this is about changing underlying systems (tracking new data, therefore needing a new integration), that would be an addition to the manual work also. And when those systems change, that already broke the manual version. The drop-out example in the post touches 15 systems, and we did have a written list, which was often slightly out of date because the program kept evolving. What’s different now is that Claude can flag when there’s something else we might want to link, because it’s already made the same change somewhere similar, rather than us trying to remember system no 16.
You are right, measuring AI maintenance burden without accounting for maintenance that had to be done regardless of AI deployment would not get you anywhere.
Interesting you are have Claude identifying opportunities to simplify/reuse/streamline, this could actually mean the per system maintenance burden will go down the more systems you connect. Pretty much the opposite of what I was thinking. Thank you.
This was interesting to read, especially the point about the marginal cost of automating additional tasks falling as the underlying infrastructure gets better. One thing I was wondering about is how the maintenance side is developing as you add more automations.
At 11% you can probably still understand each automation fairly well, but if the aim is to automate a much larger share of program operations, some of the work presumably shifts from doing the task itself to keeping a growing set of automations working as forms, Drive structures, naming conventions, staff roles etc. change. I don’t know if you are already measuring this, but something like human minutes per run or per program round, including review, exceptions and repairs, might be useful alongside the percentage of tasks automated. It could help distinguish automations that really reduce operational load from ones that mostly move that load somewhere less visible.
This is an interesting point, and no, we’re not measuring that exactly. The main measure we use is how much we get done with the same resources: either more output for the same time (like the referrals example, roughly 3-4x for about the same time spent), or being able to take on additional work at all (like the Talent Directory Automated Matching, we have something exciting coming up at HIP that’s a mix of both).
On maintenance: if this is about changing underlying systems (tracking new data, therefore needing a new integration), that would be an addition to the manual work also. And when those systems change, that already broke the manual version. The drop-out example in the post touches 15 systems, and we did have a written list, which was often slightly out of date because the program kept evolving. What’s different now is that Claude can flag when there’s something else we might want to link, because it’s already made the same change somewhere similar, rather than us trying to remember system no 16.
You are right, measuring AI maintenance burden without accounting for maintenance that had to be done regardless of AI deployment would not get you anywhere.
Interesting you are have Claude identifying opportunities to simplify/reuse/streamline, this could actually mean the per system maintenance burden will go down the more systems you connect. Pretty much the opposite of what I was thinking. Thank you.