As a data scientist who frequently wonders about this question, thank you for this post, it was a really interesting read!
But Iâm a bit confused about your final section on âgenerative AIâ roles, and the claim that data people have no special advantage here. What sort of roles do you have in mind exactly? If youâre talking only about prompt engineering, or just about the skill of being able to use a product like Claude code/âcowork well, then I can see where youâre coming from, though I think I still weakly disagree. For example, if you have an understanding of how these models work under the hood then you are probably better placed to understand which tasks it will be good at and which it might struggle with (this includes really basic stuff, like knowledge cutoffs in training data, and the fact it wonât necessarily remember what you asked it about the day before without a memory feature bolted on, and therefore wonât improve day by day etc). You are also probably in a better position to understand things like which tasks will require higher reasoning, and which wonât (which very often doesnât line up with their perceived human difficulty).
But that aside, I donât think applying generative AI to practical problems is just about prompt engineering or becoming a claude cowork super-user. If you are a business that wants to create an automated workflow where you run a certain prompt over a certain dataset at scale, on the basis of certain triggers, then you are doing something that looks a lot like data engineering (and you are going to need to write code). And if youâre hitting context window limits, youâll have to manage context, by building some kind of agent orchestration/âscaffolding. Thatâs maybe closer to being a brand new skill, but it still requires strong coding ability, combined with the ability to evaluate the performance of a software tool empirically, which is a specific combination of skillsets that data scientists should already have.
Itâs true that with AI coding assistants, people are writing less and less code by hand. But you still need strong coding ability to use AI coding assistants effectively. Software engineering has not yet been fully automated. Thereâs a possible future where that changes soon, and maybe thatâs the future informing your advice in that final section? But my personal feeling would be that in that world, most knowledge jobs probably follow shortly after anyway (if not before), so trying to plan your career development around this possible future seems challenging!
As a data scientist who frequently wonders about this question, thank you for this post, it was a really interesting read!
But Iâm a bit confused about your final section on âgenerative AIâ roles, and the claim that data people have no special advantage here. What sort of roles do you have in mind exactly? If youâre talking only about prompt engineering, or just about the skill of being able to use a product like Claude code/âcowork well, then I can see where youâre coming from, though I think I still weakly disagree. For example, if you have an understanding of how these models work under the hood then you are probably better placed to understand which tasks it will be good at and which it might struggle with (this includes really basic stuff, like knowledge cutoffs in training data, and the fact it wonât necessarily remember what you asked it about the day before without a memory feature bolted on, and therefore wonât improve day by day etc). You are also probably in a better position to understand things like which tasks will require higher reasoning, and which wonât (which very often doesnât line up with their perceived human difficulty).
But that aside, I donât think applying generative AI to practical problems is just about prompt engineering or becoming a claude cowork super-user. If you are a business that wants to create an automated workflow where you run a certain prompt over a certain dataset at scale, on the basis of certain triggers, then you are doing something that looks a lot like data engineering (and you are going to need to write code). And if youâre hitting context window limits, youâll have to manage context, by building some kind of agent orchestration/âscaffolding. Thatâs maybe closer to being a brand new skill, but it still requires strong coding ability, combined with the ability to evaluate the performance of a software tool empirically, which is a specific combination of skillsets that data scientists should already have.
Itâs true that with AI coding assistants, people are writing less and less code by hand. But you still need strong coding ability to use AI coding assistants effectively. Software engineering has not yet been fully automated. Thereâs a possible future where that changes soon, and maybe thatâs the future informing your advice in that final section? But my personal feeling would be that in that world, most knowledge jobs probably follow shortly after anyway (if not before), so trying to plan your career development around this possible future seems challenging!