Reisinger.Tech

Stop measuring AI adoption and focus on outcomes

As a software engineer at a large company, I have access to the latest and best AI tools available on the market at no cost to me. As I gain experience with these tools, and as they continue to mature, I’m starting to get a handle on this new way of working. Like many others I have become a code steward - a full-time reviewer. I’m also increasingly aware that I am inept at describing what I want the tool to accomplish. In fact, this has always been the difficult part of software engineering in my career. Most of my time “coding” is actually spent looking at things - observing, testing, reading documentation, and looking at others’ code.

As a senior engineer, I don’t spend most of my time “coding” though. I spend most of it planning, documenting, and reviewing other people’s work. I’m not currently a manager, but as one it would be easy to soak in the numbers of “AI compliance” and feel as though a team is making progress. But, now, more than ever, it is easier to write the wrong code than it has ever been. The extra time we saved by using an AI to write code for us - where did it go? Did it go into planning, documenting, and reviewing instead? Well, unfortunately, no - it’s going into code review and correcting the AI when it hallucinates.

This is ok I guess - I certainly had moments before the LLM revolution where I coded the wrong thing for a day, only to find out later - after, you know, talking to a real person - that a completely different approach was necessary or that my understanding of the problem was askew and I needed to start over again. The same is true with AI - it often takes multiple iterations to get something right. What’s changed is that I no longer have a complete mental model of the technical approach to solve the problem before tackling it - I delegate this responsibility to the AI, in the hope that it can save me from some hard thinking. But often, the AI needs to be guided and corrected, which means I end up doing the hard thinking after all.

If we still care about delighting people with our work, making customers happy and delivering value, we should measure those things hard. Measuring AI adoption is not a proxy for true value creation; it’s a shareholder panic indicator.