Stop Measuring AI by How Much You Use It
Usage is the easiest number to grow and the hardest one to defend.
Open the dashboard and there it is, glowing green and climbing. Seats activated. Queries this month. Tokens consumed. Adoption, up and to the right, the most beautiful direction a line can travel. Everyone is using it. The number is huge. You feel, for a moment, like the future is being handled. That feeling is the most expensive thing in the building, because the number underneath it is a lie that tells the truth. People really are using it. It really does mean nothing.
Usage measures motion, and motion is the most convincing disguise that progress ever wore. A treadmill is extraordinarily busy. It also takes you nowhere, beautifully, at any speed you like.
Usage measures activity. It says nothing about whether any of it mattered. You cannot eat activity.
Why the wrong number is always the popular one
Usage wins because usage is easy, and easy metrics breed like rabbits. You can count a query without understanding it. You can chart a token without ever asking what it bought. Vanity metrics are mirrors that only show your good side, and we keep them on the wall because the reflection is flattering and the truth is a lot of work. Value hides exactly where it is hardest to count, which is not a reason to count something else. It is the reason the people who do the counting get paid.
A tool can be used constantly and change nothing. Your team can run ten thousand queries this quarter and ship the exact same work they would have shipped without the tool, only now with a bill attached. Tokens are receipts, not results. And if the only thing that went up this year is the invoice, you did not adopt AI. You subscribed to it.
Measure the thing you actually wanted
You never wanted usage. Nobody ever wanted usage. You wanted the things usage was supposed to produce, and somewhere along the way you started worshipping the smoke instead of the fire.
These are harder to count, which is precisely why people count usage instead. But you become what you measure, so measure carefully, or you will drown in the wrong thing while congratulating yourself for the depth. Adoption is not transformation, the same way a gym membership is not a six-pack and a full bookshelf is not a working mind.
Turn the tool off for a week and watch what breaks. If the answer is "nothing, but the dashboard looks sad," you never had value. You had a habit with a subscription fee.
A busy tool is not a useful tool. A loud metric is not a meaningful one. And a number that only ever goes up was probably never telling you the truth.
"But usage is a leading indicator"
Fair, and worth conceding fully. Outcomes are slow and noisy, and in the first weeks of a rollout you genuinely cannot measure them yet. Usage is the early pulse. Zero usage is a real and useful alarm: if nobody touched the tool, nothing good is coming. So in the beginning, yes, watch the usage.
The trouble is what happens next. A leading indicator only earns its keep if you eventually demand the lagging one it was supposed to predict. Most organisations never make that second move, because outcomes are hard and usage is easy, so the early pulse quietly becomes the permanent scoreboard. And the moment usage becomes the target, Goodhart's Law arrives: when a measure becomes a target, it stops being a good measure. People start generating activity to feed the number, and the dashboard glows green while nothing actually changes. Use usage as a thirty-day pulse. Then make it prove itself against an outcome, or stop trusting it.
Where this lands
Stop measuring how much you use it. Start measuring what changed because you did. If nothing changed, the usage was never the point. It was the alibi.
I write about AI, data, and learning regularly at pinaldave.com, and I have been teaching this hands-on in my AI workshops.