The cost of tokens used as a percentage of your salary is the correct metric at NVidia and other firms.
As far as I can tell, any activity performed with a chatbot is deemed to be a) productive and b) more productive than any conceivable activity not talking to a chatbot
My company was using token burn as an AI adoption metric, but they’ve started clawing that back in search of better metrics.
After a bunch of people all blew through $2000 of tokens over a weekend building “something”, leadership cooled their heels and is pivoting to trying to use AI more intelligently.
And yet we can still measure productivity by how fast it takes to get a feature deployed and we can measure customer satisfaction… I still don’t understand why that’s insufficient. If adding AI made the line go up, then it is a benefit. If line not go up, why buy AI?
Instead it seems like a lot of companies are just looking for new magic numbers to chase in order to justify their predetermined conclusions.
Instead it seems like a lot of companies are just looking for new magic numbers to chase in order to justify their predetermined conclusions.
That describes politics everywhere. Just people trying to find numbers that they can point to that “show” they are doing a good job, with tons of confirmation bias.
At a high level I think it’s similar to integration of any other vendor. ROI calculation based on cost of vendor and value of costs reduced and/or revenue gained post integration. Companies may have some granular attribution models to say X investment in AI project was directly tied to Y outcomes valued at $Z.
Some execs are admitting that AI is costing more than humans.
each org will have its own KPI but from what I have seen from our customers many of them try to track how much employee time has been saved by handing work off to the “ai tooling”
Wouldn’t that mean that KPIs need to be applied to the output, just as they are to employees’ work? Otherwise, no comparison would be possible at all: To measure how much time an employee saves by using AI, one must measure the value of the AI’s output. If you don’t do that, the time savings could be attributed to any other factor - such as the employee working overtime because they don’t want to be replaced by AI, which in this scenario is simply not possible because, due to the lack of metrics, it is unknown whether AI can replace the employee at all.
Given the hype and the types of people that push this shit, I can’t imagine any fruitful KPIs being applied. A good KPI should be hard/impossible to game, regularly tracked, and should have understood limits or action triggers (i.e. what result from a KPI is acceptable vs needs improvement).
Had a yearly review where one of my metrics was decreasing NCRs. The numbers showed more than a 25% reduction in NCRs from the previous year when my goal was like 5%. Of course this was over covid and that weren’t accounting for volume of orders which plummeted that year. On paper I did fucking phenomenal.
Generally the only one I know of is for use. Technically, the KPI should be seeing increases in worker productivity, but a lot of jobs don’t have meaningful KPI’s.
I have to agree. My boss technically looks at one KPI for me and my team, but it’s a badly lagging indicator and I can only influence less than half of it.
His real KPI for me is how many times he gets called by someone complaining about me or asking me to reassign my priorities to what they think is important. The lower the better.
When I used to work at IBM, we always had to quantify various aspects of our job. If you can’t measure the job, how do you do job performance evaluations?
It ends up being qualitative based on roles and responsibilities.
How so? Even managers have quantitative kpi’s.
https://corporatefinanceinstitute.com/resources/data-science/ai-kpis-tracking-performance/
For Operations:
Reduction in manual processing time – AI accelerates loan approval review, compliance checks, and reporting, freeing finance teams for higher-value analysis.
Increase in automated transactions – AI-driven trading, payments, and credit risk models process thousands of transactions in seconds.
Lower error rates – AI minimizes costly reporting errors, compliance mistakes, and financial miscalculations.
These measurements are all over the internet.
This place is as good as any to recommend the book “The Tyranny of Metrics” by Jerry Z Muller.
If by historically you mean the last 80 years or so, you’re using words wrong.
It’s been almost 40 years since the late 1980s. That’s certainly a historical context, especially since it has had such a profound impact on the world we live in today - not for the better, if you ask me.
