Short answer
An AI agent can sound helpful and fail the task. It may say “order created” after a tool returned an error, or write a useful summary without updating the system it was meant to update. Message count and pleasant answers are not enough. Ask what work was completed correctly, with what human help, cost, and elapsed time.
In one sentenceNo single metric proves an agent works; track task completion, correctness, human intervention, cost and time together.
Measure the work, not only the conversation
No single universal metric proves an agent works. For operational discussion, Smartservices uses five work measures: completion, correctness, intervention, cost, and time. This is an editorial operating model, not an industry standard. Define each measure for the specific task and its risk.
The five work metrics
1. Completion
Did the task reach a verifiable end state? “The agent replied” is not completion if the goal was to create an order. Define completed, incomplete, cancelled, and handed-off states. Prefer external evidence, such as an ID in the destination system.
2. Correctness
Is the final result correct against criteria agreed before testing? A task may be technically “completed” with the wrong amount. Report correct tasks divided by all started tasks and correct tasks divided by completed tasks. State both denominators. Track false success separately: a convincing confirmation of an action that never occurred.
3. Intervention
How many cases needed a person? Separate planned approvals, error correction, and escalation for missing context. Human involvement is not automatically failure. In a sensitive process it may be the intended control. Measure human time and decision quality as well as frequency.
4. Cost
Include model calls, tools, infrastructure, reviews, and retries. Cost per conversation can hide multiple conversations needed for one job. Cost per correctly completed task is often more useful, with volume and period stated. Token expense is only one part of operating cost.
5. Time
Measure from request to useful result. Separate model latency, tool wait, queues, and review. A mean can hide severe slow cases; examine percentiles and service expectations where relevant. A fast agent that creates rework may slow the business overall.
Exceptions explain the headline numbers
The five measures describe outcomes. A second layer explains failures: tool errors, retries, timeouts, missing data, policy blocks, invalid actions, rollbacks, and escalations. Attach events to a task and execution stage. Avoid logging more sensitive content than diagnosis requires.
Tool reliability deserves its own view. A polished model response can accompany a failed API call. A trace should connect request → plan → tool calls → action → result. Traces and logs help locate the break, but verify the business outcome in the destination system.
A dashboard with honest denominators
Hypothetical example, not a Smartservices result: 100 comparable tasks begin. 92 reach a “completed” state; 87 are verified correct. Completion is 92/100, correctness across started tasks is 87/100, and correctness among completed tasks is 87/92. Investigate the five completed-but-incorrect tasks. There are 11 human interventions, 8 retries, and 2 tool failures. These groups may overlap; do not add them as if they represent separate tasks.
| Measure | Result | Next question |
|---|---|---|
| Completed | 92/100 | What blocked the other 8? |
| Verified correct | 87/100; 87/92 completed | Why were 5 completed results wrong? |
| Human intervention | 11 tasks | Planned review or repair? |
| Retries | 8 tasks | Which tool or data caused them? |
| Tool failures | 2 tasks | Were they detected and recovered? |
Another hypothetical example: 95% of responses score well in a conversational review, but only 60% of tasks finish correctly. Both can be true. Conversation quality and operational success are different measures. Do not call the 95% “agent accuracy.”
Evaluate before launch and keep evaluating
Build a representative evaluation set: routine tasks, ambiguous requests, exceptions, missing data, and requests outside permission. Define the expected result and evidence. Compare prompt, model, and tool changes against the same cases without optimizing on the test set itself. Review failure patterns, not only one aggregate score.
In production, monitor traces and outcomes as policies, documents, APIs, and case mix change. Last month’s passing evaluation does not protect against a modified integration. Version the workflow, sample outputs with people, and retain a way to stop or reverse actions where appropriate. Human intervention should be designed, not improvised after an incident.
Business results need another layer
Where measurable, add service quality, time saved, errors avoided, or team capacity. Keep business outcomes separate from agent telemetry. Lower cost per task might reflect less review or undetected mistakes. More intervention may be positive if it prevents a harmful action. No magic score substitutes for process analysis.
Frequently asked questions
How do I know whether an agent works?
Define the job and external evidence of correct completion. Measure the five work metrics with clear denominators and exceptions.
What is task completion rate?
Completed tasks divided by eligible started tasks. It does not imply correctness; measure that separately.
How should I calculate cost per task?
Add model, tools, infrastructure, and human time for the period. Divide by correctly completed tasks if that is the unit of value, and state the scope.
Is human intervention an error?
Not always. It may be a required approval. Separate planned review, exception handling, and failure repair.
What does a trace show?
Execution steps and tool interactions. It helps diagnosis but does not replace checking the business result.
How Smartservices uses these measures
Smartservices evaluates AI agents as work inside a process, not isolated conversations. Our method defines outcomes, controls, and measurement before autonomy expands. The five-measure model supports that discussion; it is not a universal standard or a performance promise.