{"id":845,"date":"2026-10-03T18:22:17","date_gmt":"2026-10-03T21:22:17","guid":{"rendered":"https:\/\/www.smartservices.com.ar\/?post_type=insight&p=845"},"modified":"2026-10-03T18:22:20","modified_gmt":"2026-10-03T21:22:20","slug":"ai-agent-metrics-observability","status":"publish","type":"insight","link":"https:\/\/www.smartservices.com.ar\/en\/insights\/ai-agent-metrics-observability\/","title":{"rendered":"How to Measure an AI Agent: Metrics That Actually Matter"},"content":{"rendered":"
\n

Short answer<\/h2>\n

An AI agent can sound helpful and fail the task. It may say \u201corder created\u201d after a tool returned an error, or write a useful summary without updating the system it was meant to update. Message count and pleasant answers are not enough. Ask what work was completed correctly, with what human help, cost, and elapsed time.<\/p>\n<\/section>\n

In one sentence<\/strong>No single metric proves an agent works; track task completion, correctness, human intervention, cost and time together.<\/span><\/p>\n

\"No<\/a>
Concept diagram. Open for a larger view.<\/figcaption><\/figure>\n

Measure the work, not only the conversation<\/h2>\n

No single universal metric proves an agent works. For operational discussion, Smartservices uses five work measures: completion, correctness, intervention, cost, and time<\/strong>. This is an editorial operating model, not an industry standard. Define each measure for the specific task and its risk.<\/p>\n

The five work metrics<\/h2>\n

1. Completion<\/h3>\n

Did the task reach a verifiable end state? \u201cThe agent replied\u201d is not completion if the goal was to create an order. Define completed, incomplete, cancelled, and handed-off states. Prefer external evidence, such as an ID in the destination system.<\/p>\n

2. Correctness<\/h3>\n

Is the final result correct against criteria agreed before testing? A task may be technically \u201ccompleted\u201d with the wrong amount. Report correct tasks divided by all started tasks and correct tasks divided by completed tasks. State both denominators. Track false success separately: a convincing confirmation of an action that never occurred.<\/p>\n

3. Intervention<\/h3>\n

How many cases needed a person? Separate planned approvals, error correction, and escalation for missing context. Human involvement is not automatically failure. In a sensitive process it may be the intended control. Measure human time and decision quality as well as frequency.<\/p>\n

4. Cost<\/h3>\n

Include model calls, tools, infrastructure, reviews, and retries. Cost per conversation can hide multiple conversations needed for one job. Cost per correctly completed task is often more useful, with volume and period stated. Token expense is only one part of operating cost.<\/p>\n

5. Time<\/h3>\n

Measure from request to useful result. Separate model latency, tool wait, queues, and review. A mean can hide severe slow cases; examine percentiles and service expectations where relevant. A fast agent that creates rework may slow the business overall.<\/p>\n

Exceptions explain the headline numbers<\/h2>\n

The five measures describe outcomes. A second layer explains failures: tool errors, retries, timeouts, missing data, policy blocks, invalid actions, rollbacks, and escalations. Attach events to a task and execution stage. Avoid logging more sensitive content than diagnosis requires.<\/p>\n

Tool reliability deserves its own view. A polished model response can accompany a failed API call. A trace should connect request \u2192 plan \u2192 tool calls \u2192 action \u2192 result. Traces and logs help locate the break, but verify the business outcome in the destination system.<\/p>\n

A dashboard with honest denominators<\/h2>\n

Hypothetical example, not a Smartservices result:<\/strong> 100 comparable tasks begin. 92 reach a \u201ccompleted\u201d state; 87 are verified correct. Completion is 92\/100, correctness across started tasks is 87\/100, and correctness among completed tasks is 87\/92. Investigate the five completed-but-incorrect tasks. There are 11 human interventions, 8 retries, and 2 tool failures. These groups may overlap; do not add them as if they represent separate tasks.<\/p>\n

\n\n\n\n\n\n\n\n\n\n
Hypothetical dashboard for 100 tasks<\/caption>\n
Measure<\/th>\nResult<\/th>\nNext question<\/th>\n<\/tr>\n<\/thead>\n
Completed<\/td>\n92\/100<\/td>\nWhat blocked the other 8?<\/td>\n<\/tr>\n
Verified correct<\/td>\n87\/100; 87\/92 completed<\/td>\nWhy were 5 completed results wrong?<\/td>\n<\/tr>\n
Human intervention<\/td>\n11 tasks<\/td>\nPlanned review or repair?<\/td>\n<\/tr>\n
Retries<\/td>\n8 tasks<\/td>\nWhich tool or data caused them?<\/td>\n<\/tr>\n
Tool failures<\/td>\n2 tasks<\/td>\nWere they detected and recovered?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Another hypothetical example: 95% of responses score well in a conversational review, but only 60% of tasks finish correctly. Both can be true. Conversation quality and operational success are different measures. Do not call the 95% \u201cagent accuracy.\u201d<\/p>\n

Evaluate before launch and keep evaluating<\/h2>\n

Build a representative evaluation set: routine tasks, ambiguous requests, exceptions, missing data, and requests outside permission. Define the expected result and evidence. Compare prompt, model, and tool changes against the same cases without optimizing on the test set itself. Review failure patterns, not only one aggregate score.<\/p>\n

In production, monitor traces and outcomes as policies, documents, APIs, and case mix change. Last month’s passing evaluation does not protect against a modified integration. Version the workflow, sample outputs with people, and retain a way to stop or reverse actions where appropriate. Human intervention<\/a> should be designed, not improvised after an incident.<\/p>\n

Business results need another layer<\/h2>\n

Where measurable, add service quality, time saved, errors avoided, or team capacity. Keep business outcomes separate from agent telemetry. Lower cost per task might reflect less review or undetected mistakes. More intervention may be positive if it prevents a harmful action. No magic score substitutes for process analysis.<\/p>\n

Frequently asked questions<\/h2>\n

How do I know whether an agent works?<\/h3>\n

Define the job and external evidence of correct completion. Measure the five work metrics with clear denominators and exceptions.<\/p>\n

What is task completion rate?<\/h3>\n

Completed tasks divided by eligible started tasks. It does not imply correctness; measure that separately.<\/p>\n

How should I calculate cost per task?<\/h3>\n

Add model, tools, infrastructure, and human time for the period. Divide by correctly completed tasks if that is the unit of value, and state the scope.<\/p>\n

Is human intervention an error?<\/h3>\n

Not always. It may be a required approval. Separate planned review, exception handling, and failure repair.<\/p>\n

What does a trace show?<\/h3>\n

Execution steps and tool interactions. It helps diagnosis but does not replace checking the business result.<\/p>\n

How Smartservices uses these measures<\/h2>\n

Smartservices evaluates AI agents<\/a> as work inside a process, not isolated conversations. Our method<\/a> defines outcomes, controls, and measurement before autonomy expands. The five-measure model supports that discussion; it is not a universal standard or a performance promise.<\/p>\n

\n

Sources and references<\/h2>\n
    \n
  1. AI RMF Measure \u2014 NIST<\/a><\/li>\n
  2. GenAI semantic conventions \u2014 OpenTelemetry<\/a><\/li>\n
  3. AgentBench: Evaluating LLMs as Agents<\/a><\/li>\n<\/ol>\n<\/section>\n