A practical test matrix for finding agent failures before production

Agent evaluation becomes useful when it stops asking for one overall score. Three Red Hat engineering posts published September 3 describe different failure boundaries: whether a request should pass a guardrail, whether a model’s intended tool call survives parsing, and whether a complete multistep run produces a sound result. Put together, they suggest a practical test matrix rather than a single benchmark.

Boundary 1: policy classification

Start with the cheapest unit: a labeled prompt and an expected allow-or-block decision. Red Hat’s local EvalHub walkthrough warns that risk-only datasets reward a useless guardrail that blocks everything, so the test set must contain both safe and risky prompts.

The article’s example shows why this boundary needs explicit accuracy and latency budgets. A nuanced regex configuration scored 51.1% accuracy with 13.7 ms median latency and 20.7 ms p95 latency. A DeBERTa classifier reached 88.93% accuracy, but median latency rose to 45.2 ms and p95 to 294 ms. That supports a layered design: narrow regex checks for known patterns, followed by a broader classifier when needed. Teams should record false allows, false blocks and latency distributions, not just aggregate accuracy.

Boundary 2: tool-call transport

A policy can classify correctly while the agent still fails to act. Red Hat’s tool-calling analysis identifies parser-dependent failures across XML-style wrappers, JSON shapes and special-token formats. It also calls out mismatched field labels such as arguments versus parameters, dropped calls in multi-call responses, and reasoning text mixed into the payload.

This row of the matrix should run whenever a model, model version or serving engine changes. Use fixtures for every supported native format; verify tool name, argument values, call count and execution result; and make “no call fired” a hard failure rather than an empty success. Include malformed and partially valid payloads so that the system proves it fails loudly.

Boundary 3: trace-level behavior

The widest test asks whether the whole workflow remained correct after each component interacted. Red Hat’s OpenShift AI tutorial captures LLM calls, tool invocations and node transitions in MLflow, then uses IBM CLEAR through EvalHub to score reasoning, relevance and tool usage. CLEAR generates critiques and clusters them by recurring pattern, frequency, severity and the step where they originated.

The tutorial distinguishes a standard mode, which evaluates an agent step holistically, from SPARC mode, which separates tool calls into their own scoring rows. The latter is the better fit when transport defects may masquerade as reasoning errors. It lets a team tell whether the model chose the wrong action, the parser corrupted the right action, or a later node reasoned over an already-wrong result.

A release gate, not a dashboard

The matrix should run at three moments: on every guardrail configuration change, on every model or serving-stack change, and before release against representative end-to-end traces. Promote only when all three boundaries pass their own thresholds. A high trace score should not excuse broken parser fixtures, and a fast guardrail should not pass if it blocks legitimate work.

That separation turns evaluation into diagnosis. Policy tests say whether the system knows what to permit; transport tests say whether intent became an action; trace evaluation says whether the action remained useful inside the workflow. The useful output is not one leaderboard number, but a failure assigned to the layer that can actually fix it.