Red Hat puts trace-level agent evaluation into an OpenShift AI workflow

Red Hat has published a deployable pattern for evaluating multistep AI agents on OpenShift AI 3.4 or later. The tutorial joins MLflow trace capture, IBM’s CLEAR error-analysis framework and EvalHub’s OpenShift-native job orchestration into one workflow aimed at failures that ordinary model benchmarks miss.

What the pattern evaluates

The workflow starts with structured MLflow traces that preserve LLM calls, tool invocations and node transitions. EvalHub then schedules an OpenShift job using the IBM CLEAR adapter, which reads those traces and writes evaluation results back to MLflow.

CLEAR uses a judge model to score reasoning quality, relevance and tool use. Rather than stopping at one aggregate score, it groups critiques into recurring failure patterns with frequency, severity and attribution to the agent step where each problem occurs. The tutorial accepts any judge endpoint exposing an OpenAI-compatible chat-completions API; its examples include Gemini 2.5 Flash and a self-hosted vLLM endpoint.

Standard mode versus SPARC

The useful operational choice is whether to evaluate each step holistically or split tool calls from reasoning. In standard mode, CLEAR judges a step’s reasoning and any tool calls together. With separate_tools: true, its SPARC mode creates separate scoring rows for each function call.

That distinction gives teams a way to tell whether an agent is reasoning poorly, selecting the wrong tool or supplying malformed arguments. The tradeoff is cost and latency: Red Hat says SPARC makes roughly twice as many judge-model calls for the same traces.

How teams can use it

The tutorial deploys MLflow and EvalHub on the cluster, registers the ibm-clear provider, stores judge credentials in an OpenShift secret and submits standard and SPARC evaluations through EvalHub’s REST API. Results land in MLflow as metrics and artifacts, including structured JSON and an interactive HTML report.

For platform teams, the practical value is repeatability. Evaluation runs become OpenShift jobs rather than a manual review exercise, while MLflow workspaces provide namespace-backed multitenant isolation. Teams can compare per-agent scores across runs as prompts, retrieval or tool definitions change.

The guide recommends pinning the judge-model version so score movement reflects changes in the agent rather than an upstream model update. It also suggests limiting trace volume while iterating, then using the generated metrics and JSON artifacts as inputs to CI/CD gates.

This is still a reference workflow rather than a turnkey quality policy: teams must define acceptable thresholds and decide which judge endpoint and trace data are appropriate for their environment. But it gives OpenShift AI users a concrete path from opaque agent logs to repeatable, step-level evidence about where an agent fails.