Red Hat traces OpenShift AI agent answers from model calls to tool output with MLflow

Red Hat has published a practical walkthrough for tracing agentic AI requests with MLflow on Red Hat OpenShift AI. The example matters because it treats an incorrect-looking answer as a distributed-systems investigation rather than assuming the language model hallucinated.

What the trace reveals

The walkthrough uses a mortgage-lending assistant whose answer disagrees with figures on an executive dashboard. MLflow groups the request into one distributed trace containing model calls, a tool invocation, token and latency metrics, and agent log events. In Red Hat’s example, the first model call chooses an action, the ceo_pipeline_summary tool returns internal data, and a second model call composes the response.

The timing view attributes almost all of the roughly five-second request to the two model calls; the tool itself completes in 75 milliseconds. Opening the tool span then shows that the assistant received the same figures it reported. That narrows the fault from a vague “AI got it wrong” complaint to a concrete disagreement between the tool’s data and the dashboard—potentially a query, cache, definition, or date-range issue.

This is the useful distinction for platform teams. Conventional API telemetry can show whether an endpoint succeeded and how long it took. Agent tracing connects the user-visible answer to the prompts, model calls, retrieved context, tool parameters, outputs, and timing that produced it. Red Hat says MLflow supports OpenTelemetry semantics and automatic tracing integrations for agent frameworks and tools.

The OpenShift AI reference application

The accompanying Red Hat AI Quickstart is a deployable multi-agent mortgage application rather than a toy chat window. It defines five persona-specific LangGraph agents, role-scoped tools, Keycloak identity, PostgreSQL with pgvector, append-only audit events, PII masking, and Helm deployment to OpenShift.

For OpenShift AI 3.4 and later, the quickstart documents MLflow workspace configuration and Kubernetes RBAC resources for client authentication. The Helm chart can create a service account, cluster role, and binding; operators then provide an MLflow route, experiment, workspace, and tracking token. The application correlates its audit trail with MLflow traces.

What teams should try

AI application teams can use the quickstart to test three operational questions before production: whether a trace can reconstruct a disputed answer, whether latency is attributable to models versus tools, and whether trace access is isolated to the intended workspace and service account.

The demo does not prove that observability makes an agent correct. It shows something more concrete: when an answer is wrong, traces can identify which component supplied the bad input and give the responsible team a reproducible starting point.