Red Hat’s AI architecture guidance makes ownership the first deployment decision

Red Hat’s latest AI architecture guidance points to a less glamorous deployment decision than model selection: deciding who owns the failure modes around the model. Read together, its four-layer architecture, hosting and Day 2 guide, and llm-d flow-control walkthrough form a responsibility map for platform teams.

The constant is the stack: compute, model storage, inference and serving, then integration. The variable is where the operating boundary sits.

Managed APIs move the boundary upward

With a managed model API, the provider normally operates compute, model artifacts and serving. The platform team avoids GPU scheduling, weight distribution and inference-engine patching, but still owns the application-facing integration layer: credentials, traffic policy, retrieval systems, audit, retries and degraded behavior.

That makes managed inference the smallest infrastructure commitment, not an operations-free choice. Provider throttling, price changes, model updates and outages remain external dependencies. If retrieval data stays inside the organization while prompts leave through an API, the data path and egress boundary must be explicit.

Decision test: choose a managed API when provider control over the first three layers is acceptable and rapid access matters more than operating the serving estate. Keep application fallbacks, retrieval stores and tool governance inside the platform plan.

Self-hosting moves responsibility down the stack

Self-hosting keeps prompts, model artifacts and serving inside infrastructure the organization controls. It also makes the platform team responsible for GPU capacity, model storage, runtime updates, replicas, autoscaling and incident response.

Red Hat’s guidance places OpenShift under that operating model, with OpenShift AI managing model lifecycle and serving. The Day 2 guide recommends promoting more than a model identifier: prompts, retrieval-index versions and agent tool configuration should move together so rollback restores one known system state.

The operational metric also changes. Average request count is not enough for capacity planning; peak concurrency, token throughput, GPU memory, cold starts and queue pressure determine whether a self-hosted endpoint meets its service objective.

Decision test: self-host when data boundaries, predictable high-volume economics or control over the runtime justify owning all four layers. Require named operators for model artifacts, serving, retrieval and agent tools before production approval.

Shared inference adds an admission-control boundary

Consolidating tenants on one GPU pool can improve utilization, but it creates a new responsibility between the gateway and vLLM. Red Hat’s llm-d guide separates admission—when a request may advance—from routing—where that request should run.

Flow control in Red Hat AI Inference 3.5 uses priority bands, tenant fairness and request ordering in the Endpoint Picker. At saturation, lower-priority work can wait while higher-priority traffic continues to dispatch. Queue limits and time-to-live settings decide when overload becomes delay, expiry or rejection.

This is not just a tuning detail. Once workloads share a pool, the platform team is defining service policy across tenants. Red Hat’s published tests use a specific model, request mix and H100 environment, and the guide says thresholds must be validated against each deployment.

Decision test: use a shared inference pool when the organization can define service objectives and acceptable shedding behavior for every workload class. Test saturation deliberately, observing both Endpoint Picker queues and vLLM queue, KV-cache and preemption metrics.

The architecture review starts with ownership

The three deployment patterns do not remove work; they relocate it. Managed APIs transfer infrastructure operations to a provider while leaving integration and continuity with the application team. Self-hosting returns compute, storage and serving to the platform boundary. Shared inference adds policy decisions about who waits when capacity runs out.

A useful architecture review should therefore name an owner, control and failure plan for each layer. If one of those cells is blank, the deployment decision is not finished—even if the model endpoint already works.