Red Hat’s internal AI gateway shifted most engineering traffic to open models

Red Hat says an internal model-serving platform has reversed the mix of AI traffic used by its engineering organization. Consumption that began 70/30 in favor of commercial frontier APIs has shifted to 30/70, with self-hosted open-weight models now carrying most routine coding and evaluation work.

The Engineering Model-as-a-Service pilot, or EnMaaS, processed more than one trillion tokens in a single 30-day period across Red Hat’s global engineering organization. Red Hat reports effective token-cost reductions of 40% to 70% compared with its previous frontier-model pricing, depending on prompt-cache efficiency and routing.

One front door, two serving paths

EnMaaS presents engineers with one SSO-backed portal and a unified API key. Behind it, the Praxis gateway routes requests either to commercial APIs or to internally hosted models served through llm-d and vLLM.

Praxis is a Rust-based proxy framework contributed to the CNCF in September. Its data plane performs prompt inspection, body-based routing, API translation and content guardrails in one process rather than a chain of proxies and sidecars. A policy engine contributed by IBM researchers applies security and safety rules.

The gateway also accounts for tokens and spend by user, team and model. Administrators can set rate limits and monthly budgets, or share capacity across business units. When a team exhausts its commercial-model allowance, Praxis can route work to a hosted open model instead of stopping IDE completion or agent workflows; limits may be configured as reporting-only or enforced controls.

llm-d supplies the distributed backend

For the hosted path, llm-d coordinates model-server fleets and uses live load, predicted latency and prefix affinity for routing. It tiers KV-cache data across GPU, CPU and NVMe storage, and supports prefill/decode disaggregation and multi-node expert parallelism. Red Hat says its deployment serves roughly a dozen open models across hundreds of GPUs.

The pilot’s adoption result is more useful than a simple cost comparison. Developers moved toward open models after the organization removed separate accounts and incompatible onboarding flows, suggesting that gateway friction can influence model choice as strongly as benchmark scores.

Red Hat also places OpenShell around agents in its internal software-development lifecycle to isolate access to files, credentials and external networks.

The caveat for other platform teams

These are Red Hat’s self-reported internal results, not an independent benchmark, and the savings depend on workload mix, cache behavior and infrastructure costs. Still, the architecture offers a concrete pattern: centralize identity and accounting at the gateway, preserve access to multiple model classes, and route routine work to owned capacity without making developers change tools.

Organizations considering the design should measure end-to-end cost, including GPU idle time and gateway operations, while tracking whether fallback routing changes response quality for the tasks being moved.