OpenShift AI model-comparison demo proves the setup, not the model verdict

Red Hat AI Americas has published a reference build for comparing a base Gemma 3 12B instruction model with an “abliterated” variant on OpenShift AI 3.4. The useful part is not a winner declared by the repository—it does not publish benchmark results—but the infrastructure it makes inspectable: two KServe LLMInferenceService resources behind the MaaS gateway, separate OpenShift projects, per-persona access, an lm-evaluation-harness notebook and MLflow logging. The repository’s README describes the build as a reusable demo rather than a product benchmark.

What can be reproduced

The evaluation notebook sends both models through MaaS /completions endpoints using the same google/gemma-3-12b-it tokenizer, chat template and four concurrent requests. It runs MMLU and IFEval through lm_eval, extracts an accuracy or strict-match metric, logs each model’s metrics to the comparison-benchmark MLflow experiment, and prints the delta.

That is a useful operational test: teams can verify that both endpoints are reachable through the governed path, preserve run artifacts and compare scores under a shared harness. The serving manifests also make the deployment shape explicit. Each model template requests one replica, the same configurable CPU, memory and GPU resources, and the same vLLM arguments while attaching to maas-default-gateway.

Where the conclusion stops

The notebook defaults to 20 samples per task. Its “retention check” passes when the abliterated model reaches at least 95% of the base model’s score on every returned benchmark. That threshold is a demo assertion in the notebook, not a statistical test, and the repository contains no completed result files, confidence intervals or repeated runs.

The comparison therefore can reproduce a run; it cannot, as shipped, support a general claim that abliteration preserves model quality or improves safety. MMLU and IFEval cover limited dimensions, the sample cap is small, and a two-model comparison does not isolate model provenance, serving-version or run-to-run effects. Teams using the pattern for a decision should increase the sample size, record exact image and model revisions, repeat runs and add task-specific quality and safety evaluations.

What the RBAC demo proves

The RBAC template gives the data-scientist group edit access to llm and admin access to model-comparison; the AI-engineer group receives admin access only to model-alteration. The presenter flow then asks the operator to confirm that the engineer cannot see the comparison project and that the engineer’s MaaS subscription denies the altered model.

The important boundary is that Helm creates namespaces, groups and RoleBindings, but the MaaS subscriptions, authorizations and user API keys are configured manually. The access-control result is reproducible only if those post-deployment steps are completed and the denial test is actually run. That makes the project a strong platform demonstration: it connects model evaluation to identity, gateway policy, traces and GPU operations without pretending that the scaffolding itself establishes a model verdict.