A field demo turns Kubeflow Trainer v2 into a distributed fine-tuning path on OpenShift AI

Red Hat AI Americas has published a reference implementation for distributed fine-tuning on Red Hat OpenShift AI 3.4. The demo uses Orthogonal Subspace Fine-Tuning (OSFT) through Kubeflow Trainer v2, with MLflow for training metrics and vLLM on KServe for the resulting model.

The example is deliberately narrow: a banking-support router must map customer complaints to one of 77 intents and return a strict JSON decision. That makes the workflow easier to inspect than a general-purpose chatbot demonstration because the expected output has an explicit schema.

What the build connects

The repository architecture starts in an OpenShift AI workbench, submits training through the training-hub runtime and distributes the job across two nodes with two GPUs per node. Training workers and the workbench share a ReadWriteMany persistent volume, while MLflow records metrics. After the checkpoint is converted into a Hugging Face-compatible layout, the operator deploys it from the OpenShift AI interface through a vLLM serving runtime.

The supplied notebook walks through data preparation, training submission, log streaming, checkpoint comparison and endpoint testing. Supporting manifests create the namespace, service account, MLflow role binding and persistent volume claim. A custom workbench image is included to avoid installing dependencies during a live run.

Who should use it

This is most useful to platform teams evaluating OpenShift AI’s newer training stack rather than teams looking for a turnkey banking application. The documented prerequisites assume OpenShift 4.19 or later, OpenShift AI 3.4, Kubeflow Trainer v2 with the training-hub runtime, MLflow, KServe, vLLM, GPU capacity and RWX storage.

Those requirements are also the practical value of the example: it exposes the interfaces between the workbench, training service account, distributed workers, shared model storage, experiment tracker and serving runtime. The troubleshooting section calls out the resulting failure modes, including missing Trainer RBAC, unavailable training runtimes, PVC provisioning errors and vLLM rejecting a generated quantization configuration.

What to try

Teams should first reproduce the example with its small routing task and verify three boundaries: that the workbench service account can submit training jobs, that every worker can write to the shared checkpoint path, and that MLflow receives the expected run data. Only then should they substitute a larger dataset or model.

The repository was published as a field build, not a product release. Its value is as an inspectable integration pattern: it shows how OpenShift AI components can carry a fine-tuning job from notebook submission through distributed execution and into a served endpoint, while leaving the operational prerequisites visible rather than hiding them behind a managed workflow.