Red Hat’s JobSet POC maps distributed GPU training on OpenShift AI 3.4

Red Hat Ecosystem Engineering has published a proof of concept for distributed GPU training on Red Hat OpenShift AI 3.4. The repository brings together the JobSet Operator, Kubeflow Trainer v2 and NVIDIA GPU Operator, with an executable notebook and manifests rather than a product announcement.

The project’s latest documented revision landed Aug. 9. It is useful as a field build: a concrete map of the operators, custom resources and execution paths platform teams must connect before distributed training becomes a repeatable workload on OpenShift AI.

What the POC exercises

The repository tests two entry points into the same execution chain. A Kubeflow TrainJob can create a JobSet, or a team can submit a raw JobSet; both paths produce Kubernetes Jobs and GPU-backed pods that run PyTorch distributed data parallel through torchrun and NCCL.

Four test cases cover progressively more realistic behavior. The first validates a two-GPU collective through the TrainJob API. The second repeats the workload with a native JobSet and manual distributed configuration. The third runs a fraud-detection workload across one, two and four GPUs, including suspend and resume behavior and GPU telemetry. The fourth exercises JobSet features including multiple replicated job templates, dependencies, coordinator selection, failure and success policies, and volume-claim policies.

The sample workload uses a simulated credit-card transaction dataset and an embedding-based tabular neural network. The repository includes MinIO, data-preparation and training manifests alongside separate one-, two- and four-GPU TrainJob definitions.

The platform prerequisites matter

The POC targets OpenShift Container Platform 4.21 or later and OpenShift AI 3.4. Its installation notes call for Node Feature Discovery, the NVIDIA GPU Operator, cert-manager and the JobSet Operator. The OpenShift AI DataScienceCluster must enable the Trainer component while removing the legacy training operator; Kueue is optional for this test.

That dependency list is as important as the training code. It shows that Trainer v2 is only one layer: GPU node discovery, drivers and device plugins, webhook certificates, the JobSet controller and a cluster training runtime all have to be available before the notebook can launch work.

What to try

The shortest path is to create an OpenShift AI workbench, clone the repository and run poc-walkthrough.ipynb. Training executes on GPU worker nodes rather than inside the workbench, so the workbench itself can use a small default profile. The notebook deploys, verifies and cleans up each test case.

Teams evaluating Trainer v2 can use the POC as a validation harness, then replace the sample dataset and model while retaining the resource flow. It remains a proof of concept rather than a released product component, so production adoption still requires teams to align the example with their supported OpenShift AI configuration and operational controls.