Event-driven operator automates OpenShift must-gather collection after cluster warnings

Red Hat Developer has published an engineering walkthrough of an open-source operator designed to preserve failure evidence from OpenShift and Kubernetes fleets before recovery work overwrites it. The event-driven diagnostic operator watches a central hub cluster for warning events, chooses a diagnostic payload and launches a Kubernetes Job to collect data from the affected managed cluster.

The project addresses a familiar incident-response gap: by the time an engineer starts a manual must-gather, volatile logs or system state may already be gone. The operator is intended to trigger collection when an incident begins, persist the output away from the diagnostic pod and clean up the temporary workload later.

How the event becomes a diagnostic job

The operator’s public repository documents a rule-based path from event to evidence. An EventReconciler filters for Kubernetes events with Type=Warning; predefined regular expressions then map messages such as etcd corruption or CNI failure to specific must-gather images. Events that match no rule are skipped.

For a match, the controller identifies the spoke cluster from the involved object, namespace or message text. It copies that cluster’s kubeconfig secret into the operator namespace, then creates an independent Kubernetes Job. The job mounts ReadWriteMany storage and writes diagnostic output to a cluster-specific directory. Kubernetes’ TTL controller removes the completed job while the collected logs remain available for later analysis.

That division of labor is the useful design choice: the hub-side controller returns to watching events instead of running a long collection inline. The project targets hub-and-spoke environments such as those managed with Red Hat Advanced Cluster Management, where a single controller may need to react to failures across many clusters.

What platform teams need to evaluate

This is an engineering project, not a newly announced OpenShift or ACM product capability. The repository currently places diagnostic rules in Go source, uses placeholder image references that operators must replace, and requires cluster-wide event access, managed-cluster kubeconfig secrets and an RWX storage class. Those requirements make the security and operational review consequential: the controller handles privileged credentials and may create diagnostic workloads in response to broadly generated warning events.

The current implementation limits unnecessary work by collecting only for messages that match configured patterns. The author says CRD-based dynamic rule management is planned, while possible AI-assisted root-cause analysis is described only as a future direction. For now, the concrete contribution is narrower and testable: event-triggered capture of must-gather evidence, offloaded to disposable jobs with persistent storage.