Red Hat’s EvalHub walkthrough turns guardrail tuning into a measurable trade-off

A new Red Hat Developer walkthrough shows how to move LLM guardrail testing beyond hand-picked prompts and into a repeatable local benchmark. The September 3 article uses EvalHub with a community NeMo Guardrails provider to compare a nuanced regular-expression filter with a DeBERTa classifier against labeled prompt-injection data.

What the benchmark exposes

The example’s regex configuration reached 51.1% accuracy. Its allowed-prompt recall was high at 96.66%, but its blocked-prompt recall was only 16.53%, meaning it missed most prompts the benchmark expected it to stop. The measured median decision latency was 13.7 milliseconds.

The DeBERTa configuration reached 88.93% accuracy, with 82.27% blocked-prompt recall and 97.72% allowed-prompt recall. That improvement came with a median latency of 45.2 milliseconds and a 294-millisecond p95. The article’s useful contribution is not simply that one approach scored higher: it makes the safety-versus-latency trade-off visible in the same evaluation workflow.

The dataset design is equally important. Red Hat’s example mixes prompts that should pass with prompts that should be blocked. A risk-only dataset can reward a useless guardrail that rejects everything, so accuracy must be considered alongside the system’s ability to preserve legitimate use. EvalHub reports separate precision, recall and F1 values for allowed and blocked classes, as well as errors and latency.

How to reproduce it

The walkthrough points to a supporting repository and a setup script that creates a local environment and starts EvalHub. Users then invoke evalhub eval run with a benchmark name, the NeMo provider and a selected guardrail configuration. Results are retrieved by job identifier after the run finishes.

EvalHub supplies built-in NeMo benchmarks for prompt injection and jailbreaks, and for toxicity and profanity. Their dataset mappings identify the prompt column, label column and which labels should be blocked or allowed. Teams can use those mappings as a template for their own labeled data rather than assuming a public benchmark represents their application traffic.

What platform teams should carry forward

The article proposes layering fast, targeted regex checks with a slower classifier for cases that need broader detection. That is a defensible starting pattern, but teams should rerun the measurement with their own prompts, failure costs and latency budget before adopting the sample thresholds.

The local workflow is also a useful pre-deployment gate for OpenShift AI. It lets developers reject weak guardrail configurations before they become a shared service, while preserving the benchmark definition and metrics needed for later comparison. A planned third part will address deployment at scale; this installment is focused on proving that a configuration works before promoting it.