TL;DR — The chart is three real resources: a ServingRuntime running vLLM, a KServe InferenceService pointing at a PersistentVolumeClaim, and the PVC itself. It looks small. It still took two real debugging sessions to get a pod to Ready: a placeholder image digest that was never meant to survive contact with a real cluster, and a single values.yaml field — model.path — silently doing the job of two paths that must never be equal. The chart, the fixes, and the commands to reproduce both are below.
Why this post
A previous post worked out whether a model fits on a GPU. This one is the next step: actually standing up meta-llama/Llama-3.1-8B-Instruct on Red Hat OpenShift AI (RHOAI), as a Helm chart you can helm install and get a working OpenAI-compatible endpoint out of.
The chart itself is short — four templated resources. What’s worth writing down is the two failures that happened between “helm install completes” and “the model actually answers a request.” Neither was exotic. Both are the kind of mistake that’s obvious once you see it and easy to reproduce if you don’t.
The shape of the chart
Four templates, one Helm release:
templates/
templates/
├── namespace.yaml # the project the dashboard needs to see it
├── pvc.yaml # PVC for the weights, created or referenced
├── servingruntime.yaml # vLLM container + CLI args
├── inferenceservice.yaml # KServe object that ties runtime + storage together
└── route.yaml # optional external OpenShift Routevalues.yaml drives all of it:
values.yaml
namespace: models
model:
name: llama-3-1-8b-instruct
storagePath: /llama-3.1-8b-instruct
mountPath: /mnt/models
servedModelName: llama-3-1-8b-instruct
storage:
create: true
claimName: llama-3-1-8b-instruct-pvc
size: 40Gi
storageClassName: ""
accessModes:
- ReadWriteOnce
servingRuntime:
name: vllm-llama-3-1-8b-runtime
image: registry.redhat.io/rhaii/vllm-cuda-rhel9@sha256:5800e12b2a465f...
extraArgs:
- "--max-model-len=8192"
- "--gpu-memory-utilization=0.90"
- "--dtype=bfloat16"
resources:
requests: { cpu: "4", memory: 24Gi, nvidia.com/gpu: "1" }
limits: { cpu: "8", memory: 32Gi, nvidia.com/gpu: "1" }
inferenceService:
minReplicas: 1
maxReplicas: 1
annotations:
serving.kserve.io/deploymentMode: RawDeployment
route:
enabled: false
host: ""Two design choices worth calling out:
- The PVC is pre-staged, not downloaded by the chart. The gated Hugging Face model has to come from a token belonging to an account that’s been granted access — that’s a human step, not something to bake into
helm install. The chart creates the PVC; a separate one-shotJob(below) or a manual pod fills it. RawDeploymentmode, not Serverless. RHOAI can run KServe on top of Knative Serving (“Serverless” mode) or as plain KubernetesDeployments(“RawDeployment”). This chart targets RawDeployment because it doesn’t assume Knative Serving is installed — check which one your cluster runs withoc get crd services.serving.knative.dev; if that CRD is missing, RawDeployment is the only mode that will actually schedule a pod.
Failure 1 — a placeholder digest that looked real
The very first helm install failed before a pod was even scheduled:
Failed to apply default image tag "quay.io/modh/vllm@sha256:4f5509961...":
couldn't parse image name: invalid checksum digest lengthThe digest in values.yaml was a placeholder written while scaffolding the chart — the right shape for a sha256 digest, the wrong length, and not tied to any image that actually exists. It never should have shipped, and it’s a useful reminder that a @sha256:... pin looks authoritative even when it’s fabricated — nothing about the YAML makes that visible.
The fix was to stop guessing and read the real image off the cluster’s own RHOAI installation, since RHOAI already ships built-in vLLM ServingRuntime templates in redhat-ods-applications:
terminal
registry.redhat.io/rhaii/vllm-cuda-rhel9@sha256:5800e12b2a465f159...That’s the actual CUDA vLLM runtime the cluster’s operator ships and already has pull access to — swapped straight into servingRuntime.image. Lesson: when a chart needs an image digest for a vendor-shipped runtime, pull it from the cluster’s own catalog of templates, not from memory or a placeholder. RHOAI’s built-in templates are the ground truth for what actually resolves on that cluster.
Failure 2 — one field playing two incompatible roles
With the image fixed, the pod scheduled, pulled, and crashed on startup:
huggingface_hub.errors.HFValidationError: Repo id must be in the form
'repo_name' or 'namespace/repo_name': '/mnt/models/llama-3.1-8b-instruct'.That error is huggingface_hub refusing to treat a filesystem path as a Hugging Face repo ID — which only happens when vLLM’s --model flag isn’t pointed at a real local directory. The chart’s first version had a single model.path value doing double duty:
…consumed identically in two templates that need different things:
templates/inferenceservice.yaml (before)
Those two lines look parallel and aren’t. storageUri needs a path relative to the PVC volume root — where the weights live on the volume. --model needs the path inside the running container — where KServe mounts that volume. In RawDeployment mode with a pvc:// URI, KServe volume-mounts the PVC directly using the storageUri path as a subPath, landing it at a fixed /mnt/models in the container — it does not preserve the source path as a nested subdirectory. Reusing the same value for both meant the container looked for /mnt/models/llama-3.1-8b-instruct when the weights were actually sitting at /mnt/models (the PVC’s /llama-3.1-8b-instruct mounted with subPath).
The fix was splitting one field into two, each named for what it actually is:
values.yaml (after)
templates/inferenceservice.yaml (after)
Lesson: when a value crosses a Kubernetes-object boundary — a PVC path here, a container path there — give each side its own field, even if they happen to look identical today. They’re only accidentally the same string; nothing enforces that they stay that way, and when they diverge the error shows up two layers away from the line that’s actually wrong.
Staging the weights
The chart creates the PVC but never fills it — that’s deliberate, and it’s also exactly the gap that produces HFValidationError a second time if skipped: an empty /mnt/models looks the same to vLLM as a wrong path, because there’s no config.json at that location either way.
A one-shot Job, run once and discarded, mounts the same PVC and downloads the gated model with a Hugging Face token:
stage-weights-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: stage-llama-3-1-8b-weights
namespace: models
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: download
image: registry.access.redhat.com/ubi9/python-312:latest
command: ["/bin/bash", "-c"]
args:
- |
set -euo pipefail
pip install --no-cache-dir -U "huggingface_hub[cli]"
hf download meta-llama/Llama-3.1-8B-Instruct \
--local-dir /mnt/models/llama-3.1-8b-instruct \
--exclude "*.pth" --exclude "original/*"
env:
- name: HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token
key: HF_TOKEN
volumeMounts:
- name: models
mountPath: /mnt/models
volumes:
- name: models
persistentVolumeClaim:
claimName: llama-3-1-8b-instruct-pvcTwo things worth flagging in that manifest:
--excludeneeds to be repeated per pattern, not passed as trailing bare arguments. The newerhf downloadCLI treats every bare word after the repo ID as an explicit filename to fetch — so--exclude "*.pth" "original/*"gets parsed as--exclude "*.pth"plus a separate request for a literal file namedoriginal/*, which 404s.--exclude "*.pth" --exclude "original/*"is unambiguous.- The excludes matter. Skipping the redundant
.pthcheckpoint and theoriginal/directory avoids downloading the same weights twice in different formats — vLLM only reads the safetensors.
Run it, watch it, clean it up:
terminal
oc create secret generic hf-token -n models --from-literal=HF_TOKEN=$HF_TOKEN
oc apply -f stage-weights-job.yaml
oc logs -f job/stage-llama-3-1-8b-weights -n models
# once it completes
oc delete job stage-llama-3-1-8b-weights -n models
oc delete pod -n models -l app.kubernetes.io/instance=llama-3-1-8bThat last line matters: the predictor pod only reads the PVC at startup, so a pod that crash-looped against an empty volume needs to be recreated, not just left to retry — restarting it after the Job completes is what actually picks up the now-populated weights.
Making it show up in the RHOAI dashboard
Everything above gets the model serving. It doesn’t get it listed — RHOAI’s dashboard only shows namespaces carrying one specific label as Data Science Projects:
templates/namespace.yaml
Without it, the ServingRuntime’s own opendatahub.io/dashboard: "true" label is necessary but not sufficient — the object exists, the pod runs, and the dashboard shows nothing, because the namespace it lives in was never eligible to appear as a project in the first place. The ServingRuntime additionally carries opendatahub.io/recommended-accelerators: '["nvidia.com/gpu"]', matching what RHOAI’s own built-in CUDA runtime template sets, so the GPU badge renders instead of a blank field in the serving-runtime picker.
Exposing it outside the cluster
RawDeployment mode gives the InferenceService a ClusterIP Service and KServe’s own internal URL — enough to call from inside the cluster, not enough to reach from a laptop. route.yaml is an optional OpenShift Route for that:
templates/route.yaml
{{- if .Values.route.enabled }}
apiVersion: route.openshift.io/v1
kind: Route
spec:
{{- if .Values.route.host }}
host: {{ .Values.route.host }}
{{- end }}
to:
kind: Service
name: {{ .Values.model.name }}-predictor
port:
targetPort: http1
tls:
termination: edge
insecureEdgeTerminationPolicy: Redirect
{{- end }}host is optional — omit it and OpenShift assigns one from the cluster’s default wildcard domain:
Calling it
vLLM’s OpenAI-compatible server ships a FastAPI-generated Swagger UI at /docs for free — useful for exploring the endpoint set without reading source:
https://<route-host>/docsAnd the actual inference call:
Wrapping up
Nothing in this chart is architecturally novel — a ServingRuntime, an InferenceService, a PVC, a Route. What made it take two debugging passes instead of zero was ordinary: a digest typed from memory instead of read from the cluster, and one values.yaml field asked to mean two different things depending which template read it. Neither shows up in helm lint. Both show up immediately in oc logs, once you know which container’s boundary you’re actually looking at — the PVC volume, or the running container’s filesystem.
The chart is in posts/rhoai-llama-helm/ alongside this post — Chart.yaml, values.yaml, the templates, and the staging Job, ready to helm install against a RHOAI cluster that already has the CUDA vLLM runtime available.