Upgrading OKD 4.15 to 4.21 in Production: A Field Log

Six years ago I wrote about installing OKD 4.4 on AWS. This is the sequel nobody plans for: a production OKD cluster on AWS that had been sitting on 4.15 for well over a year, brought up to 4.21 in five upgrade windows over one weekend, including the cross-OS migration from Fedora CoreOS to CentOS Stream CoreOS. Every hop had at least one surprise, and almost none of them are documented in one place.

The starting point

Between 4.15 and today, the OKD project changed its base OS from Fedora CoreOS (FCOS) to CentOS Stream CoreOS (SCOS). That migration is not a regular upgrade: 4.16 is an explicit stepping-stone release you force onto the cluster with a manual workaround and leave immediately, and 4.18 was never released as an independent OKD-SCOS minor at all. The chain that actually works, per the OKD working group guidance:

Note the K8s column: because 4.18 does not exist on the OKD-SCOS path, window 3 jumps two Kubernetes minors. Do not assume the OCP version mapping holds; extract the real Kubernetes version from each payload before you trust it.

Pre-flight surprise: the catalog that was disabled years ago

The first blocker appeared before any upgrade was triggered. Bumping the cluster-logging channel failed with:

ResolutionFailed: no operators found from catalog redhat-operators
in namespace openshift-marketplace

Every Red Hat operator on the cluster referenced a catalogsource that had been disabled at install time, years earlier, and nobody had noticed since: installed CSVs keep running without their catalog, they just silently stop receiving updates. The fix is one patch:

$ oc patch operatorhub cluster --type=merge \
    -p '{"spec":{"sources":[{"name":"redhat-operators","disabled":false},
        {"name":"redhat-marketplace","disabled":false}]}}'

Both catalogs came up healthy within thirty seconds, and OLM immediately auto-resolved several operators that were quietly stuck: OADP, the EFS CSI driver, devworkspace. Audit oc get operatorhub cluster -o yaml before any catch-up like this; a disabled default source caps every operator that references it, and an operator capped below your target OKD version will block the cluster upgrade later.

Window 1: FCOS to SCOS needs manual surgery

The 4.15 to 4.16 migration cannot complete on its own. The 4.16 payload ships CRDs whose CEL validation rules the old 1.28 kube-apiserver cannot compile, so the CVO gets stuck early, by design:

Could not update customresourcedefinition
"infrastructures.config.openshift.io" (47 of 903)

The documented workaround forces the kube-apiserver to roll first: pause the CVO, patch the kube-apiserver-operator deployment by hand with values from the new payload, wait for all masters to come up on the new kube-apiserver, then resume the CVO.

$ oc adm upgrade --allow-explicit-upgrade --force \
    --to-image=quay.io/okd/scos-release@sha256:0de353901f9ab5ecb14c2583d16d24561df23d1bf46fe03f218f2ffb8f134096

# wait for the expected CRD error, then:
$ oc -n openshift-cluster-version scale deploy/cluster-version-operator --replicas=0

$ oc -n openshift-kube-apiserver-operator set image \
    deploy/kube-apiserver-operator \
    kube-apiserver-operator=quay.io/okd/scos-content@sha256:37d6b6c13d864deb7ea925acf2b2cb34305333f92ce64e7906d3f973a8071642

$ oc -n openshift-kube-apiserver-operator set env \
    deploy/kube-apiserver-operator \
    IMAGE=quay.io/okd/scos-content@sha256:5c9128668752a9b891a24a9ec36e0724d975d6d49e6e4e2d516b5ba80ae2fb23 \
    OPERATOR_IMAGE=quay.io/okd/scos-content@sha256:37d6b6c13d864deb7ea925acf2b2cb34305333f92ce64e7906d3f973a8071642 \
    OPERAND_IMAGE_VERSION=1.29.10 \
    OPERATOR_IMAGE_VERSION=4.16.0-okd-scos.1

# masters roll to the new kube-apiserver over ~20 minutes, then:
$ oc -n openshift-cluster-version scale deploy/cluster-version-operator --replicas=1

Three details that cost me time:

1. There are two builds behind the 4.16.0-okd-scos.1 tag on quay. Use the digest the docs reference, not whatever the tag currently points at; the docs validated that specific build.
2. The docs text says OPERAND_IMAGE_VERSION=1.29.6. The actual payload manifest says 1.29.10. Always extract the values from 0000_20_kube-apiserver-operator_06_deployment.yaml inside the target payload instead of copying them from any document, including this one.
3. The docs list three env vars. You need four. OPERATOR_IMAGE_VERSION selects which per-version entry of the cluster FeatureGate object the operator reads at startup. Leave it pointing at the old version and the new operator binary panics on a feature gate that does not exist in the old entry:

panic: feature "ShortCertRotation" is not registered in FeatureGates

I learned point 3 the hard way in window 3, which reuses the same workaround for the 4.17 to 4.19 hop. The operator pod crash-looped until the fourth env var was patched in, and then went to Running and leader-elected within three minutes.

Once the CVO resumed, the machine-config operator pivoted all nine nodes from FCOS to SCOS, and about ninety minutes after the workaround started the cluster reported 4.16.0-okd-scos.1 with zero customer-facing API downtime. Window 2, the 4.16 to 4.17 pass-through, was an ordinary oc adm upgrade --to-image and needs no story.

Payload signatures: when –force is justified

From window 3 onward, every trigger without --force was rejected:

ReleaseAccepted=False reason=RetrievePayload
The update cannot be verified: unable to verify sha256:<digest>
against keyrings: verifier-public-key-ci

OKD-SCOS payloads are signed by the OKD CI key, and the in-cluster keyring regularly fails to recognize the signer on newer releases. It is a recurring upstream issue. The temptation is to just always pass --force, and that is a mistake: the flag waives signature verification and all upgrade precondition checks, including incompatible-operator and admin-ack gates that exist to stop you. My rule after five windows of this:

• Trigger without --force first, every window.
• Add it only when the rejection message is exactly the verifier-public-key-ci signature error and oc adm upgrade showed Upgradeable=True immediately before.
• Trust comes from pulling quay.io/okd/scos-release by a digest taken from the OKD release controller, not from the keyring you just bypassed.

Window 3 mid-flight: two more blockers

The 4.17 to 4.19 hop, carrying two Kubernetes minors, stalled twice more after the kube-apiserver surgery.

First, a CRD from an older minor refused to update:

Could not update customresourcedefinition
"machineconfignodes.machineconfiguration.openshift.io" (786 of 925):
status.storedVersions[0]: Invalid value: "v1alpha1":
must appear in spec.versions

The API had graduated from v1alpha1 to v1, and Kubernetes refuses a CRD update that would orphan a stored version. The clean fix, if and only if no objects of that type exist:

$ oc get machineconfignodes -A
No resources found

$ oc delete crd machineconfignodes.machineconfiguration.openshift.io
$ oc -n openshift-cluster-version rollout restart deploy/cluster-version-operator

The CVO recreates the CRD from the payload on its next sync; the rollout restart is needed because its retry backoff will not pick up the deletion quickly on its own. If objects do exist, you need the storage version migrator instead, not a delete.

Second, the new console pods crash-looped:

F validate.go:76] invalid flag: plugins-order,
error: list must only contain currently enabled plugins

The console CR still listed a plugin from an operator version that no longer ships it. The 4.17 console tolerated the dangling reference; the 4.19 console validates and refuses to start. One patch to remove the orphan from spec.plugins and the pods started within thirty seconds. Cross-check oc get console.operator.openshift.io cluster against oc get consoleplugin before upgrading, not during.

Window 4: SCOS 10 dropped runc

The 4.19 to 4.20 hop is the big one: CentOS Stream 9 to CentOS Stream 10 underneath, kernel 5.14 to 6.12. Shortly after the node pivots began, one worker and one master went NotReady and stayed there. The failure signature is worth memorizing, because from the outside the nodes look dead while the OS is perfectly healthy:

• oc get nodes shows the stuck nodes NotReady with a stale OS and kubelet version, frozen at the pre-reboot state
• oc get csr shows no Pending CSRs at all
• the cloud console shows the instance running and reachable, and the serial console log shows a clean boot right up to the network scripts, then nothing from crio or kubelet
• force-rebooting the instance changes nothing

The root cause: CentOS Stream 10 removed the runc package entirely, crun is the only container runtime left. This cluster carried two old MachineConfigs, generated years earlier, that pinned crio’s default runtime:

# /etc/crio/crio.conf.d/01-ctrcfg-defaultRuntime
default_runtime = "runc"

After the pivot, crio reads that file, fails to find the runc binary, and restarts forever. No runtime means kubelet never registers, which means no CSR, which means the cluster’s view of the node never updates. Recovery took three steps:

1. Cluster side, delete the override MachineConfigs so the rendered config stops referencing runc:

$ oc delete mc 00-override-master-generated-crio-default-container-runtime \
    00-override-worker-generated-crio-default-container-runtime

2. Node side, on each stuck node, via the cloud serial console: reboot, interrupt GRUB, append rd.break to the kernel line, and from the emergency shell:

switch_root:/# mount -o remount,rw /sysroot
switch_root:/# chroot /sysroot
sh-5.1# sed -i 's/"runc"/"crun"/' /etc/crio/crio.conf.d/01-ctrcfg-defaultRuntime
sh-5.1# exit
switch_root:/# exit

The node boots, crio starts with crun, kubelet registers, Ready within about a minute.

3. The machine-config daemon then marks the node Degraded, because the file on disk no longer matches the old desired state. Revert the edit through oc debug node without restarting crio; the MCD revalidates, clears Degraded, and rolls the node forward to the new rendered config that has no override file at all.

Two operational lessons, both cheap before the window and expensive during: check for these MachineConfigs in every pre-flight from 4.20 on and delete them before triggering,

$ oc get mc | grep -E '00-override.*crio-default-container-runtime'

and enable serial console access at the cloud account level ahead of time. On AWS it is a one-time aws ec2 enable-serial-console-access, it costs nothing, and enabling it while two nodes are down wastes the minutes you need most. Our break-glass SSH MachineConfigs turned out to be useless in the moment because nobody had the matching private key at hand; verify that path periodically or do not count it as a recovery layer.

Also filed under window 4: switch the cluster to cgroup v2 before this hop if it is still on v1,

$ oc patch nodes.config.openshift.io cluster --type=merge \
    -p '{"spec":{"cgroupMode":"v2"}}'

which is its own full node roll, better done as a separate step than discovered as a requirement mid-window.

Window 5 and the shape of a window

After all that, 4.20 to 4.21 was the upgrade you want every upgrade to be: same OS major, same kernel series, Kubernetes 1.34 has zero API removals, about eighty minutes end to end. The difference was not luck; by window 5 the pre-flight gate had absorbed every incident from windows 1 through 4. The gate that emerged, run before every trigger:

• all cluster operators available, no MCP degraded, all nodes Ready
• oc adm upgrade clean: no Upgradeable=False, no incompatible operators, admin-acks applied
• every operator’s CSV declares support for the target minor; bump channels first, and re-check after every hop, because catalogs rename channels between minors
• removed-API check via oc get apirequestcount for the target minor
• disk headroom on every node, SELinux Enforcing, no stray rpm-ostree layered packages
• from 4.20 on: no runc override MachineConfigs, serial console enabled
• snapshot every root volume, take an etcd snapshot off-cluster, verify the daily Velero backup completed

And one habit that earned its place: a T+24h alert audit after each window. Some upgrade debt only surfaces as alerts hours later; we caught a storage nodeplugin pod that had been silently stuck for almost two days this way. The upgrade is not done when the ClusterVersion says Completed, it is done when the next day’s alert sweep comes back quiet.

Takeaways

• Multi-hop OKD catch-ups are a sequence of small, well-understood windows, not one big one. Budget a window per minor and expect each to have its own personality.
• The FCOS to SCOS force upgrade works, but read the payload manifest yourself: the docs had a wrong version value, and the env var list was incomplete when I ran it.
• --force is a scalpel for one specific signature bug, verified by exact message match, never a default.
• OS-major hops inside minor-looking upgrades (SCOS 9 to 10) are where node-level assumptions die. Inventory every MachineConfig that touches the container runtime, and have an out-of-band console before you need it.
• Keep an incident catalog next to the SOP in git. Every one of these problems is now a named pre-flight check, and the next upgrade inherits all of them.

Credits

The manual kube-apiserver-operator procedure builds on the OKD working group’s upgrade documentation and on years of upstream work by Vadim Rutkovsky on the OKD project and the cluster-kube-apiserver-operator, which is exactly the component this procedure patches by hand.

Resources

• OKD force upgrade to 4.16
• OKD FCOS to SCOS migration notes
• OKD 4.16 and 4.17 release announcement
• OKD 4.19 release notes
• OKD 4.20 release notes
• OKD 4.21 release notes
• OKD release controller
• okd-project/okd#2255: 4.20 nodes NotReady after upgrade (runc)
• okd-project/okd#2125: payload signature verification failures
• machine-config-operator#5389: auto-migrate runc to crun