Kubernetes Upgrade Runbook: Version Compatibility, API Removals, and Rollback Planning
Kubernetesplatform engineeringcluster operationscloud infrastructureDevOps

Kubernetes Upgrade Runbook: Version Compatibility, API Removals, and Rollback Planning

DDeploy Website Editorial Team
2026-11-10
7 min read

A practical Kubernetes upgrade runbook covering compatibility, API removals, disruption planning, verification, and recovery decisions.

A safe Kubernetes upgrade is less about running one command than proving that every dependency, workload, and recovery path is ready for change. This runbook gives platform teams a repeatable way to assess version compatibility, detect removed APIs, sequence control-plane and node changes, estimate upgrade effort, communicate maintenance, and verify recovery before touching production.

Overview

A Kubernetes upgrade changes the control plane, node components, APIs, admission behavior, and often the managed add-ons around the cluster. Treat it as a platform change with a defined scope, not as routine package maintenance.

The objective is to answer five questions before the maintenance window begins:

  1. Is the target Kubernetes version supported by the current provider, operating system, container runtime, and infrastructure?
  2. Do manifests, custom resources, Helm charts, and controllers still use supported APIs?
  3. Can workloads tolerate node draining and temporary capacity changes?
  4. Can the team restore the cluster or its critical data if the upgrade fails?
  5. What observable evidence will prove that the upgrade succeeded?

Start by recording the current state: cluster version, node-pool versions, platform provider, operating system image, container runtime, CNI, CSI drivers, ingress controller, metrics stack, policy engines, admission webhooks, autoscalers, backup tooling, and critical workloads. A cluster inventory makes hidden dependencies visible and gives the team a baseline for comparison.

For workload-level preparation, use the Kubernetes deployment checklist. Teams managing infrastructure through code should also connect the upgrade to their Terraform plan, review, and drift workflow.

How to estimate

Estimate an upgrade by workstream rather than by elapsed command time. The control-plane change may be short, while API remediation, add-on testing, node rotation, and post-upgrade observation determine the actual effort.

Use this upgrade effort model

Total effort = discovery + remediation + test upgrade + production change + observation + contingency.

Give each component a simple estimate in engineer-hours or team-hours. Do not present the result as a precise forecast; use it to compare upgrade options and expose missing work.

WorkstreamWhat to measureEvidence of readiness
DiscoveryInventory, provider policy review, API scan, dependency mapVersion matrix and owner list are complete
RemediationManifest, chart, controller, and policy changesChanges are reviewed and tested
Test upgradeRehearsal time, failed scenarios, data restore testKnown issues have owners and mitigations
Production changeControl-plane, node-pool, and workload sequencingMaintenance steps and stop conditions are approved
ObservationMonitoring period and verification checksService health remains within agreed thresholds
ContingencyBuffer for retries, provider delays, or rollback workRecovery decision and escalation path are documented

For a decision worksheet, record the current version, target version, upgrade path permitted by the provider, number of clusters and node pools, production workload criticality, add-on compatibility status, backup test date, planned maintenance window, and named approvers. Add a confidence rating—high, medium, or low—to each input. A low-confidence input is usually a reason to test further, not to average the uncertainty away.

Compatibility sequence

Build a compatibility matrix before selecting the maintenance date:

ComponentCheckDecision
Control planeSupported source-to-target path and provider policyProceed only on an approved path
kubelet and node imageVersion relationship, OS image, runtime, and bootstrap configurationUse the provider's supported node sequence
API clientsCI/CD tools, operators, scripts, and dashboardsUpdate clients that depend on removed behavior
Workload APIsDeployments, Ingress, autoscaling, policy, and custom resourcesConvert manifests before the upgrade
Admission webhooksService reachability, certificates, timeouts, and failure policyTest both allowed and rejected requests
Add-onsCNI, CSI, ingress, DNS, monitoring, logging, and autoscalingConfirm compatibility from maintained documentation

Kubernetes version-skew rules and managed-provider policies change over time. Check the documentation for the exact release path rather than relying on a remembered rule. In general, plan control-plane and node changes in the sequence required by the provider, and avoid assuming that a node pool can run indefinitely against a substantially different control-plane version.

Inputs and assumptions

Detect deprecated and removed APIs

Search both live resources and source repositories. A live-cluster scan can reveal objects created by older charts, while repository searches find manifests that have not yet been applied.

  • List API versions used by deployed resources and compare them with the target release's removal and deprecation documentation.
  • Inspect Helm chart templates, Kustomize overlays, generated YAML, and admission policies.
  • Review custom resource definitions and confirm that operators support the target version.
  • Check CI/CD scripts for explicit API paths, discovery assumptions, or hard-coded version checks.
  • Test conversion or migration procedures for resources whose storage or schema changes.

Do not treat a successful manifest apply as proof of compatibility. A resource may be accepted while a controller, webhook, or client later fails to reconcile it. Include reconciliation, rollout, deletion, and restore tests.

Validate backups and recovery

Confirm that the backup captures the data you actually need. A control-plane backup alone may not restore application databases, object storage, external secrets, or provider-managed resources. Perform a restore into an isolated environment where practical, then verify that the restored cluster can schedule workloads, mount storage, resolve services, and accept traffic.

Plan for disruption

Review PodDisruptionBudgets, replica counts, topology spread, anti-affinity, startup probes, termination grace periods, and available capacity. A disruption budget is a workload protection mechanism, not a guarantee that a node drain will finish. It can also block maintenance if replicas or capacity are insufficient.

Before draining a node, confirm that replacement capacity exists and that critical services have more than one viable replica. Identify singleton workloads, local storage, unmanaged pods, and jobs that may require a specific operator decision.

Test webhooks and add-ons

Admission webhooks can affect every deployment and upgrade action. Test certificate validity, DNS resolution, service endpoints, timeout behavior, namespace selectors, and failure policy. A webhook that is unavailable during maintenance can block unrelated operations.

Test add-ons as a set rather than individually. Networking, storage, ingress, autoscaling, observability, policy, and GitOps controllers often interact. The observability tools comparison can help structure the signals used for this validation, while the GitOps workflow guide provides a useful model for reconciliation and drift checks.

Worked examples

Example: low-risk staging upgrade

A team has one staging cluster, a small number of node pools, versioned manifests, and no singleton production dependencies. Its estimate might assign one workstream owner to discovery, one to API and chart remediation, and one to rehearsal and verification. The team first upgrades staging, runs application smoke tests, drains each node pool according to the planned sequence, checks webhook behavior, and observes controller logs and service metrics. Only after the evidence is recorded does it schedule production.

Example: production cluster with stateful services

A production cluster runs stateful workloads, external secrets, a custom operator, and several admission webhooks. The team should assume a larger contingency because storage recovery, operator compatibility, and webhook failure modes can extend the window. It should rehearse a restore, test node replacement, verify volume attachment behavior, and define a stop condition before production work begins. If the rehearsal cannot demonstrate recovery, the upgrade is not ready regardless of whether the version change itself appears straightforward.

Maintenance communication template

Subject: Planned Kubernetes maintenance for [cluster] on [date/time zone]

Change: Upgrade from [current version] to [target version], including [node pools or add-ons].

Expected impact: [none, brief connection changes, or specific service impact]. Workloads will be protected through [replicas, capacity, and drain plan].

Validation: The team will check scheduling, ingress, storage, deployments, error rates, latency, logs, alerts, and critical user journeys.

Stop conditions: [define failed health checks, capacity loss, data errors, or rollback decision points].

Updates and escalation: [status channel, incident contact, approver, and provider support path].

Execution and rollback decision

Take a final backup or snapshot according to the platform's documented process, record the cluster state, and freeze unrelated changes. Upgrade the control plane using the supported method. Then update or replace node pools in controlled batches, respecting disruption budgets and capacity limits. After each batch, verify node readiness, scheduling, networking, storage, and workload health before continuing.

Rollback is not always a simple version reversal. Some control-plane upgrades, data migrations, and provider operations may not support downgrade. Define recovery as one of three paths before starting: stop and continue later, restore or recreate the cluster from tested artifacts, or repair forward to the target state. The decision should name who can invoke it and what evidence triggers it. For failed deployments or production rollbacks, pair this runbook with the incident response checklist.

When to recalculate

Revisit the plan before every minor-version upgrade, node-pool refresh, provider maintenance event, or significant add-on change. Recalculate the effort when the number of workloads, node pools, clusters, or stateful services changes; when a webhook or operator is added; when backup procedures change; or when the target release removes an API used by your platform.

Review the runbook quarterly even if no upgrade is scheduled. Check provider support windows, target-version documentation, API removals, add-on compatibility, node image availability, and internal ownership. Refresh commands and screenshots when tooling changes. The exact commands are less important than preserving the decision logic: inventory, compatibility, rehearsal, controlled sequencing, evidence, and recovery.

Before approving the next upgrade, complete this final checklist:

  • Target path and provider policy are verified.
  • Deprecated APIs and custom resources are remediated or explicitly accepted.
  • Backups have been restored successfully in a test environment.
  • Add-ons, webhooks, clients, and node images are compatible.
  • Capacity, disruption budgets, and stateful workloads have been reviewed.
  • Maintenance communication, owners, stop conditions, and escalation contacts are ready.
  • Post-upgrade checks and the recovery path are written down.

Store the completed worksheet with the change record. That turns the Kubernetes upgrade from a one-time operation into a reusable platform-engineering practice.

Related Topics

#Kubernetes#platform engineering#cluster operations#cloud infrastructure#DevOps
D

Deploy Website Editorial Team

Cloud and Platform Engineering Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.