TLS certificate renewal is safest when treated as a recurring deployment process rather than an emergency ticket. This runbook explains how to inventory certificates, monitor expiration, renew through ACME or a managed provider, rotate private keys, deploy complete certificate chains, validate load balancers and Kubernetes secrets, and keep a clear rollback path.
Overview
A certificate can be valid at the certificate authority and still fail in production. The wrong certificate may be attached to a load balancer, an intermediate certificate may be missing, a Kubernetes secret may not have been reloaded, or a newly generated private key may not match the certificate. Successful TLS certificate deployment therefore has three separate goals:
- Renew: obtain a certificate with the correct names and validity period.
- Deploy: install the certificate, private key, and required intermediate chain in every termination point.
- Verify: confirm that real clients receive the intended certificate and that the application remains healthy.
Start with an inventory, not with the first expiration alert. A certificate inventory should identify every public and internal hostname, certificate owner, renewal method, termination point, secret or resource name, and validation method. Include certificates managed outside the main cloud account, such as those installed on appliances, legacy servers, development environments, or third-party integrations.
For infrastructure managed as code, record the certificate reference and deployment configuration in version control without committing private keys. A GitOps workflow can make changes reviewable, while secret storage remains in an appropriate secret manager. For related deployment controls, see the GitOps workflow guide.
What to track
Certificate inventory
Use a table or service catalog with fields such as:
- Hostname and environment, including wildcards and alternate names
- Certificate authority and renewal method
- Not-before and expiration timestamps
- Certificate fingerprint or serial identifier
- Termination point, such as a reverse proxy, CDN, ingress, load balancer, or application server
- Secret name, listener, configuration path, or cloud resource identifier
- Private-key rotation status and access owner
- Automatic renewal status and last successful renewal
- Alert recipients, escalation path, and documented rollback method
Track the certificate's subject alternative names, not only its common name. A renewed certificate that omits an API hostname or alternate domain can appear healthy for the main website while breaking a secondary service.
Expiration and renewal signals
Monitor both the certificate served to clients and the renewal system that produces it. Useful signals include days until expiration, the last successful issuance, the next scheduled renewal attempt, ACME challenge failures, provider API errors, and deployment or reload failures. Alert at multiple thresholds rather than waiting for one final warning. A practical starting point is an early planning alert, a higher-priority alert when the remaining window is short, and an urgent alert when renewal or deployment has failed near expiration. Set thresholds according to your provider's renewal behavior, escalation time, and ability to perform an emergency change.
Private keys and chains
Every renewal should answer two questions: does the private key match the new certificate, and will clients receive the complete chain they need? Key rotation is generally preferable when replacing a certificate, unless an approved operational reason requires retaining the existing key. Generate keys in a controlled environment, restrict permissions, and avoid printing them in CI logs or shell history.
For a local inspection, these OpenSSL checks compare the public key material without exposing the private key contents:
openssl x509 -in certificate.pem -pubkey -noout | openssl pkey -pubin -outform der | sha256sum
openssl pkey -in private-key.pem -pubout | openssl pkey -pubin -outform der | sha256sumThe digests should match. Validate the certificate and chain separately as well. Do not assume that a browser warning is caused by expiration; hostname mismatch, an incomplete chain, unsupported protocol settings, and a wrong listener are common alternatives.
Cadence and checkpoints
Monthly or quarterly review
On a regular cadence, review the inventory for certificates that are missing an owner, approaching their renewal window, attached to retired infrastructure, or using a manual process. Confirm that monitoring covers the externally served endpoint rather than only the file stored on disk. Check that alerts reach a maintained team channel and that at least one person can perform the renewal outside normal business hours.
Review the renewal job's recent history. A successful test run is useful, but it does not prove that production deployment succeeded. Record issuance, installation, reload, and endpoint verification as separate checkpoints. If scheduled automation is involved, audit the schedule and timezone assumptions; the cron expression guide provides a method for reviewing recurring jobs.
Before renewal
- Confirm the certificate covers every required hostname and environment.
- Confirm account credentials, DNS permissions, HTTP challenge paths, or other validation dependencies.
- Check whether a CDN, load balancer, ingress controller, or service mesh terminates TLS before the application.
- Prepare a new private key and certificate request if the process uses explicit key rotation.
- Test the deployment in a non-production environment when the termination path is shared or complex.
- Define the rollback artifact, owner, and verification command before making the change.
ACME and managed-certificate workflows
For ACME, separate domain validation from installation. The client may successfully complete a challenge while the resulting certificate is never copied to the active proxy. Check challenge logs, issuance output, file permissions, and the service reload step. DNS-based validation also requires careful handling of propagation and delegated zones; document which account owns the relevant DNS record.
With a cloud-managed certificate, confirm that the certificate is attached to the correct listener, region, project, or account. Provider consoles can show a certificate as active while traffic is still routed through another resource. Verify the public endpoint after the attachment and after any required proxy or ingress reload.
Kubernetes deployment checkpoint
For Kubernetes, update the intended TLS secret in the intended namespace, then verify that the ingress or gateway references that secret. A secret update may not immediately change every running component, depending on the controller and termination architecture. Check controller events, rollout status, and the certificate served externally. Avoid placing private keys directly in manifests or source control.
Certificate changes can coincide with resource or rollout changes. If a renewal causes pods to restart or an ingress controller to reload, review capacity and scheduling assumptions using the Kubernetes resource requests and limits workflow.
How to interpret changes
Use staged validation after deployment:
- Local artifact check: inspect dates, names, issuer information, key matching, and chain files.
- Service check: confirm the proxy, load balancer, ingress, or application accepted the configuration and reloaded successfully.
- External check: connect to each important hostname from outside the deployment network and inspect the served certificate.
- Application check: test representative HTTPS requests, redirects, API clients, health checks, and mutual-TLS integrations where applicable.
- Monitoring check: confirm expiration monitors now report the new certificate and that no alert remains tied to an old endpoint.
A short overlap period is useful when the platform permits it: retain the previous known-good deployment artifact for rollback, but do not leave obsolete private keys broadly accessible. For configuration-driven systems, submit the change through the normal review path. Terraform-managed certificate attachments should be reviewed for provider and state implications; the Terraform provider version guide covers safe upgrade planning that also applies to certificate-related infrastructure changes.
Troubleshooting table
| Symptom | Likely causes | Checks |
|---|---|---|
| Browser reports an expired certificate | Old listener, stale CDN edge, failed reload, or incorrect hostname | Inspect the certificate served by the exact hostname and identify the terminating resource. |
| Hostname mismatch | Missing subject alternative name or wrong certificate attached | Compare requested hostname with the certificate names and listener configuration. |
| Handshake failure | Key mismatch, incompatible TLS settings, client-certificate issue, or incomplete configuration | Compare public keys, inspect server logs, and test with a controlled TLS client. |
| Some clients fail while browsers work | Missing intermediate chain or client compatibility difference | Serve the intended chain and test from more than one client environment. |
| Kubernetes still serves the old certificate | Wrong namespace or secret, controller cache, or different TLS termination point | Check the resource reference, controller events, rollout behavior, and external endpoint. |
| ACME renewal fails | Challenge route blocked, DNS record unavailable, rate or account issue, or permission change | Read the ACME client log, verify challenge reachability, and confirm DNS or account permissions. |
Emergency renewal
If expiration is near, assign one operator to the change and another to verification when possible. Capture the current configuration, issue or retrieve the replacement certificate, validate the key and chain, deploy to one termination point or controlled slice, and test externally before completing the rollout. Keep the old certificate available for immediate rollback until the new deployment is confirmed. After recovery, document why automation or alerting did not prevent the emergency and schedule corrective work.
When to revisit
Revisit this runbook monthly or quarterly, depending on certificate volume and operational risk. Perform an additional review after a domain migration, provider or certificate-authority change, load balancer replacement, Kubernetes ingress change, secret-management change, or failed renewal. Also review it whenever browser or platform requirements, ACME client behavior, cloud-provider procedures, or internal security policy changes.
At each review, remove retired endpoints from the inventory, test one renewal path, verify alert ownership, and confirm that rollback artifacts are accessible without exposing private keys. Treat the inventory as an operational tool, not a document that is updated only after an outage.
Next action: create the inventory table today, select one certificate with an upcoming renewal window, and perform a controlled renewal with artifact, service, external, and application checks recorded. Schedule the next review before closing the change.