GitOps Workflow Guide: Pull Requests, Reconciliation, Drift, and Rollbacks

Overview

In a GitOps workflow, a change begins as a commit or pull request rather than as a direct manual edit in a cluster. After review and approval, an automation system observes the repository and works to make the target environment match the declared state. The cluster is not treated as the place where desired configuration is authored; it is the place where that configuration is continuously applied and checked.

This separates two responsibilities that are often mixed together. Developers and platform engineers propose the desired state in Git, while a GitOps controller performs deployment and reports whether the environment matches that state. The controller’s repeated comparison is called reconciliation. If the repository says that a service should run a particular image and the cluster differs, the controller attempts to correct the difference or reports why it cannot.

Developer change
      |
      v
Pull request -> review, tests, policy checks
      |
      v
Merge to environment branch or directory
      |
      v
GitOps controller watches repository
      |
      v
Reconcile desired state with cluster
      |
      +--> Healthy: record status and observe
      |
      +--> Drift or failure: alert, investigate, or restore

GitOps is not simply “put YAML in Git.” A workable implementation also defines repository ownership, approval rules, promotion between environments, secret handling, observability, and recovery procedures. The right design depends on the team’s security model and deployment requirements, but the core loop remains the same: propose, review, merge, reconcile, verify, and recover when necessary.

Typical repository structures

There is no single correct layout, but two patterns are common. An application repository can contain source code, tests, and build configuration, while a separate environment repository contains deployment manifests or package values. Alternatively, a single repository can contain application and environment directories. In either model, make the ownership boundary obvious and keep production changes easy to locate.

A useful structure may distinguish reusable base configuration from environment-specific overlays, for example:

environments/
  development/
  staging/
  production/
applications/
  service-a/
  service-b/
platform/
  ingress/
  policies/

Choose a layout that supports review and promotion without requiring people to copy large files manually. Keep generated output controlled and explain how an image version, chart version, or configuration value moves from one environment to the next.

Checklist by scenario

Before creating the first deployment

• Define which repository and branch or directory represent each environment.
• Document who owns application configuration, platform configuration, and access to production.
• Choose a manifest format or packaging approach that the team can validate consistently.
• Separate non-sensitive configuration from credentials and other secrets.
• Configure repository permissions, required reviews, status checks, and protected production paths.
• Give the GitOps controller only the cluster permissions it needs, and separate responsibilities where practical.
• Decide how deployment health will be measured: readiness, rollout status, service checks, or another explicit signal.

For a routine application change

1. Build and test the application in the CI/CD pipeline. Produce an immutable image reference where possible rather than relying on a moving tag.
2. Open a pull request that changes the deployment reference or environment configuration.
3. Include the reason for the change, expected impact, test evidence, and rollback path in the pull request.
4. Run manifest validation, policy checks, security checks, and any environment-specific tests before approval.
5. Have the appropriate owners review the change, especially when it affects production, networking, data stores, or access controls.
6. Merge only after required checks pass. The GitOps controller should detect the new desired state and begin reconciliation.
7. Confirm that the controller reports a successful sync or reconciliation, then verify the workload and user-facing behavior.
8. Record relevant deployment links, alerts, and follow-up work so the change remains traceable.

For broader CI/CD design context, compare this process with the stages and quality gates described in the CI/CD pipeline design guide.

For promotion from staging to production

• Use a deliberate promotion mechanism, such as a reviewed pull request that updates the production environment.
• Confirm that the artifact promoted to production is the one tested in staging, unless the change is explicitly rebuilding it.
• Review configuration differences rather than assuming staging and production are equivalent.
• Check migration order, capacity assumptions, feature flags, and dependency compatibility.
• Define who can approve the promotion and what evidence they need.
• After reconciliation, perform post-deployment checks and monitor the relevant service indicators.

For suspected drift

• Identify whether the difference is an unauthorized manual change, a controller failure, a stale repository state, or an intentional exception.
• Capture the live state and recent events before changing anything, while avoiding the exposure of secret values.
• Compare the repository’s desired state with the cluster’s effective state, including overlays, defaults, and generated resources.
• Decide whether to codify the intended change in Git or remove the unintended live change.
• Investigate repeated drift instead of repeatedly correcting symptoms. It may indicate unclear ownership, excessive permissions, or an incomplete declaration.

For a rollback

• Define in advance what qualifies as a rollback: a known-good image, a previous configuration commit, or a staged operational procedure.
• Prefer reverting the responsible Git change so the repository and cluster remain aligned.
• Review database and data compatibility before reverting application code.
• Watch reconciliation status and deployment health during the rollback.
• Document the cause, the recovery steps, and any required follow-up change.

For an incident-focused sequence, use the incident response checklist for failed deployments and production rollbacks.

Argo CD and Flux: a practical comparison

Both can support continuous delivery with GitOps. The decision should follow team skills, repository structure, access requirements, troubleshooting preferences, and existing platform tooling—not a generic claim that one is universally better.

What to double-check

Secrets and sensitive configuration

GitOps improves change visibility, but putting a secret directly into a repository can create a lasting exposure even if the file is later deleted. Use an approved secret-management approach, encrypted representation, or external secret reference that fits the organization’s controls. Review who can decrypt or read values, how rotation works, and whether pull-request checks might print sensitive data in logs.

Drift and exceptions

Some teams allow temporary emergency changes in a live environment. If that is necessary, define who can make them, how they are recorded, how long they may remain, and how the intended state is reconciled afterward. An undocumented exception creates a confusing split between Git and the cluster.

Health and observability

A successful reconciliation does not necessarily mean that users are receiving a healthy service. Connect deployment status with application logs, metrics, traces, alerts, and synthetic or endpoint checks. The observability tools comparison can help structure that broader view.

Infrastructure boundaries

Decide which resources belong in the GitOps deployment flow and which are managed by infrastructure as code or a cloud control plane. Avoid having multiple systems continuously manage the same resource. For a related review of planning, applying, and managing drift, see the Terraform infrastructure as code workflow.

Rollback realism

A revert is not automatically safe. Schema changes, irreversible data operations, external APIs, and configuration changes can make application rollback incomplete. Test recovery procedures with representative dependencies and make the result accessible to the on-call team.

Common mistakes

• Allowing direct production edits as the normal path. This weakens auditability and makes the next reconciliation surprising. Treat emergency access as an exception with a documented reconciliation step.
• Approving changes without deployment evidence. A valid manifest can still produce an unhealthy service. Require checks that reflect both configuration validity and runtime behavior.
• Using ambiguous image tags. A tag that changes underneath a deployment makes it harder to identify what was tested and what should be restored. Prefer traceable artifact references.
• Copying environment files manually. Manual duplication causes configuration drift. Use structured overlays, explicit values, or another repeatable promotion method.
• Giving the controller excessive permissions. Broad access increases the impact of a compromised repository, credential, or controller. Scope access by cluster, namespace, resource, and environment where practical.
• Ignoring failed reconciliation. A controller that cannot apply desired state is an operational signal, not background noise. Route failures to the team that can resolve them.
• Choosing a tool from its interface alone. Compare repository workflows, multi-cluster needs, access controls, failure behavior, upgrade practices, and team familiarity before selecting Argo CD, Flux, or another option.

If your team is deciding between this model and a more conventional release process, the guide to GitOps versus traditional CI/CD provides a useful decision framework.

When to revisit

Review this GitOps workflow before seasonal planning cycles, platform migrations, major repository reorganizations, or changes to deployment ownership. Revisit it whenever the controller, Kubernetes distribution, manifest tooling, secret system, CI/CD pipeline, or access model changes. Tool capabilities and configuration syntax can change across major releases, so verify commands and integrations against the documentation for the versions you operate rather than copying old examples unchanged.

Schedule a recovery drill after a significant production architecture change. Start with a known test deployment: open a controlled pull request, verify policy checks, observe reconciliation, create or identify a safe drift scenario, and perform a documented revert. Record how long each step takes and where operators needed information that the runbook did not provide.

Use this final review before adopting or revising the workflow:

1. Write down the source of truth for every environment.
2. Trace one change from pull request to reconciliation to runtime verification.
3. Confirm that production approval, secret handling, and controller permissions are explicit.
4. Test drift detection and decide how an emergency change returns to Git.
5. Perform a rollback using a reversible, representative change.
6. Update ownership, links, alerts, and tool-specific instructions after the exercise.

A GitOps workflow is reliable when the desired state is clear, approvals are meaningful, reconciliation is observable, and recovery is practiced. Keep this checklist with the repository or deployment runbook, and review it whenever the systems around it change.