GitOps Operating Model for Kubernetes: Why Git is the Single Source of Truth for Infrastructure

Your DevOps team just spent 3 hours troubleshooting why production doesn't match staging. Someone ran a manual kubectl apply that wasn't documented. Another engineer edited a ConfigMap directly. A third person updated a deployment via the dashboard. Now nobody knows the actual state of the cluster.

This is the chaos that GitOps eliminates.

Here's the problem: Traditional Kubernetes deployments rely on push-based CI/CD pipelines and manual interventions. Someone (or something) runs kubectl apply to push changes to the cluster. But there's no guarantee that what's in Git matches what's running. Drift happens constantly.

GitOps flips this model: Git is the single source of truth. Automated agents continuously reconcile cluster state with Git. If anything drifts, it's automatically corrected. Every change goes through Git, which means every change is auditable, reversible, and reviewable.

The data is compelling: Organizations using GitOps deploy 3.5x more frequently, recover from failures 24x faster, and reduce production incidents by 52% (CNCF GitOps Working Group Survey, 2024). Yet only 31% of Kubernetes users have adopted GitOps—most still use push-based deployments.

The gap costs velocity, reliability, and sanity.

What GitOps Actually Means

GitOps is an operating model where:

1. Git is the single source of truth for infrastructure and application definitions
2. Automated agents continuously observe Git and cluster state
3. Pull-based reconciliation automatically syncs cluster state to match Git
4. All changes happen via Git (pull requests, reviews, audit trail)
5. Self-healing when manual changes occur (auto-revert to Git state)

Origin: Coined by Weaveworks in 2017, now a CNCF standard practice.

Traditional vs. GitOps Deployment

Traditional Push-Based CI/CD:

Developer → Commit → CI/CD Pipeline → kubectl apply → Kubernetes Cluster

Problems:

• CI/CD has cluster credentials (security risk)
• Manual changes possible (drift)
• No easy rollback (re-run pipeline? which version?)
• Hard to audit (who changed what when?)
• Cluster state not in Git (documentation vs. reality mismatch)

GitOps Pull-Based:

Developer → Commit → Git Repository
                       ↓
                  GitOps Agent (in cluster)
                       ↓
                  Observes Git + Cluster
                       ↓
                  Auto-syncs differences
                       ↓
                  Kubernetes Cluster

Benefits:

• Git has all the answers (audit trail, history, rollback)
• No external system needs cluster credentials
• Automatic drift detection and correction
• Simple rollback (git revert)
• Cluster state documented perfectly (Git = reality)

The Four GitOps Principles

From the OpenGitOps project (CNCF):

1. Declarative

• System state described declaratively (YAML manifests, Helm charts, Kustomize)
• Not imperative scripts ("run these commands")
• Easy to understand desired state

2. Versioned and Immutable

• All declarations stored in Git
• Every change is a commit (immutable history)
• Easy to rollback (revert commit)
• Clear audit trail (who, what, when, why)

3. Pulled Automatically

• Agents in cluster pull from Git (not pushed from external CI/CD)
• Continuous reconciliation (detects and fixes drift)
• Self-healing system

4. Continuously Reconciled

• Agents continuously observe Git and cluster
• Automatically apply changes when Git updates
• Automatically fix when cluster drifts from Git

GitOps Architecture & Tools

GitOps Tool Landscape

Two dominant GitOps agents:

1. Argo CD (Most Popular)

Strengths:

• Comprehensive UI (visualize applications and sync status)
• Multi-cluster support (manage 100+ clusters from one control plane)
• RBAC and access controls (enterprise-ready)
• Large ecosystem and community

Best for: Organizations wanting UI, multi-cluster management, complex applications

2. Flux CD (CNCF Graduated)

Strengths:

• Lightweight and Kubernetes-native
• GitOps Toolkit architecture (composable components)
• Built-in secrets management integration
• Lower resource footprint

Best for: Organizations wanting simplicity, GitOps purists, lower overhead

Both are production-ready. Argo CD has more enterprise features; Flux is simpler and more modular.

GitOps Reference Architecture

┌─────────────────────────────────────────────────────────┐
│                     Git Repository                       │
│  ┌──────────────────┐  ┌──────────────────┐            │
│  │  Infrastructure  │  │   Applications   │            │
│  │  - Namespaces    │  │  - Deployments   │            │
│  │  - RBAC          │  │  - Services      │            │
│  │  - CRDs          │  │  - ConfigMaps    │            │
│  └──────────────────┘  └──────────────────┘            │
└─────────────────────────────────────────────────────────┘
                        ↓ Pull & Sync
┌─────────────────────────────────────────────────────────┐
│               Kubernetes Cluster                         │
│  ┌──────────────────────────────────────────────────┐  │
│  │            GitOps Agent (Argo CD / Flux)          │  │
│  │  - Observes Git repository                        │  │
│  │  - Observes cluster state                         │  │
│  │  - Reconciles differences                         │  │
│  │  - Applies changes                                │  │
│  └──────────────────────────────────────────────────┘  │
│                        ↓                                 │
│  ┌──────────────────────────────────────────────────┐  │
│  │         Application Resources                     │  │
│  │  - Deployments, Services, ConfigMaps, Secrets    │  │
│  └──────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘
                        ↓ Observe & Report
┌─────────────────────────────────────────────────────────┐
│              Monitoring & Alerting                       │
│  - Sync status, health status, drift detection          │
└─────────────────────────────────────────────────────────┘

Git Repository Structure

Mono-repo vs. Multi-repo Strategy:

Mono-repo (Single Repository):

gitops-repo/
├── infrastructure/
│   ├── namespaces/
│   ├── rbac/
│   ├── ingress/
│   └── monitoring/
├── applications/
│   ├── app1/
│   │   ├── base/
│   │   └── overlays/
│   │       ├── dev/
│   │       ├── staging/
│   │       └── production/
│   └── app2/
└── clusters/
    ├── dev-cluster/
    ├── staging-cluster/
    └── prod-cluster/

Pros: Simple, single source of truth, easy to see everything
Cons: All apps in one repo, potential for large repo

Multi-repo (Separate Repositories):

infra-gitops/           # Infrastructure definitions
app1-gitops/            # App 1 deployments
app2-gitops/            # App 2 deployments
cluster-config-gitops/  # Cluster-level configuration

Pros: Separation of concerns, team autonomy, smaller repos
Cons: More complex, need to manage multiple repos

Recommendation: Start with mono-repo. Split into multi-repo when:

• Multiple teams managing different applications
• Need different access controls per application
• Repo becomes unwieldy (>10 applications)

Implementing GitOps: Step-by-Step

Phase 1: Foundation Setup (Week 1-2)

Step 1: Install GitOps Agent (Argo CD Example)

Install Argo CD in Kubernetes cluster:

# Create namespace
kubectl create namespace argocd

# Install Argo CD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for pods to be ready
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

# Expose Argo CD API/UI (choose one):
# Option 1: Port forward (testing)
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Option 2: LoadBalancer (cloud)
kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

# Option 3: Ingress (production)
# Create Ingress resource with TLS

# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Login to Argo CD:

# Install Argo CD CLI
brew install argocd  # macOS
# or download from https://argo-cd.readthedocs.io/en/stable/cli_installation/

# Login
argocd login <ARGOCD_SERVER>  # e.g., localhost:8080 or argocd.example.com
argocd account update-password  # Change default password

Step 2: Create GitOps Repository

Initialize Git repository:

mkdir gitops-repo
cd gitops-repo
git init

# Create structure
mkdir -p infrastructure/{namespaces,rbac,monitoring}
mkdir -p applications/{app1,app2}/{base,overlays/{dev,staging,production}}
mkdir -p clusters/{dev,staging,production}

# Initialize with basic resources
cat <<EOF > infrastructure/namespaces/app-namespaces.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: applications
---
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
EOF

git add .
git commit -m "Initial GitOps repository structure"
git remote add origin <GIT_URL>
git push -u origin main

Step 3: Connect Argo CD to Git Repository

Register repository in Argo CD:

# Via CLI:
argocd repo add https://github.com/yourorg/gitops-repo \
  --username <USERNAME> \
  --password <PERSONAL_ACCESS_TOKEN>

# Or via UI: Settings → Repositories → Connect Repo

Alternative: Private key authentication (SSH):

# Generate SSH key
ssh-keygen -t rsa -b 4096 -f ~/.ssh/gitops_deploy_key

# Add public key to GitHub/GitLab (Settings → Deploy Keys)
# Add private key to Argo CD (Settings → Repositories → Connect Repo using SSH)

Phase 2: Deploy First Application (Week 2-3)

Step 1: Prepare Application Manifests

Example: Simple web application with Kustomize

Base manifest (applications/webapp/base/deployment.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: nginx:1.21
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: webapp
spec:
  selector:
    app: webapp
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP

Base Kustomization (applications/webapp/base/kustomization.yaml):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

Production overlay (applications/webapp/overlays/production/kustomization.yaml):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

replicas:
  - name: webapp
    count: 5

images:
  - name: nginx
    newTag: 1.21.6-alpine

patchesStrategicMerge:
  - ingress.yaml

commonLabels:
  environment: production

Commit to Git:

git add applications/webapp/
git commit -m "Add webapp application manifests"
git push origin main

Step 2: Create Argo CD Application

Define Argo CD Application (applications/webapp/argocd-app.yaml):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: webapp-production
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/yourorg/gitops-repo
    targetRevision: HEAD
    path: applications/webapp/overlays/production
  
  destination:
    server: https://kubernetes.default.svc
    namespace: applications
  
  syncPolicy:
    automated:
      prune: true      # Delete resources not in Git
      selfHeal: true   # Revert manual changes
    syncOptions:
      - CreateNamespace=true
  
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Ignore if HPA is managing replicas

Create application in Argo CD:

# Via CLI:
kubectl apply -f applications/webapp/argocd-app.yaml

# Or via UI: Applications → New App → Fill form

Argo CD will:

1. Connect to Git repository
2. Read manifests from specified path
3. Apply them to Kubernetes cluster
4. Monitor Git for changes
5. Auto-sync when Git updates
6. Self-heal if cluster drifts

Step 3: Verify Deployment

Check via UI:

• Open Argo CD UI
• See webapp-production application
• View sync status (Synced / Out of Sync)
• View health status (Healthy / Progressing / Degraded)
• Visualize resource tree (Deployment → ReplicaSet → Pods)

Check via CLI:

# Application status
argocd app get webapp-production

# Sync status
argocd app sync webapp-production

# View logs
argocd app logs webapp-production

# Application history
argocd app history webapp-production

Phase 3: Advanced GitOps Patterns (Week 3-4)

Pattern 1: Progressive Delivery (Canary Deployments)

Using Argo Rollouts for canary deployments:

Install Argo Rollouts:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Rollout manifest (instead of Deployment):

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: webapp
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20      # 20% traffic to new version
      - pause: {duration: 5m}
      - setWeight: 50      # 50% traffic
      - pause: {duration: 5m}
      - setWeight: 80      # 80% traffic
      - pause: {duration: 5m}
      # Automatic full rollout if no issues
      
      analysis:
        templates:
        - templateName: success-rate
        startingStep: 1
        
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: myapp:v2.0
        ports:
        - containerPort: 80

AnalysisTemplate (success rate monitoring):

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
  - name: success-rate
    interval: 1m
    successCondition: result >= 0.95
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
          sum(rate(http_requests_total{job="webapp",status=~"2.."}[1m])) /
          sum(rate(http_requests_total{job="webapp"}[1m]))

Automatic canary rollout with analysis:

• Deploy new version via Git commit
• Argo CD applies Rollout
• Argo Rollouts progressively shifts traffic
• Analysis runs at each step (Prometheus metrics)
• Automatic rollback if success rate drops below 95%

Pattern 2: Multi-Cluster GitOps

Managing multiple clusters (dev, staging, prod) from one Argo CD instance:

Register additional clusters:

# Get kubeconfig for staging cluster
kubectl config use-context staging-cluster

# Register in Argo CD
argocd cluster add staging-cluster --name staging

# Repeat for production
kubectl config use-context prod-cluster
argocd cluster add prod-cluster --name production

Application of Apps Pattern (manage all apps from one place):

apps/app-of-apps.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: applications
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourorg/gitops-repo
    targetRevision: HEAD
    path: applications
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

This Application points to directory containing other Application manifests—Argo CD automatically creates all applications.

Pattern 3: Secrets Management

Problem: Don't commit secrets to Git (security risk).

Solution Options:

Option 1: Sealed Secrets (Recommended)

# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

# Install kubeseal CLI
brew install kubeseal

# Create secret
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=secretpassword \
  --dry-run=client -o yaml > secret.yaml

# Seal it (encrypts using cluster public key)
kubeseal --format=yaml < secret.yaml > sealed-secret.yaml

# Commit sealed secret to Git (safe!)
git add sealed-secret.yaml
git commit -m "Add sealed database credentials"
git push

# In cluster, SealedSecret controller decrypts to Secret

Option 2: External Secrets Operator

# External secret that pulls from AWS Secrets Manager / Azure Key Vault / GCP Secret Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: prod/database
      property: username
  - secretKey: password
    remoteRef:
      key: prod/database
      property: password

Pattern 4: GitOps with Helm

Helm charts in GitOps:

Application using Helm:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prometheus
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    chart: kube-prometheus-stack
    targetRevision: 45.7.1
    helm:
      values: |
        prometheus:
          prometheusSpec:
            retention: 30d
            storageSpec:
              volumeClaimTemplate:
                spec:
                  resources:
                    requests:
                      storage: 50Gi
        grafana:
          adminPassword: <sealed-secret-ref>
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Or: Helm values in Git repository:

applications/prometheus/
├── Chart.yaml
├── values-dev.yaml
├── values-staging.yaml
└── values-production.yaml

Phase 4: Operations & Best Practices (Ongoing)

GitOps Workflow

Day-to-day developer experience:

1. Make change locally:

cd gitops-repo
# Edit application/webapp/overlays/production/deployment.yaml
# Change image tag: nginx:1.21 → nginx:1.22
git add .
git commit -m "Update webapp to nginx 1.22"

2. Create pull request:

• Push to feature branch
• Open PR in GitHub/GitLab
• Reviewers check changes
• CI runs validation (yamllint, kustomize build, security scans)

3. Merge to main:

• Argo CD detects change (polling or webhook)
• Automatically syncs cluster to match Git
• Deployment happens (canary if configured)
• Monitor via Argo CD UI or CLI

4. Rollback if needed:

# Option 1: Revert Git commit
git revert HEAD
git push

# Argo CD automatically rolls back

# Option 2: Via Argo CD UI
# Click "History & Rollback" → Select previous revision → Rollback

Best Practices

1. Separate Application Code and Deployment Config

app-code-repo/          # Application source code
  - src/
  - Dockerfile
  - CI pipeline

gitops-repo/            # Deployment manifests
  - applications/
  - infrastructure/

Why: Decouples code changes from deployment changes. CI builds image, updates image tag in gitops-repo.

2. Environment Promotion via Git

Deploy to Dev → Auto-sync from main branch
Test in Dev → Manual approval
Promote to Staging → Create PR to staging branch
Test in Staging → Manual approval
Promote to Production → Create PR to production branch
Deploy to Production → Manual sync or auto-sync

3. Drift Detection and Prevention

# Enforce GitOps (prevent manual changes)
syncPolicy:
  automated:
    prune: true       # Delete resources not in Git
    selfHeal: true    # Revert manual kubectl apply changes

4. Notification and Alerting

# Slack notifications for sync failures
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.slack: |
    token: $slack-token
  template.app-sync-failed: |
    message: |
      Application {{.app.metadata.name}} sync failed!
      Sync Status: {{.app.status.sync.status}}
  trigger.on-sync-failed: |
    - when: app.status.sync.status == 'OutOfSync'
      send: [app-sync-failed]

5. RBAC and Access Control

# AppProject for team isolation
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  description: Team A applications
  sourceRepos:
  - 'https://github.com/yourorg/gitops-repo'
  destinations:
  - namespace: 'team-a-*'
    server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace
  namespaceResourceWhitelist:
  - group: 'apps'
    kind: Deployment
  - group: ''
    kind: Service

Troubleshooting Common Issues

Issue 1: Application Stuck in "Progressing"

Symptom: Argo CD shows application health as "Progressing" indefinitely.

Causes:

• Pod failing to start (ImagePullBackOff, CrashLoopBackOff)
• Missing dependencies (ConfigMap, Secret)
• Resource limits too low

Debug:

# Check application status
argocd app get <APP_NAME>

# View application logs
argocd app logs <APP_NAME>

# Describe pods directly
kubectl describe pod <POD_NAME> -n <NAMESPACE>

# Check events
kubectl get events -n <NAMESPACE> --sort-by='.lastTimestamp'

Issue 2: Sync Fails with "ComparisonError"

Symptom: Argo CD can't compare desired state (Git) with actual state (cluster).

Causes:

• Invalid YAML syntax
• Missing CustomResourceDefinition (CRD)
• Kustomize build error

Debug:

# Validate YAML locally
yamllint applications/

# Test Kustomize build
kustomize build applications/webapp/overlays/production

# Check CRDs
kubectl get crds

Issue 3: Secrets Not Decrypting

Symptom: SealedSecret exists, but Secret not created.

Causes:

• Sealed Secrets controller not running
• Secret sealed for different cluster
• Encryption key mismatch

Debug:

# Check Sealed Secrets controller
kubectl logs -n kube-system -l name=sealed-secrets-controller

# Verify SealedSecret
kubectl describe sealedsecret <NAME> -n <NAMESPACE>

# Re-seal if necessary
kubeseal --fetch-cert > pub-cert.pem
kubeseal --cert=pub-cert.pem < secret.yaml > sealed-secret.yaml

Migration Path: From Traditional CI/CD to GitOps

Don't migrate everything at once. Incremental approach:

Phase 1: Infrastructure as Code (Week 1-2)

• Move all Kubernetes manifests to Git
• Use Kustomize or Helm for environment-specific config
• Keep existing CI/CD for deployment (for now)

Phase 2: Install GitOps Agent (Week 2-3)

• Install Argo CD or Flux
• Deploy one non-critical application via GitOps
• Keep CI/CD for other applications

Phase 3: Migrate Applications (Month 2-3)

• Migrate applications one by one
• Start with dev environment
• Move to staging, then production
• Keep CI/CD as backup initially

Phase 4: Deprecate Old CI/CD (Month 3-4)

• Disable kubectl access in CI/CD pipelines
• Remove cluster credentials from CI/CD
• Full GitOps for all deployments
• CI/CD only builds images and updates Git

Phase 5: Advanced Patterns (Month 4+)

• Implement progressive delivery
• Multi-cluster management
• Automated secret rotation
• Full observability integration

Measuring GitOps Success

Key metrics to track:

Deployment Frequency:

• Before GitOps: X deployments/week
• After GitOps: 3-5x increase

Lead Time for Changes:

• Before: Code commit → production (hours/days)
• After: Minutes (automated sync)

Mean Time to Recovery (MTTR):

• Before: Hours (manual investigation and fix)
• After: Minutes (git revert + auto-sync)

Change Failure Rate:

• Before: X% of deployments cause incidents
• After: 50-70% reduction (review process, automated rollback)

Configuration Drift:

• Before: Unknown (no tracking)
• After: Zero (self-healing enabled)

The Bottom Line

GitOps is not just a deployment method—it's an operating model that makes infrastructure declarative, auditable, and reliable.

Organizations using GitOps achieve:

• Faster deployments: 3-5x increase in deployment frequency
• Safer releases: 50% reduction in change failure rate
• Faster recovery: 24x faster MTTR (git revert + auto-sync)
• Better audit: Complete history in Git
• Zero drift: Self-healing prevents configuration drift

The cost of GitOps: Initial setup (1-2 weeks), learning curve, discipline to enforce Git workflow.

The cost of not using GitOps: Manual deployments, configuration drift, slow rollbacks, poor auditability, team frustration.

If you're running Kubernetes at any scale, GitOps is the standard you should adopt.

Need Help Implementing GitOps?

If you're running Kubernetes and want to improve deployment velocity, reliability, and auditability, you don't have to figure out GitOps alone. I help organizations design and implement GitOps operating models that leverage tools like Argo CD and Flux.

Schedule a 30-minute GitOps consultation to discuss your Kubernetes deployment challenges and build a migration plan to GitOps.

Want insights on Kubernetes, DevOps, and cloud-native practices? Join my monthly newsletter for practical patterns, tools, and strategies for modern infrastructure management.