StatefulSets & Persistent Storage in Kubernetes

Kubernetes StatefulSets are the go-to solution for managing stateful applications that require stable identities, persistent storage, and ordered deployment patterns—essential for databases, distributed systems, and caching layers.

If you’re new to Kubernetes or setting up a cluster, consider exploring Kubernetes distributions like k3s or MicroK8s for development, or installing Kubernetes with Kubespray for production-grade clusters.

This nice image is generated by AI model Flux 1 dev.

What Are StatefulSets?

StatefulSets are a Kubernetes workload API object designed specifically for managing stateful applications. Unlike Deployments that treat all pods as interchangeable, StatefulSets maintain a unique identity for each pod with guarantees about ordering and uniqueness.

Key Features:

• Stable Network Identifiers: Each pod gets a predictable hostname that persists across restarts
• Persistent Storage: Dedicated PersistentVolumeClaims that follow pods through reschedules
• Ordered Deployment: Pods are created sequentially (0, 1, 2…) and terminate in reverse order
• Ordered Updates: Rolling updates proceed in order, ensuring application stability

StatefulSets are critical for applications like PostgreSQL, MySQL, MongoDB, Cassandra, Elasticsearch, Kafka, ZooKeeper, Redis, and etcd—any workload where pod identity and data persistence matter.

Understanding Persistent Storage in Kubernetes

Kubernetes provides a sophisticated storage abstraction layer that decouples storage management from pod lifecycle:

Storage Components

PersistentVolume (PV): A piece of storage in the cluster provisioned by an administrator or dynamically created via a StorageClass. PVs exist independently of pods.

PersistentVolumeClaim (PVC): A request for storage by a pod. PVCs bind to available PVs that match their requirements (size, access mode, storage class).

StorageClass: Defines different “classes” of storage with various provisioners (AWS EBS, GCE PD, Azure Disk, NFS, etc.) and parameters like replication, performance tiers, and backup policies.

Access Modes

• ReadWriteOnce (RWO): Volume mounted as read-write by a single node
• ReadOnlyMany (ROX): Volume mounted as read-only by many nodes
• ReadWriteMany (RWX): Volume mounted as read-write by many nodes (requires special storage backends)

StatefulSet Storage Architecture

StatefulSets use volumeClaimTemplates to automatically create PersistentVolumeClaims for each pod replica. This is fundamentally different from Deployments:

How volumeClaimTemplates Work

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-service
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "fast-ssd"
      resources:
        requests:
          storage: 10Gi

When you create this StatefulSet:

1. Kubernetes creates pod mysql-0 and PVC data-mysql-0
2. Then creates pod mysql-1 and PVC data-mysql-1
3. Finally creates pod mysql-2 and PVC data-mysql-2

Each pod gets its own dedicated 10GB persistent volume. If mysql-1 is deleted or rescheduled, Kubernetes recreates it and reattaches the same data-mysql-1 PVC, preserving all data.

Creating a Headless Service for StatefulSets

StatefulSets require a Headless Service to provide stable network identities:

apiVersion: v1
kind: Service
metadata:
  name: mysql-service
spec:
  clusterIP: None  # This makes it a headless service
  selector:
    app: mysql
  ports:
  - port: 3306
    name: mysql

This creates DNS entries for each pod:

• mysql-0.mysql-service.default.svc.cluster.local
• mysql-1.mysql-service.default.svc.cluster.local
• mysql-2.mysql-service.default.svc.cluster.local

Applications can connect directly to specific pod instances using these stable DNS names.

StatefulSet Deployment Patterns

For teams managing complex Kubernetes deployments, Helm Charts provide a powerful way to package and deploy StatefulSets with templating, versioning, and dependency management. Helm simplifies managing StatefulSet configurations across different environments.

Ordered Scaling

When scaling up from 3 to 5 replicas:

kubectl scale statefulset mysql --replicas=5

Kubernetes creates pods in order: mysql-3 → waits for Ready → mysql-4

When scaling down from 5 to 3:

kubectl scale statefulset mysql --replicas=3

Kubernetes terminates in reverse: mysql-4 → waits for termination → mysql-3

Rolling Updates

StatefulSets support two update strategies:

OnDelete: Manual updates—pods only update when you delete them RollingUpdate: Automatic sequential updates in reverse ordinal order

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2  # Only update pods with ordinal >= 2

The partition parameter enables canary deployments—you can update high-numbered pods first and test before rolling out to all replicas.

Storage Best Practices

Dynamic Provisioning

Always use StorageClasses for dynamic volume provisioning:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
allowVolumeExpansion: true
reclaimPolicy: Retain

allowVolumeExpansion: Enables resizing PVCs without recreating them reclaimPolicy: Retain keeps PV data after PVC deletion, Delete removes it automatically

PVC Retention Policies

Kubernetes 1.23+ supports persistentVolumeClaimRetentionPolicy:

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain    # Keep PVCs when StatefulSet is deleted
    whenScaled: Delete     # Delete PVCs when scaling down

Options:

• Retain: Keep PVCs (default behavior, safest)
• Delete: Automatically delete PVCs (useful for dev environments)

Backup Strategies

Volume Snapshots: Use VolumeSnapshot resources to create point-in-time backups

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: mysql-snapshot-20251113
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: data-mysql-0

Application-Level Backups: Use tools like mysqldump, pg_dump, or Velero for database-specific backups

Distributed Replication: Configure application-level replication (MySQL replication, PostgreSQL streaming replication) as the first line of defense

Real-World Use Cases

Database Cluster (PostgreSQL)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:16
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "standard"
      resources:
        requests:
          storage: 20Gi

Distributed Cache (Redis)

For Redis clusters, you need both StatefulSet and careful configuration:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis-service
  replicas: 6
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:7-alpine
        command:
        - redis-server
        - "--appendonly"
        - "yes"
        - "--appendfsync"
        - "everysec"
        ports:
        - containerPort: 6379
          name: redis
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 5Gi

Message Queue (Kafka)

Kafka requires both persistent storage for logs and stable network identities for broker coordination:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka-service
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:7.5.0
        ports:
        - containerPort: 9092
          name: kafka
        env:
        - name: KAFKA_BROKER_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: data
          mountPath: /var/lib/kafka/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Gi

Monitoring and Troubleshooting

For a comprehensive reference of Kubernetes commands used in this section, see the Kubernetes Cheatsheet.

Check StatefulSet Status

# View StatefulSet details
kubectl get statefulset mysql
kubectl describe statefulset mysql

# Check pod creation order and status
kubectl get pods -l app=mysql -w

# View PVC status
kubectl get pvc
kubectl describe pvc data-mysql-0

Common Issues

Pod Stuck in Pending: Check PVC status and storage availability

kubectl describe pod mysql-0
kubectl get events --sort-by='.lastTimestamp'

Storage Full: Expand PVC if StorageClass allows

kubectl patch pvc data-mysql-0 -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

Pod Won’t Terminate: Check for application-level locks or volume unmount issues

kubectl delete pod mysql-0 --grace-period=0 --force

Metrics to Monitor

• Storage Usage: Monitor PVC capacity and usage percentage
• I/O Performance: Track IOPS, throughput, and latency
• Pod Restarts: Frequent restarts may indicate storage issues
• PVC Binding Time: Slow binding suggests provisioning problems

Migration Strategies

When migrating to StatefulSets, ensure your Kubernetes cluster is properly configured. For homelab or small cluster setups, review our comprehensive comparison of Kubernetes distributions to choose the right platform for your workload requirements.

From Deployment to StatefulSet

1. Create StatefulSet with volumeClaimTemplates
2. Scale down Deployment gracefully
3. Restore data from backups to StatefulSet pods
4. Update DNS/Service references
5. Delete old Deployment and PVCs

Backup Before Migration

# Snapshot existing PVCs
kubectl get pvc -o yaml > pvc-backup.yaml

# Create volume snapshots
kubectl apply -f volume-snapshot.yaml

Security Considerations

Storage Encryption

Enable encryption at rest using StorageClass parameters:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: encrypted-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  encrypted: "true"
  kmsKeyId: arn:aws:kms:us-east-1:123456789012:key/abcd-1234

Access Control

Use RBAC to restrict who can create/modify StatefulSets and PVCs:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: statefulset-manager
rules:
- apiGroups: ["apps"]
  resources: ["statefulsets"]
  verbs: ["get", "list", "create", "update", "delete"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "create", "delete"]

Network Policies

Restrict pod-to-pod communication:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: mysql-network-policy
spec:
  podSelector:
    matchLabels:
      app: mysql
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - protocol: TCP
      port: 3306

Performance Optimization

Storage Performance Tiers

Choose appropriate StorageClasses based on workload:

• High IOPS: Databases with heavy random read/write (gp3, io2)
• High Throughput: Log aggregation, analytics (st1, sc1)
• Balanced: General purpose applications (gp3)

Pod Distribution

Use pod anti-affinity to spread StatefulSet pods across availability zones:

spec:
  template:
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: mysql
            topologyKey: topology.kubernetes.io/zone

Resource Requests and Limits

Set appropriate resources for consistent performance:

resources:
  requests:
    cpu: "2"
    memory: "4Gi"
    ephemeral-storage: "10Gi"
  limits:
    cpu: "4"
    memory: "8Gi"
    ephemeral-storage: "20Gi"

Advanced Patterns

Stateful Application with Init Containers

Use init containers for database initialization:

spec:
  template:
    spec:
      initContainers:
      - name: init-mysql
        image: mysql:8.0
        command:
        - bash
        - "-c"
        - |
          if [[ ! -d /var/lib/mysql/mysql ]]; then
            mysqld --initialize-insecure --datadir=/var/lib/mysql
          fi          
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql

Multi-Container Pods for Sidecars

Add backup sidecars or monitoring agents:

spec:
  template:
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        # ... mysql config ...
      - name: backup-sidecar
        image: mysql-backup:latest
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          readOnly: true

ConfigMaps for Dynamic Configuration

Separate configuration from StatefulSet definition:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
data:
  my.cnf: |
    [mysqld]
    max_connections=200
    innodb_buffer_pool_size=2G    
---
spec:
  template:
    spec:
      containers:
      - name: mysql
        volumeMounts:
        - name: config
          mountPath: /etc/mysql/conf.d
      volumes:
      - name: config
        configMap:
          name: mysql-config

Useful Links

• Kubernetes StatefulSets Documentation
• Persistent Volumes
• Storage Classes
• Volume Snapshots
• StatefulSet Best Practices
• CSI Drivers
• Velero Backup Solutions
• Rook Ceph Storage
• Kubernetes Cheatsheet
• Install Kubernetes with Kubespray
• Comparison of Kubernetes Distributions for a 3-Node Homelab
• Kubernetes distributions - quick overview of kubeadm, k3s, MicroK8s, Minikube, Talos Linux and RKE2
• Master Helm Charts: Kubernetes Package Management Guide