Storage Overview
Last Updated: September 11, 2026
The moment you run stateful workloads on Kubernetes, storage stops being "something you attach" and becomes a domain that dictates performance, cost, and availability. This section covers cloud storage in the order that matters in practice: how to choose → what it actually measures → how to operate it.
What's in this section
| Document | What it covers |
|---|---|
| EBS gp2 vs gp3 Measured Benchmark | Why two identical 100GiB volumes differ by 10x in performance — fio-measured IOPS/latency/throughput and the gp2 burst-credit cliff |
The Kubernetes storage fundamentals and hands-on EKS configuration are covered in depth elsewhere in this book. Read this section together with:
- Kubernetes Storage — PV/PVC, StorageClass, dynamic provisioning, access modes
- EKS Storage Part 1: EBS, EFS — CSI driver installation and basic usage
- EKS Storage Part 2: FSx for Lustre, S3, snapshots, performance
- EKS Storage Part 3: monitoring, troubleshooting, cost
The storage stack at a glance
Understanding the path from an application write to the physical volume tells you which layer to blame when performance disappoints:
application write()
→ mounted volume filesystem (ext4/xfs)
→ guest kernel and block device
→ EC2 EBS path (IOPS/bandwidth shared by all volumes)
→ EBS service and volume (per-volume IOPS/throughput limits)The volume's own limits and the instance-level EBS bandwidth/IOPS limits are separate budgets. An m5.xlarge has a baseline of roughly 6,000 IOPS — driving three gp3 volumes at 3,000 IOPS simultaneously requests 9,000 IOPS, above that sustained baseline. Check instance burst capacity and all other volume traffic as well.
Choosing AWS storage
| Service | Access mode | Characteristics | Best fit |
|---|---|---|---|
| EBS (gp3/io2) | RWO (single node) | Block; latency depends on type/load/queue depth | Databases, single-pod state |
| EFS | RWX (multi node) | NFS, ms-level latency, elastic capacity | Shared config/content, shared ML training data |
| FSx for Lustre | RWX | Parallel filesystem, high throughput | HPC, large-scale ML training |
| S3 (Mountpoint CSI) | RWX (read-heavy) | Object; different operation semantics from POSIX/NFS | Data lakes, models and artifacts |
| Instance store | Node-local | NVMe, lowest latency, ephemeral | Caches, shuffle data, scratch space |
RWO permits multiple Pods on one node; it is not a single-Pod guarantee. Exceptions such as io2 Multi-Attach require separate support and filesystem/application concurrency design. Mountpoint S3 is not a general POSIX shared filesystem: check modification, rename and locking support for the workload. Instance-store data may survive reboot but can be lost on stop/termination.
Why measure instead of reading spec sheets
Storage is where the gap between the datasheet and lived experience is widest. The classic traps:
- Small gp2 burst credits — volumes with a baseline below 3,000 IOPS can burst using available credits. Duration depends on initial balance, capacity and load. A full 100 GiB volume at 3,000 IOPS calculates to about 33 minutes; a short test can miss post-depletion performance.
- Volume limits vs instance limits — see the stack diagram above.
- Conclusions change with iodepth — a queue-depth-1 latency test and a queue-depth-32 IOPS test describe entirely different properties of the same volume.
The EBS gp2 vs gp3 measured benchmark demonstrates each of these traps with fio.