Amazon S3 Integration Methods

Amazon S3 Integration Methods An architecture diagram generated by Archify. Application pods · Web · data · ML · logs · Architecture component Application pods Web · data · ML · logs IRSA / Pod Identity · Temporary credentials · Architecture component IRSA / Pod Identity Temporary credentials Mountpoint S3 CSI · Mounts as a file system · Architecture component Mountpoint S3 CSI Mounts as a file system AWS SDK · Hadoop S3A · Direct object API calls · Architecture component AWS SDK · Hadoop S3A Direct object API calls Amazon S3 · Data lake · backup · models · Architecture component Amazon S3 Data lake · backup · models Assumes role Mounts volume Library calls SigV4 authorization GET / PUT GET / PUT Legend Frontend Backend Database Security

Choosing an approach

  • • Mountpoint for Amazon S3 CSI helps when existing code expects files
  • • Direct AWS SDK calls give fine control over multipart uploads and conditional requests
  • • Hadoop S3A is the standard path for analytics engines such as Spark and Hive

Permission design

  • • Take temporary credentials through IRSA or EKS Pod Identity instead of static access keys
  • • Grant least privilege per prefix in both the bucket policy and the IAM policy
  • • s3:GetObject must name the object path, as in arn:aws:s3:::bucket/*

Common use cases

  • • Raw data lake storage and landing analytics output
  • • Backup and archive, with lifecycle policies tiering to Glacier
  • • Machine learning model artifacts and long-term log retention