Flink state and sink-commit boundaries

Flink state and sink-commit boundaries An architecture diagram generated by Archify. MSK / Kafka · Source topic · Architecture component MSK / Kafka Source topic Flink runtime · Match the connector profile · Architecture component Flink runtime Match the connector profile RocksDB keyed state · Memory / local SST · TaskManagers RocksDB keyed state Memory / local SST S3 checkpoint storage · Shared SST + metadata · Architecture component S3 checkpoint storage Shared SST + metadata Kafka Output Topic · read_committed consumers · Architecture component Kafka Output Topic read_committed consumers S3 / Iceberg · Catalog / table commits · Architecture component S3 / Iceberg Catalog / table commits consume Keyed state Checkpoint / restore Checkpoint → transaction commit DynamicRecord → table commit TaskManagers

Checkpoint state

  • • Shared SST references and compaction
  • • Storage, retention and restore requirements

Sink commit

  • • Kafka: checkpoint + transaction + read_committed
  • • No global transaction across sinks/tables

Checked version profiles

  • • Kafka 5.0.0-2.2 / Flink 2.2.1
  • • Iceberg 1.11.0 / Flink 2.1.3