State, Checkpointing, and Streaming Patterns Quiz
This quiz tests your understanding of the HashMap vs RocksDB state backend trade-off, the difference between checkpoints and savepoints, how Kafka exactly-once delivery works through Flink's two-phase-commit protocol, and when Flink SQL/Table API is the better fit over the DataStream API.
Multiple Choice Questions
- What is the core trade-off between
HashMapStateBackendandEmbeddedRocksDBStateBackend?- A) HashMap is off-heap and RocksDB is on-heap
- B) HashMap is faster but limited by heap memory; RocksDB is slower per-access but can spill state to disk, scaling beyond memory
- C) RocksDB doesn't support checkpointing at all
- D) There is no functional difference — it's purely a naming choice
Show Answer
Answer: B) HashMap is faster but limited by heap memory; RocksDB is slower per-access but can spill state to disk, scaling beyond memory
Explanation: HashMapStateBackend keeps state as native Java objects on the JVM heap, which is fast but bounded by available memory and adds GC pressure as state grows. EmbeddedRocksDBStateBackend stores state off-heap in a local RocksDB instance that spills to disk, trading some per-record latency (serialization overhead) for the ability to hold state far larger than memory.
- Which state backend is required to enable incremental checkpoints?
- A) HashMapStateBackend
- B) EmbeddedRocksDBStateBackend
- C) Either one, with no configuration difference
- D) Neither — incremental checkpoints are always on
Show Answer
Answer: B) EmbeddedRocksDBStateBackend
Explanation: Incremental checkpointing (execution.checkpointing.incremental: true) relies on persisting only the RocksDB SSTable files that changed since the previous checkpoint. HashMapStateBackend has no equivalent on-disk file structure to diff against, so it only supports full checkpoints.
- What exactly does an incremental checkpoint persist?
- A) A full copy of every key's current value
- B) Only the changed RocksDB SSTable files since the last checkpoint, plus a manifest referencing still-valid prior files
- C) Only the job's configuration, not its state
- D) A diff computed by re-reading every key from the previous checkpoint
Show Answer
Answer: B) Only the changed RocksDB SSTable files since the last checkpoint, plus a manifest referencing still-valid prior files
Explanation: Rather than re-uploading the entire state, an incremental checkpoint persists only the SSTable deltas produced since the previous checkpoint, along with a manifest recording which earlier SSTable files are still needed to reconstruct full state on restore.
- Under what condition can restoring from an incremental checkpoint actually be slower than restoring from a full checkpoint?
- A) It is never slower under any condition
- B) When checkpoint storage access is network-bound, since recovery may require fetching many more individual files
- C) When the job has no state at all
- D) When
execution.checkpointing.modeis set toAT_LEAST_ONCE
Show Answer
Answer: B) When checkpoint storage access is network-bound, since recovery may require fetching many more individual files
Explanation: Recovering from an incremental checkpoint means fetching the latest delta plus every prior file the manifest still references — more individual fetches than a full checkpoint's single snapshot. If the path to storage (e.g., S3) is network-bound, that extra fetch count can make recovery slower. If the bottleneck is instead CPU/IOPS on the TaskManager, incremental checkpoints typically recover faster because there's less total data to write back to RocksDB.
- What is the fundamental difference between a checkpoint and a savepoint?
- A) They are the same mechanism with different names
- B) Checkpoints are automatic and used for failure recovery; savepoints are explicit, user-triggered snapshots for planned upgrades and migrations
- C) Savepoints are stored in memory, checkpoints on disk
- D) Checkpoints can only be taken once per job's lifetime
Show Answer
Answer: B) Checkpoints are automatic and used for failure recovery; savepoints are explicit, user-triggered snapshots for planned upgrades and migrations
Explanation: Both use the same filesystem-based checkpoint storage backend, but they serve different purposes: checkpoints are triggered automatically by Flink on a fixed interval and used for failure recovery, while savepoints are explicitly triggered (by a user or the Operator's FlinkStateSnapshot) and retained as durable artifacts for deliberate upgrades, migrations, or version bumps.
- The Flink Kubernetes Operator's
last-stateupgrade mode restores from which artifact?- A) The most recent savepoint
- B) The most recent checkpoint
- C) A full re-read of the source topic from the beginning
- D) A manually exported state file
Show Answer
Answer: B) The most recent checkpoint
Explanation:last-state upgrade mode restores from the last completed checkpoint rather than a savepoint, which is what makes it fast and fully automatic — at the cost of being tied to the specific job graph that produced that checkpoint. Deliberate version bumps or migrations should use an explicit savepoint instead.
- In Flink's
KafkaSinktwo-phase-commit protocol for exactly-once delivery, what does the KafkaCommitter do?- A) It writes records to Kafka inside an open transaction
- B) It commits the Kafka transaction only after the corresponding Flink checkpoint completes successfully
- C) It deletes old checkpoints from S3
- D) It negotiates the initial TCP connection to the broker
Show Answer
Answer: B) It commits the Kafka transaction only after the corresponding Flink checkpoint completes successfully
Explanation: The KafkaWriter writes records inside an open Kafka transaction between checkpoints; those records stay invisible to read_committed consumers. Only once the enclosing Flink checkpoint completes does the KafkaCommitter commit the transaction, making the records visible downstream.
- Why does
KafkaSinkwithEXACTLY_ONCErequire a stabletransactionalIdPrefix?- A) It's purely cosmetic and shows up in logs only
- B) Flink derives each subtask's transactional ID from it, and on restore it must match prior IDs so open transactions from before a failure can be correctly resolved
- C) It sets the Kafka topic's replication factor
- D) It determines the number of partitions in the sink topic
Show Answer
Answer: B) Flink derives each subtask's transactional ID from it, and on restore it must match prior IDs so open transactions from before a failure can be correctly resolved
Explanation: Each sink subtask's actual Kafka transactional ID is derived from the configured prefix. If this prefix changes between runs, Flink can't line up the IDs used before a restart with the ones it uses after, breaking its ability to correctly abort or resolve transactions left open by a failure.
- What is the direct latency cost of using
KafkaSinkwithDeliveryGuarantee.EXACTLY_ONCE?- A) None — output latency is unaffected
- B) Output becomes visible to
read_committedconsumers roughly one checkpoint interval later, since commits only happen after checkpoints complete - C) It doubles the number of Kafka partitions required
- D) It requires disabling checkpointing entirely
Show Answer
Answer: B) Output becomes visible to read_committed consumers roughly one checkpoint interval later, since commits only happen after checkpoints complete
Explanation: Because a Kafka transaction only commits once its enclosing Flink checkpoint completes, downstream consumers reading with read_committed see output delayed by roughly one checkpoint interval — a 60-second interval means up to ~60 seconds of added end-to-end latency.
- What risk does setting the checkpoint interval too short introduce when using exactly-once
KafkaSink?- A) It has no downside — shorter is always better
- B) It can flood the Kafka broker's transaction coordinator with transactional IDs to track, since each checkpoint cycle opens a fresh transaction per sink subtask
- C) It disables incremental checkpoints automatically
- D) It forces the job to switch to
AT_LEAST_ONCEmode
Show Answer
Answer: B) It can flood the Kafka broker's transaction coordinator with transactional IDs to track, since each checkpoint cycle opens a fresh transaction per sink subtask
Explanation: Every checkpoint cycle opens a new transaction per sink subtask. Pushing the checkpoint interval down to just a few seconds across many parallel subtasks means the coordinator has to track many more transactional IDs, adding load. Checkpoint interval should be tuned as a balance between output latency and coordinator load, not purely for faster recovery.
- What capability does Flink's Dynamic Iceberg Sink add over the standard Iceberg sink?
- A) It can only write to a single, statically defined table
- B) It can write to multiple Iceberg tables from one sink, choosing the destination table per record and evolving schema automatically based on record content
- C) It removes the need for a schema entirely
- D) It replaces the need for a Kafka source
Show Answer
Answer: B) It can write to multiple Iceberg tables from one sink, choosing the destination table per record and evolving schema automatically based on record content
Explanation: The Dynamic Iceberg Sink extends the standard Iceberg sink to route each record to a table determined at runtime from its content, and to evolve that table's schema automatically as needed — a strong fit for CDC fan-out from a shared Kafka topic across many source tables.
- When is a simpler connector-based pipeline (e.g., MSK → Data Firehose → S3 Tables/Iceberg, or MSK Connect with an Iceberg sink connector) preferable to Flink on EKS?
- A) Never — Flink is always the better choice
- B) When the pipeline is a straight passthrough or simple format conversion with no real computation needed
- C) Only when using DataStream API instead of Table API
- D) Only when the source is not Kafka
Show Answer
Answer: B) When the pipeline is a straight passthrough or simple format conversion with no real computation needed
Explanation: Flink earns its operational cost when a pipeline needs actual computation — joins, windowed aggregation, per-record routing, complex event-time handling, or dynamic multi-table fan-out. For simple passthrough or format conversion, a fully managed Firehose pipeline or an MSK Connect sink connector is less to build and operate.
- For which kind of use case is Flink SQL / Table API the recommended starting point?
- A) Typical ETL, aggregation, and windowing that can be expressed declaratively
- B) Only for jobs that need custom operators with fine-grained checkpoint control
- C) Only for jobs with no state at all
- D) Table API cannot connect to Kafka or Iceberg
Show Answer
Answer: A) Typical ETL, aggregation, and windowing that can be expressed declaratively
Explanation: Flink SQL/Table API is the recommended entry point for most jobs because it requires far less code and ships with built-in connectors for Kafka, Iceberg, JDBC, and more. DataStream API remains necessary when a job needs custom operators, complex event-time/state logic, or fine control over checkpointing and backpressure behavior that the Table API planner doesn't expose.
- Why would a job need to drop down to the DataStream API instead of staying in Flink SQL / Table API?
- A) DataStream API is always faster for every workload
- B) The job needs custom operators, complex event-time/state logic, or fine-grained control over checkpointing/backpressure that the SQL planner doesn't expose
- C) SQL cannot read from Kafka
- D) DataStream API requires less code in every case
Show Answer
Answer: B) The job needs custom operators, complex event-time/state logic, or fine-grained control over checkpointing/backpressure that the SQL planner doesn't expose
Explanation: The Table API planner decides operator behavior on your behalf, which is fine for standard ETL/aggregation logic but limiting when a job needs a hand-written operator, direct access to custom state, or manual control over checkpoint alignment and backpressure — cases where DataStream API's lower-level control is necessary.
- What role do watermarks play in event-time windowing?
- A) They set the number of partitions for a window
- B) They are a heuristic signal asserting no more records older than the watermark should arrive, letting windows know when it's safe to close and emit results
- C) They control which state backend is used
- D) They determine the Kafka transactional ID prefix
Show Answer
Answer: B) They are a heuristic signal asserting no more records older than the watermark should arrive, letting windows know when it's safe to close and emit results
Explanation: Watermarks are periodically injected into the stream and assert that no further records with an older event-time timestamp should arrive. Windows rely on this signal, rather than wall-clock processing time, to decide when to close and emit a correct result — which keeps output consistent even under consumer lag or replay of historical data.