Overview
Streaming Data Processing is 5% of the ABDE exam. It covers implementing streaming pipelines that capture, process and deliver continuously flowing data, working with stream processing frameworks such as Apache Flink, Spark Structured Streaming and Kafka Streams, using message brokers, and building resilient real-time data flows. DASCA names Apache Kafka, so expect its core vocabulary: topics, partitions, offsets, producers, consumer groups and retention.
The domain exists because use cases such as fraud detection, live operational dashboards and anomaly alerts lose their value if data arrives hours later. Streaming brings problems batch rarely faces: unbounded data, events arriving out of order, duplicate delivery after failures and state that must survive restarts. It complements batch ingestion in Domain 1 and supplies the speed layer or single stream in the Lambda and Kappa architectures of Domain 5.
How it's tested
Expect compact scenarios about throughput, ordering, late events, duplicates or failures in a real-time pipeline. You choose the partitioning key, delivery guarantee, window type or resilience mechanism that BEST meets the requirement. Read for the words 'per customer', 'late', 'duplicate' and 'restart', which usually point straight to the concept under test.
-
Scale and order the stream
Choose partition keys and consumer group sizing to balance throughput with per-key ordering.
-
Get the time right
Pick event-time windows, watermarks and allowed lateness so results are correct when events arrive late or out of order.
-
Survive failure
Select checkpointing, replication, idempotent sinks or dead-letter topics to avoid loss or duplicates.
A payments stream must process each account's events in order, and throughput must scale. Kafka only guarantees ordering within a partition, so the producer should key messages by account ID, sending each account's events to one partition while spreading accounts across many partitions. A single-partition topic would preserve order but cap throughput at one consumer.
Practical applications 4
Official practical applications from the DASCA outline, in outline order.
6.1 Implementing streaming pipelinesCapturing, processing and delivering continuously flowing data.
- What it means
- A streaming pipeline continuously captures events from sources such as applications, devices or CDC logs, processes them as they arrive, for example filtering, enriching or aggregating them, and delivers results to sinks such as databases, dashboards, lakes or alerting systems.
- What to understand
- Streams are unbounded, so aggregations need windows, such as tumbling (fixed, non-overlapping), sliding or hopping (overlapping) and session windows (closed by a gap in activity). Event time, when the event happened, differs from processing time, when the system sees it, and correct business results use event time. Enrichment joins against reference data must handle that reference data changing too.
- How it shows up
- A scenario asks for per-minute counts that stay correct when mobile devices upload events after reconnecting, which requires event-time windows rather than processing-time windows.
- Common confusion
- Using processing time for business metrics, so late-arriving events land in the wrong window.
6.2 Stream processing frameworksEngines that run stateful computations over streams.
- What it means
- Stream processing frameworks such as Apache Flink, Spark Structured Streaming and Kafka Streams execute continuous transformations, joins and aggregations with managed state, checkpoints and time semantics.
- What to understand
- Flink processes events one at a time with low latency and rich event-time support; Spark Structured Streaming uses micro-batches by default and suits teams already using Spark; Kafka Streams is a library embedded in applications that reads from and writes to Kafka. Watermarks track how far event time has progressed and decide when a window can close, and allowed lateness defines how long late events can still update results.
- How it shows up
- You choose a framework or setting from latency requirements, existing skills and whether the pipeline only moves data between Kafka topics.
- Common confusion
- Thinking a watermark is a rate limit or a backlog measure; it is an estimate of event-time progress used to close windows.
6.3 Message brokersDurable, partitioned transport between producers and consumers.
- What it means
- A message broker such as Apache Kafka stores streams of records in topics split into partitions, lets producers append records and lets consumers read them at their own pace by tracking offsets.
- What to understand
- Kafka preserves ordering only within a partition, and records with the same key go to the same partition. Within one consumer group, each partition is read by at most one consumer, so adding consumers beyond the partition count leaves the extra ones idle. Kafka retains records for a configured time or size regardless of whether they were read, which enables replay, while a log-compacted topic keeps the latest record per key.
- How it shows up
- A consumer group with 12 consumers reading a 6-partition topic is not keeping up, and the right answer is to increase partitions, since the extra consumers sit idle.
- Common confusion
- Treating Kafka like a traditional queue where messages disappear once read, or expecting ordering across the whole topic.
6.4 Building resilient real-time data flowsNo loss, controlled duplicates and graceful recovery.
- What it means
- Resilient flows keep running correctly through broker, network and processor failures using replication, acknowledgments, checkpoints, retries, idempotent writes and dead-letter handling.
- What to understand
- Delivery guarantees are at-most-once (may lose), at-least-once (may duplicate) and exactly-once (each event affects results once), and end-to-end exactly-once needs a capable engine plus a transactional or idempotent sink. Kafka durability comes from a replication factor, commonly 3, with producer acks set to all. Monitor consumer lag, the gap between the latest offset and the consumer's committed offset, as the main health signal, and send unprocessable messages to a dead-letter topic rather than blocking the stream.
- How it shows up
- After a processor restart, aggregates double count, and you choose checkpointing with an idempotent or transactional sink.
- Common confusion
- Assuming exactly-once is automatic once Kafka is used, when the sink and processor configuration determine the end-to-end guarantee.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Topic, partition and offset
A Kafka topic is a named stream split into partitions, each an ordered, append-only log in which every record has a sequential offset.
- Remember
- Partitions are the unit of parallelism and ordering, and consumers track progress by committing offsets.
- Tested as
- The vocabulary base for every broker question.
- Trap
- Assuming order holds across partitions.
-
Partition key
The partition key determines which partition a record goes to, so all records with the same key land in the same partition.
- Remember
- Key by the entity whose events must stay ordered, such as account ID, and watch for hot keys that overload one partition.
- Tested as
- Ordering-with-scale scenarios point here.
- Trap
- Using a random or null key when per-entity order matters.
-
Consumer group
A consumer group is a set of consumers sharing a topic's partitions so each partition is read by one member of the group.
- Remember
- Maximum parallelism equals the number of partitions, and separate groups each receive the full stream independently.
- Tested as
- Scaling and fan-out questions.
- Trap
- Adding more consumers than partitions to increase throughput.
-
Consumer lag
Consumer lag is the difference between the latest offset in a partition and the offset the consumer group has committed.
- Remember
- Rising lag means consumers cannot keep up and data is getting staler.
- Tested as
- The key monitoring metric for streaming health.
- Trap
- Monitoring only whether the consumer process is running.
-
Event time vs. processing time
Event time is when an event actually happened, while processing time is when the system processes it.
- Remember
- Use event time for business results, since network delays and offline devices make processing time misleading.
- Tested as
- Late and out-of-order event questions.
- Trap
- Windowing on arrival time and misplacing late events.
-
Watermark and allowed lateness
A watermark is the engine's estimate that no more events older than a given event time are expected, and allowed lateness keeps windows updatable for a while after that.
- Remember
- Longer lateness improves completeness but delays final results and holds more state.
- Tested as
- Tested with mobile, IoT and network-delay scenarios.
- Trap
- Confusing a watermark with a throughput limit.
-
Tumbling, sliding and session windows
Tumbling windows are fixed and non-overlapping, sliding (hopping) windows overlap at a set interval, and session windows group events separated by less than an inactivity gap.
- Remember
- Choose sessions for user activity bursts, tumbling for regular reports and sliding for rolling metrics.
- Tested as
- Questions ask which window fits a metric.
- Trap
- Using tumbling windows for 'last 10 minutes, updated every minute', which needs sliding windows.
-
Delivery guarantees
At-most-once may lose events, at-least-once may duplicate them, and exactly-once ensures each event affects results once.
- Remember
- At-least-once plus idempotent sinks is a common practical choice; end-to-end exactly-once needs transactional support throughout.
- Tested as
- Duplicate and loss scenarios.
- Trap
- Believing using Kafka alone gives end-to-end exactly-once.
-
Checkpointing
Checkpointing periodically saves a stream processor's state and source positions to durable storage so it can resume after failure.
- Remember
- Recovery restarts from the last checkpoint, so sinks must tolerate replayed output.
- Tested as
- Restart and recovery questions.
- Trap
- Running stateful jobs without durable checkpoints.
-
Replication factor and acks
The replication factor sets how many brokers hold copies of each partition, and producer acks set how many must confirm a write.
- Remember
- Replication factor 3 with acks=all and a minimum in-sync replica setting of 2 is a common durable configuration.
- Tested as
- Data loss questions on broker failure.
- Trap
- Using acks=0 or 1 for critical data to gain speed.
-
Dead-letter topic
A dead-letter topic receives messages that repeatedly fail processing, with error details, so the main stream keeps flowing.
- Remember
- Monitor and reprocess it; a dead-letter topic nobody watches is silent data loss.
- Tested as
- Poison-message scenarios that block a partition.
- Trap
- Retrying a malformed message forever and stalling the partition.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
At-most-once vs. at-least-once vs. exactly-once
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| At-most-once | Commit position before processing | No duplicates, but events can be lost on failure | Using it for financial or audit data |
| At-least-once | Commit position after processing | No loss, but replays can duplicate effects | Pairing it with a non-idempotent sink |
| Exactly-once | Processing and output committed atomically with source positions | Each event affects results once, with added latency and complexity | Assuming it holds when the sink is not transactional or idempotent |
Event time vs. processing time vs. ingestion time
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Event time | When the event occurred at the source | Correct for business results; needs watermarks for lateness | Not handling events that arrive after the window closes |
| Processing time | When the operator processes the event | Simple and lowest latency, but results depend on delays | Using it for billing or compliance metrics |
| Ingestion time | When the event entered the broker or pipeline | A compromise that ignores delay before ingestion | Treating it as the true time of the event |
Kafka vs. a traditional message queue
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Kafka (log-based broker) | Records retained for a set period and read by offset | Multiple consumer groups can replay the same data independently | Expecting records to be deleted once consumed |
| Traditional queue | Messages removed after a consumer acknowledges them | Work distribution without replay | Choosing it when several systems must read the same events or replay history |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
More consumers, more speed
- What candidates pick
- Scale a consumer group from 8 to 16 instances to clear lag on an 8-partition topic.
- Why it looks right
- Adding workers increases throughput in most systems.
- Why it's wrong
- Each partition is read by one consumer in the group, so the extra 8 sit idle; increasing partitions or speeding up processing is required.
Clue to watch for The consumer count already equals or exceeds the partition count.
-
Global ordering
- What candidates pick
- Rely on Kafka to deliver all events in the order produced across the topic.
- Why it looks right
- Kafka is described as an ordered log.
- Why it's wrong
- Kafka preserves ordering only within a partition; per-entity order needs keying by that entity.
Clue to watch for Events for the same account or device processed out of order.
-
Single partition fix
- What candidates pick
- Reduce the topic to one partition to guarantee ordering.
- Why it looks right
- It does guarantee order.
- Why it's wrong
- It limits throughput to one consumer; keying by the entity preserves the needed order while keeping parallelism.
Clue to watch for The requirement is order per customer, plus high or growing volume.
-
Processing-time windows
- What candidates pick
- Count events per minute by arrival time for a mobile app metric.
- Why it looks right
- It is simpler and has the lowest latency.
- Why it's wrong
- Devices upload late after reconnecting, so counts shift into wrong minutes; event-time windows with watermarks give correct results.
Clue to watch for Offline devices, network delays or out-of-order arrival.
-
Kafka equals exactly-once
- What candidates pick
- Assume duplicates after a restart cannot happen because the pipeline uses Kafka.
- Why it looks right
- Kafka advertises exactly-once semantics.
- Why it's wrong
- Exactly-once needs transactional producers and consumers or engine checkpoints plus an idempotent or transactional sink; a plain database insert sink still duplicates on replay.
Clue to watch for Duplicated rows in the sink after a failure or redeploy.
-
Retry the poison message
- What candidates pick
- Retry a malformed message indefinitely until it succeeds.
- Why it looks right
- Retrying seems to avoid data loss.
- Why it's wrong
- A malformed message fails every time and blocks its partition; route it to a dead-letter topic with error details and continue.
Clue to watch for One partition's lag grows while others are healthy, with the same error repeating.
Answer framework
For ABDE streaming questions, check these points in order.
- Confirm that the stated latency requirement truly needs streaming, rather than micro-batch or scheduled batch processing, before designing anything else.
- Choose the partition key from the ordering requirement, and size the partition count for the parallelism that consumers will need at peak.
- Use event time with watermarks and allowed lateness, and pick tumbling, sliding or session windows to match the metric.
- Set the delivery guarantee the business needs, pairing at-least-once with idempotent sinks or using end-to-end exactly-once where duplicates are unacceptable.
- Build resilience with replication, producer acks, checkpointing and dead-letter topics, and monitor consumer lag as the primary health signal for the pipeline.
Practice question
Want more like this? The ABDE practice test has questions for every domain.
Checklist
0 of 11
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Because this domain is 5% of the exam, give it a short, focused study block rather than a long one, and concentrate on the concepts that turn into questions.
Start with Kafka fundamentals: topics, partitions, offsets, keys, consumer groups and retention. If you can, run a local Kafka container, create a topic with three partitions, and start four consumers in one group to see one sit idle. That single exercise answers several common question types.
Next, study time semantics: event time versus processing time, windows and watermarks. Candidates often underestimate late data and choose processing-time answers. Draw a timeline with a late event and decide which window it belongs to under each approach.
Then cover resilience: delivery guarantees, replication and acks, checkpoints, idempotent sinks and dead-letter topics. Write one sentence for each explaining what failure it prevents.
Afterwards, complete a set of timed streaming questions and review each miss by naming the keyword you overlooked, such as per-key ordering, lateness or duplicates. Then link this material to CDC in Domain 1 and to the Lambda and Kappa architectures in Domain 5.
FAQ
How much of the ABDE exam is streaming?
DASCA weights Streaming Data Processing at 5%, which suggests about 4 of the 75 questions. DASCA does not publish per-domain counts, and streaming terms can also appear in ingestion and architecture questions.
Do I need to know Kafka administration?
DASCA names Apache Kafka for this domain, so know its concepts well: topics, partitions, keys, offsets, consumer groups, retention, replication and acks. Deep cluster administration is unlikely to be the focus of an associate exam.
Does Kafka guarantee message order?
Only within a partition. Records with the same key go to the same partition, so keying by the entity that needs ordering, such as an account, preserves order for that entity.
What is a watermark in stream processing?
A watermark is the engine's estimate of how far event time has progressed, used to decide when a window's results can be emitted. Events older than the watermark plus any allowed lateness are treated as late.
Is exactly-once delivery always necessary?
No. Many pipelines use at-least-once delivery with idempotent sinks, which gives correct results more simply. End-to-end exactly-once is worth its complexity when duplicates are unacceptable and the sink supports transactions.
When is streaming the wrong choice?
When the business only needs hourly or daily freshness, batch or micro-batch is cheaper and simpler to operate. Streaming is justified by a real latency need such as fraud detection or live alerts.
How does streaming relate to the other ABDE domains?
CDC from Domain 1 often flows through Kafka, Domain 5 uses streams in Lambda and Kappa architectures, and Domain 7 relies on streaming feeds for real-time features and model monitoring.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 6 knowledge
You have covered streaming pipelines, processing frameworks, Kafka brokers and resilient real-time flows. Test whether you can pick the right key, window and guarantee under exam timing.