Overview
Domain 5 is worth 10% of the exam and contains four official objectives. The first explains the Structured Streaming engine: its programming model of an unbounded input table, micro-batch processing, exactly-once semantics and fault tolerance through checkpoints. The second covers creating streaming DataFrames with readStream and writing them with writeStream, using output modes and sinks.
The third objective covers selection, projection, windows and aggregations on streams, and the fourth covers streaming deduplication with and without a watermark. Questions are usually short code snippets, so you need to know which output mode suits an aggregation, why a checkpoint location is required, and how watermarks bound the state Spark keeps between micro-batches.
How it's tested
Most items show readStream or writeStream code and ask what happens, what is missing or which option is valid. Look closely at the output mode, whether an aggregation is present, whether withWatermark appears before the aggregation or deduplication, and whether a checkpoint location is set. Those four details decide most answers.
-
Choose the output mode
A streaming query with or without aggregation is shown, and you select append, update or complete.
-
Bound the state
A deduplication or windowed aggregation lacks a watermark, and you identify the state growth problem or the fix.
-
Explain the guarantee
You identify how checkpoints, replayable sources and idempotent sinks combine to give exactly-once results.
A query counts events per 10-minute window and writes with outputMode('append') but has no watermark, so Spark rejects it. In append mode a window's result can only be emitted once it will never change, and without a watermark Spark cannot know that. Adding withWatermark('event_time', '15 minutes') before the groupBy lets Spark finalise windows and drop old state, so the query starts.
Objectives 4
Official objectives from the Databricks outline, in outline order.
5.1 Explain the Structured Streaming engine in Spark, including its functions, programming model, micro-batch processing, exactly-once semantics, and fault tolerance mechanismsUnbounded table model, micro-batches, checkpoints and exactly-once.
- What it means
- This objective covers how Structured Streaming works. A stream is treated as an unbounded table that grows with new rows, and a streaming query is the same DataFrame logic run incrementally. By default the engine processes data in micro-batches, each planned and executed like a small batch job.
- What to understand
- Fault tolerance relies on a checkpoint location where Spark records source offsets and state for each micro-batch in write-ahead logs. On restart it resumes from the last committed batch. End-to-end exactly-once results require a replayable source such as Kafka or files, plus an idempotent or transactional sink such as Delta Lake. Triggers control timing: default, processingTime, or availableNow.
- How it shows up
- Expect questions on what the checkpoint stores, or which combination of source and sink gives exactly-once output.
- Common confusion
- Thinking exactly-once comes from Spark alone; the source and sink must cooperate.
5.2 Create and write Streaming DataFrames and Streaming Datasets, including the basic output modes and output sinksreadStream, writeStream, output modes and sinks.
- What it means
- This objective covers building and starting streaming queries. spark.readStream.format(...).load() creates a streaming DataFrame from sources such as Kafka, files or Delta tables. writeStream sets the output mode, sink format and checkpointLocation, and start() or toTable() launches the query, returning a StreamingQuery handle.
- What to understand
- Append mode, the default, writes only new rows that will not change. Update writes rows that changed in the latest batch. Complete rewrites the whole result table and requires an aggregation. File sources need a schema supplied up front. Sinks include files, Kafka, Delta tables, foreachBatch, foreach, and console or memory for testing; the file sink supports append only.
- How it shows up
- A query and a sink are described, and you choose a valid output mode or the missing writeStream option.
- Common confusion
- Using complete mode without an aggregation, which Spark rejects.
5.3 Perform basic operations on Streaming DataFrames and Streaming Datasets, such as selection, projection, window and aggregationselect, filter, window functions and streaming aggregations.
- What it means
- This objective covers transforming streams with familiar DataFrame operations: select and filter for projection and selection, groupBy with aggregation, and event-time windows using window(col, duration, slide) for tumbling or sliding windows and session_window for activity-based windows.
- What to understand
- Stateless operations such as select and filter process each row once. Aggregations are stateful, keeping running results in the state store across micro-batches. A watermark declared with withWatermark lets Spark finalise old windows and drop their state, and it is required for append-mode window aggregations. Sorting is only allowed after an aggregation in complete mode.
- How it shows up
- Code shows a window aggregation, and you decide whether it is valid, what windows it creates, or what the watermark does.
- Common confusion
- Treating a sliding window as tumbling; a slide shorter than the duration puts each event in several windows.
5.4 Perform Streaming Deduplication in Structured Streaming, both with and without watermark usagedropDuplicates on streams and bounding state.
- What it means
- This objective covers removing duplicate events from a stream. dropDuplicates(['id']) on a streaming DataFrame keeps the first occurrence of each key across all micro-batches. Adding withWatermark on an event-time column and including that column in the deduplication keys lets Spark discard state for keys older than the watermark.
- What to understand
- Without a watermark, every key seen is stored forever, so state grows without limit. With a watermark, duplicates that arrive later than the threshold may no longer be detected. Spark 3.5 added dropDuplicatesWithinWatermark, which removes duplicates arriving within the watermark delay even when their event times differ slightly.
- How it shows up
- A deduplication query is shown, and you identify unbounded state or choose the version that limits it.
- Common confusion
- Leaving the event-time column out of the keys and expecting the watermark to bound state.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Unbounded table model
A stream is treated as a table that keeps receiving rows.
- Remember
- Queries are written like batch queries.
- Tested as
- Programming model questions.
- Trap
- Thinking streaming needs a separate API.
-
Micro-batch
Default execution unit that processes newly arrived data as a small batch.
- Remember
- Each batch is checkpointed.
- Tested as
- Engine questions.
- Trap
- Confusing it with continuous processing.
-
checkpointLocation
Storage path for offsets, commits and state.
- Remember
- Required for recovery and exactly-once.
- Tested as
- Write configuration items.
- Trap
- Sharing one checkpoint between queries.
-
Exactly-once
Each record affects the result once despite failures.
- Remember
- Needs replayable source and idempotent sink.
- Tested as
- Guarantee questions.
- Trap
- Crediting Spark alone.
-
Append mode
Emits only new final rows.
- Remember
- Default; aggregations need a watermark.
- Tested as
- Output mode questions.
- Trap
- Using it for un-watermarked aggregations.
-
Update mode
Emits rows changed in the latest trigger.
- Remember
- Good for running aggregates.
- Tested as
- Output mode questions.
- Trap
- Expecting a full table each time.
-
Complete mode
Rewrites the full result table every trigger.
- Remember
- Requires an aggregation.
- Tested as
- Output mode questions.
- Trap
- Using it for raw event passthrough.
-
Tumbling vs sliding window
Tumbling windows do not overlap; sliding windows overlap when the slide is shorter.
- Remember
- window(col, '10 minutes', '5 minutes') slides.
- Tested as
- Window questions.
- Trap
- Counting each event in one window only.
-
Watermark
Threshold for how late data may arrive before state is dropped.
- Remember
- Declared with withWatermark on an event-time column.
- Tested as
- Aggregation and deduplication items.
- Trap
- Applying it after the aggregation.
-
Streaming dropDuplicates
Keeps first occurrence of each key across batches.
- Remember
- Unbounded state without a watermark.
- Tested as
- Deduplication questions.
- Trap
- Assuming state expires on its own.
-
availableNow trigger
Processes all available data in batches, then stops.
- Remember
- Replaces the deprecated once trigger.
- Tested as
- Trigger questions.
- Trap
- Expecting it to run continuously.
-
foreachBatch
Runs batch DataFrame code on each micro-batch output.
- Remember
- Allows writes to sinks without streaming support.
- Tested as
- Sink questions.
- Trap
- Assuming exactly-once without idempotent logic.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Output modes
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Append | Only new rows that will not change | Default; needs watermark for aggregates | Using it with un-watermarked windows |
| Update | Rows that changed this trigger | Partial result per batch | Expecting all rows every time |
| Complete | Entire result table every trigger | Requires aggregation | Using it without aggregation |
Deduplication with vs without watermark
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Without watermark | dropDuplicates on key only | State kept forever | Ignoring memory growth |
| With watermark | withWatermark plus key and event time | Old state dropped | Omitting the event-time column from keys |
| dropDuplicatesWithinWatermark | Duplicates removed within the delay | Event times may differ | Expecting it before Spark 3.5 |
Triggers
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Default | Next micro-batch starts when the previous ends | Lowest latency in micro-batch mode | Thinking a trigger must be set |
| processingTime | Micro-batch at a fixed interval | Predictable cadence | Confusing it with window duration |
| availableNow | Process backlog then stop | Batch-like scheduling | Expecting it to keep running |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Complete for everything
- What candidates pick
- Pick complete mode for a simple filter-and-write query.
- Why it looks right
- Complete sounds the safest.
- Why it's wrong
- Complete mode requires an aggregation because it rewrites a result table; plain row streams use append.
Clue to watch for The query has no groupBy.
-
Spark alone is exactly-once
- What candidates pick
- Claim checkpoints alone guarantee exactly-once to any sink.
- Why it looks right
- Checkpoints are central to recovery.
- Why it's wrong
- Exactly-once also needs a replayable source and an idempotent or transactional sink; a plain foreach sink can repeat output.
Clue to watch for Options mention the sink type.
-
Watermark after groupBy
- What candidates pick
- Call withWatermark on the aggregated result.
- Why it looks right
- It looks like a final setting.
- Why it's wrong
- The watermark must be defined on the event-time column before the aggregation or deduplication that uses it.
Clue to watch for Code order of withWatermark and groupBy.
-
State expires by itself
- What candidates pick
- Assume dropDuplicates forgets old keys automatically.
- Why it looks right
- Batch jobs release memory when done.
- Why it's wrong
- Without a watermark the stream keeps every key in the state store indefinitely.
Clue to watch for The query runs continuously with no withWatermark call.
-
Window equals trigger
- What candidates pick
- Treat a 10-minute window as running every 10 minutes.
- Why it looks right
- Both use time durations.
- Why it's wrong
- Windows group by event time in the data; triggers control when micro-batches run.
Clue to watch for Both a window duration and a processingTime trigger appear.
-
Sort a stream
- What candidates pick
- Add orderBy to a non-aggregated stream.
- Why it looks right
- Sorting works in batch.
- Why it's wrong
- Sorting is only supported after aggregation in complete mode, so Spark rejects the query.
Clue to watch for orderBy appears without aggregation or complete mode.
Answer framework
Use these steps to evaluate any Structured Streaming snippet before picking an answer.
- Identify the source first, and confirm that a schema is supplied for file sources and that the source can replay data after a failure.
- Determine whether the query is stateless or stateful by looking for aggregations, windows, joins or deduplication.
- For stateful queries, check that withWatermark is set on the event-time column before the stateful operation when state must be bounded.
- Match the output mode to the query: append for final rows, update for changed rows, and complete only when the query has an aggregation.
- Confirm the sink, trigger and checkpointLocation, and judge whether the sink is idempotent enough for the stated delivery guarantee.
Practice question
Want more like this? The Spark Dev Assoc practice test has questions for every domain.
Checklist
0 of 11
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Use the rate source or a folder of small JSON files in a notebook to run real streams. Start with a stateless select and filter written to the console or memory sink, then add a groupBy count and switch between append, update and complete to see which combinations Spark accepts and what each emits.
Next, add event-time windows. Write a tumbling window and a sliding window, then add withWatermark and observe how append mode starts to emit results once windows close. Repeat with dropDuplicates on a key, then on key plus event time with a watermark, and check the state metrics in the query progress output.
Finally, learn the theory points in a short list: checkpoint contents, exactly-once requirements, trigger types and unsupported operations. Practice questions in this section are short, so a few rounds of timed snippets usually lock in the rules. Revisit Domain 3 if aggregation functions themselves feel uncertain.
FAQ
How much of the exam is Structured Streaming?
Structured Streaming is 10% of the scored exam, around four or five questions, from four official objectives.
What does the checkpoint location store?
Source offsets, commit logs and operator state for each micro-batch, so a restarted query can resume where it stopped without losing or reprocessing committed data.
Which output mode is the default?
Append. It writes only new rows that will not change, so aggregations in append mode need a watermark.
Why does my streaming aggregation fail in append mode?
Without a watermark Spark cannot know when a group is final, so it cannot emit results in append mode. Add withWatermark or use update or complete mode.
Does streaming deduplication need a watermark?
Not to work, but without one Spark stores every key forever. A watermark with the event-time column in the keys bounds that state.
Is continuous processing on the exam?
The objectives emphasise micro-batch processing. Continuous mode exists as an experimental low-latency option but is not the default engine behaviour.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 5 knowledge
Check your grasp of output modes, watermarks, checkpoints and streaming deduplication with timed questions modelled on the exam style.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Spark Dev Assoc AI tutor Get any concept explained your way