Overview
Domain 4 carries 10% of the exam and has three official objectives. The first covers tuning strategies and cluster utilisation: partitioning, repartitioning and coalescing, spotting data skew and cutting down shuffles. The second covers Adaptive Query Execution, the runtime re-optimisation feature enabled by default in current Spark. The third covers logging and monitoring, including publishing and customising logs and analysing driver and executor logs to diagnose out-of-memory failures and underused clusters.
Questions are scenario based. You might see a stage where one task runs far longer than the rest, an application that fails after a collect, or a cluster where most cores sit idle, and you must choose the most effective fix. Recognising symptoms in the Spark UI and logs is as important as knowing configuration names.
How it's tested
Questions describe symptoms and ask for a cause or remedy. Look for clues such as one slow task, many tiny tasks, heavy spill, high GC time, an error in the driver log versus an executor log, or idle cores. Then match the symptom to partitioning, skew handling, AQE, broadcasting or memory settings.
-
Diagnose the symptom
A Spark UI observation or log message is described, and you identify skew, too few partitions, spill or out-of-memory as the cause.
-
Pick the fix
Several tuning actions are offered, and you choose the one that addresses the root cause with the least added cost.
-
Explain AQE
You identify which runtime optimisation AQE applies, such as coalescing partitions, switching join strategy or splitting skewed partitions.
A join stage shows 199 tasks finishing in seconds and one task running for 40 minutes with large shuffle read. That pattern points to skew on the join key rather than a lack of memory. With AQE enabled, the skew join optimisation can split the oversized partition; otherwise salting the key spreads the rows. Adding executors would not help because one task still holds most of the data.
Objectives 3
Official objectives from the Databricks outline, in outline order.
4.1 Implement performance tuning strategies & optimize cluster utilization, including partitioning, repartitioning, coalescing, identifying data skew, and reducing shufflingPartition sizing, skew detection and shuffle reduction.
- What it means
- This objective covers practical tuning: choosing partition counts that keep every core busy without creating tiny tasks, using repartition to rebalance or coalesce to cut output files, recognising skew in task durations and shuffle sizes, and reducing shuffles through broadcast joins, early filtering and column pruning.
- What to understand
- Too few partitions leaves cores idle; too many adds scheduling overhead and tiny files. Skew appears as a few tasks with much larger input than the median in the stage detail view. Remedies include AQE skew join handling, salting hot keys, or broadcasting the small side. Filtering and selecting columns before wide operations shrinks the data that must be shuffled.
- How it shows up
- A scenario lists stage metrics, and you choose the change that balances work or removes an unnecessary shuffle.
- Common confusion
- Adding executors to fix skew; one oversized partition still runs as one task.
4.2 Describe Adaptive Query Execution (AQE) and its benefitsRuntime re-optimisation using shuffle statistics.
- What it means
- This objective covers Adaptive Query Execution, which re-plans a query at stage boundaries using statistics collected during execution. It is controlled by spark.sql.adaptive.enabled and has been on by default since Spark 3.2. Its main features are coalescing small shuffle partitions, converting sort-merge joins to broadcast joins, and splitting skewed join partitions.
- What to understand
- Static planning relies on estimates that can be wrong after filters or joins. AQE uses real shuffle sizes to merge many small partitions into fewer reasonable ones, so the default of 200 shuffle partitions matters less. It can switch to a broadcast join when one side turns out to be small, and spark.sql.adaptive.skewJoin.enabled lets it split oversized partitions.
- How it shows up
- Expect a list of optimisations where you identify which ones AQE performs automatically at runtime.
- Common confusion
- Thinking AQE changes plans before execution starts; it adapts between stages using runtime statistics.
4.3 Perform logging and monitoring of Spark applications - publish, customize, and analyze Driver logs and Executor logs to diagnose out-of-memory errors, cluster underutilization, etc.Log levels, driver vs executor logs, OOM and utilisation diagnosis.
- What it means
- This objective covers using logs and the Spark UI to understand a running or failed application. You set verbosity with spark.sparkContext.setLogLevel or a log4j2 configuration, write your own messages from the driver, and read driver logs for planning and collect failures and executor logs for task-level errors.
- What to understand
- A driver out-of-memory error usually follows collect, toPandas or a large broadcast; spark.driver.maxResultSize limits results returned. Executor OOM usually follows skewed or oversized partitions, so more partitions or memory help. Underutilisation shows as idle executors in the Executors tab or fewer tasks than cores. Spark 4.0 adds optional structured JSON logging.
- How it shows up
- A failure message and its location are described, and you decide whether the driver or executors ran out of memory and why.
- Common confusion
- Raising executor memory to fix an error caused by collecting results to the driver.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Data skew
Uneven distribution where a few keys hold most rows.
- Remember
- Shows as a few very slow tasks in one stage.
- Tested as
- Diagnosis scenarios.
- Trap
- Treating it as a memory shortage.
-
Salting
Adding a random suffix to hot keys to spread them over more partitions.
- Remember
- Requires matching changes on the other join side.
- Tested as
- Manual skew fixes.
- Trap
- Salting only one side of the join.
-
AQE
Adaptive Query Execution re-optimises plans at runtime.
- Remember
- On by default since Spark 3.2.
- Tested as
- AQE benefit questions.
- Trap
- Thinking it must be enabled manually in current versions.
-
Coalescing shuffle partitions
AQE merges small post-shuffle partitions.
- Remember
- Reduces tiny tasks after the default 200 partitions.
- Tested as
- AQE questions.
- Trap
- Confusing it with the coalesce() method call.
-
Dynamic join switching
AQE converts sort-merge to broadcast when a side is small at runtime.
- Remember
- Uses actual sizes, not estimates.
- Tested as
- Join strategy items.
- Trap
- Expecting static planning to catch every case.
-
Skew join optimisation
AQE splits oversized partitions in a sort-merge join.
- Remember
- Controlled by spark.sql.adaptive.skewJoin.enabled.
- Tested as
- Skew questions.
- Trap
- Assuming it fixes skew in every operator.
-
Spill
Data written to disk when execution memory runs out.
- Remember
- Visible as spill metrics in the stage view.
- Tested as
- Memory symptoms.
- Trap
- Ignoring it as harmless.
-
Driver OOM
Driver memory exhausted, typically by collect or toPandas.
- Remember
- spark.driver.maxResultSize caps results.
- Tested as
- Log diagnosis questions.
- Trap
- Adding executor memory instead.
-
Executor OOM
Executor memory exhausted by large partitions or skew.
- Remember
- More partitions often help.
- Tested as
- Log diagnosis questions.
- Trap
- Blaming the driver.
-
setLogLevel
Changes Spark log verbosity at runtime.
- Remember
- Levels include ERROR, WARN, INFO and DEBUG.
- Tested as
- Logging questions.
- Trap
- Expecting it to change executor log retention.
-
Cluster underutilisation
Cores sit idle because there are too few tasks or uneven work.
- Remember
- Check task count against total cores.
- Tested as
- Utilisation scenarios.
- Trap
- Adding nodes when partitions are the bottleneck.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Driver OOM vs executor OOM
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Driver OOM | Driver runs out of memory | Caused by collect, toPandas, large broadcasts | Raising executor memory |
| Executor OOM | An executor runs out of memory | Caused by large or skewed partitions | Collecting less data to the driver |
AQE features
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Partition coalescing | Merges small shuffle partitions | Fewer tiny tasks | Confusing it with df.coalesce() |
| Join strategy switch | Sort-merge to broadcast at runtime | Uses real sizes | Thinking a hint is required |
| Skew join handling | Splits oversized join partitions | Targets skewed keys | Expecting it for every aggregation |
Shuffle reduction techniques
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Broadcast join | Copy small side to executors | Avoids shuffling the large side | Broadcasting a big table |
| Filter and prune early | Drop rows and columns before wide operations | Less data moved | Filtering after the join |
| Avoid needless repartition | Remove extra repartition calls | Each adds a full shuffle | Repartitioning before every write |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
More nodes fix skew
- What candidates pick
- Scale out the cluster when one task runs far longer than the rest.
- Why it looks right
- More resources usually speed jobs.
- Why it's wrong
- A skewed partition is processed by one task, so added executors sit idle while it finishes.
Clue to watch for Most tasks finish quickly and one dominates the stage time.
-
Executor memory for driver OOM
- What candidates pick
- Increase spark.executor.memory after a failure during collect().
- Why it looks right
- Memory errors suggest more memory.
- Why it's wrong
- collect brings results to the driver, so the driver is what ran out of memory.
Clue to watch for The error appears in the driver log after collect or toPandas.
-
AQE is static
- What candidates pick
- Describe AQE as a better optimiser that plans everything up front.
- Why it looks right
- Optimisers normally run before execution.
- Why it's wrong
- AQE uses statistics gathered at stage boundaries during execution to re-plan the remaining stages.
Clue to watch for The answer mentions runtime statistics or shuffle sizes.
-
Coalesce fixes skew
- What candidates pick
- Call coalesce to balance skewed partitions.
- Why it looks right
- coalesce changes partitioning.
- Why it's wrong
- coalesce merges partitions without redistributing rows, so a hot key remains in one large partition.
Clue to watch for The goal is even work across tasks.
-
Logs show everything in one place
- What candidates pick
- Search only the driver log for a task failure.
- Why it looks right
- The driver reports job failure.
- Why it's wrong
- Task-level stack traces and memory errors appear in the failing executor's logs, linked from the Executors tab.
Clue to watch for The question asks where the root cause of a task failure is recorded.
Answer framework
When a tuning or troubleshooting scenario appears, follow these steps to separate cause from symptom.
- Locate the failure or slowdown first: decide whether it sits in the driver or the executors, and which stage, using the logs and Spark UI details described.
- Compare task durations and input sizes within the stage, since a few large outliers indicate skew rather than general slowness.
- Check the partition count against available cores to judge whether work is too coarse, too fine or leaving executors idle.
- Consider what AQE already handles, such as coalescing partitions or switching joins, before choosing a manual change.
- Pick the fix that addresses the root cause directly, such as broadcasting, salting, adjusting partitions or reducing data sent to the driver.
Practice question
Want more like this? The Spark Dev Assoc practice test has questions for every domain.
Checklist
0 of 11
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Build intuition by breaking things on purpose. Create a DataFrame where one key has most of the rows, join it to another table and inspect the stage in the Spark UI: note the summary metrics for task duration and shuffle read, and compare the maximum with the median. Then turn AQE skew handling off and on and see the difference.
Next, practise memory diagnosis. Collect a large result to provoke a driver error, then cause executor pressure with a heavy aggregation on few partitions, and read where each error is reported. Learn spark.driver.maxResultSize, spark.sql.shuffle.partitions and spark.sql.adaptive.enabled by name.
Finally, memorise a symptom-to-fix table: skew to AQE skew join or salting, many tiny tasks to fewer partitions or AQE coalescing, a slow join with a small table to a broadcast, and idle cores to more partitions. Use timed questions to rehearse picking the root cause rather than the most generic option.
FAQ
How much of the exam is Troubleshooting and Tuning Apache Spark DataFrame API Applications?
Troubleshooting and Tuning Apache Spark DataFrame API Applications is 10% of the scored exam, about four or five questions, from three official objectives.
Is AQE on by default?
Yes. spark.sql.adaptive.enabled has defaulted to true since Spark 3.2, and Databricks enables it as well.
What does AQE actually change?
It coalesces small shuffle partitions, switches sort-merge joins to broadcast joins when one side is small at runtime, and splits skewed partitions in joins.
How do I spot data skew?
In the stage detail page, compare the maximum task duration and shuffle read with the median. A few tasks far above the median indicate skew.
Where do I find executor logs?
In the Spark UI Executors tab, each executor links to its stdout and stderr logs. On Databricks, cluster log delivery can also publish them to storage.
What causes cluster underutilisation?
Usually too few partitions for the number of cores, skew that leaves most executors waiting, or work done on the driver instead of in distributed tasks.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 4 knowledge
Test whether you can read skew, memory and utilisation symptoms correctly and choose the fix Databricks expects for each one.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Spark Dev Assoc AI tutor Get any concept explained your way