Overview
Section 6 asks what you do after a pipeline is running. Two objectives use the Lakeflow Jobs UI: the run history view, where you compare current durations with past runs to spot regressions, and the job and task views, where statuses, the task graph and failure rates show whether a pipeline is healthy and which upstream task is blocking others.
The third objective moves to the Spark UI, where stage-level metrics reveal data skew, heavy shuffling and disk spill. The fourth covers two table-level features: Liquid Clustering, which organises data by chosen keys and replaces partitioning and Z-ordering for new tables, and predictive optimization, which runs maintenance on Unity Catalog managed tables automatically. The last objective is practical troubleshooting of clusters that fail to start, libraries that conflict and jobs that run out of memory.
How it's tested
Most items describe a symptom, such as a growing run duration, a stalled downstream task, one slow task in a stage or a cluster that never starts, and ask for the cause or the next step. A few ask which table feature reduces maintenance or speeds filtered queries.
-
Read the metric
Stage metrics or run durations are described and you identify skew, spill, shuffle volume or a regression.
-
Find the blocker
A job's task graph shows statuses and you identify the upstream task that caused downstream tasks to be skipped or fail.
-
Pick the table feature
Query or maintenance problems are described and you choose Liquid Clustering, predictive optimization or a manual command.
In the Spark UI, one stage shows a median task time of 8 seconds and a maximum of 14 minutes, with that task reading far more shuffle data than the rest. That pattern is data skew on the join key, not too few executors. Enabling skew handling in adaptive query execution or salting the hot key addresses it, while adding nodes would leave the one large task unchanged.
Objectives 5
Official objectives from the Databricks outline, in outline order.
6.1 Identify trends in job performance using the Lakeflow Jobs run history view to compare current execution times against historical baselinesSpotting slowdowns and anomalies from past runs.
- What it means
- Each Lakeflow job keeps a history of its runs. The run history view, often shown as a matrix of tasks by run, plots durations and outcomes over time, so you can see whether today's run is normal or an outlier compared with the usual baseline.
- What to understand
- Look for a gradual increase in duration, which often follows growing data volume or a table that needs optimization, versus a sudden jump that suggests a code change, a configuration change or a skewed input. Comparing task-level durations shows which task changed. A duration warning threshold on a job or task sends a notification when a run exceeds an expected time. For longer analysis, job run data is also available in system tables under the lakeflow schema.
- How it shows up
- Questions describe a duration pattern and ask what it suggests or which view or setting helps detect it.
- Common confusion
- Judging performance from a single run instead of comparing with the historical baseline.
6.2 Use the Lakeflow Jobs UI to monitor pipeline health by interpreting job statuses, viewing DAG-based task graphs to spot upstream blockers, and tracking pipeline run times and failure ratesReading statuses and task graphs to judge pipeline health.
- What it means
- The Jobs UI shows each run's status and, inside a run, the status of every task on the task graph. Statuses such as Succeeded, Failed, Skipped, Upstream failed, Timed out and Canceled tell you where a run broke and what was affected downstream.
- What to understand
- When a downstream task shows Upstream failed or was skipped, follow the graph back to the first failed task, since that is the root cause. The run list shows run times and success or failure over many runs, which gives a sense of reliability. Task output, logs and error messages are available from each task run. Notifications for start, success, failure and duration warnings can go to email or webhooks. For Lakeflow pipelines, the pipeline UI and event log show update status and data quality metrics.
- How it shows up
- Expect questions where a graph has several failures and you must find the origin.
- Common confusion
- Debugging the first red task you see downstream rather than the upstream task that failed first.
6.3 Identify common performance bottlenecks such as data skew, shuffling, and disk spilling by interpreting stage-level metrics in the Spark UIDiagnosing slow stages from Spark UI metrics.
- What it means
- The Spark UI breaks a job into stages and tasks. Stage pages show summary metrics for task duration, input size, shuffle read and write, and spill, which reveal where time is going.
- What to understand
- Skew shows as a maximum task duration or shuffle read far above the median, so one or a few tasks hold up the stage. Heavy shuffling shows as large shuffle read and write sizes, often from wide operations such as joins, groupBy or distinct. Spill shows as non-zero spill (memory) and spill (disk) values when partitions do not fit in executor memory. Fixes differ: skew needs AQE skew join handling or salting; shuffle volume can drop with broadcast joins or earlier filters; spill improves with more, smaller partitions or more memory per task.
- How it shows up
- Questions present metrics and ask which bottleneck they indicate and the best first fix.
- Common confusion
- Adding more workers to fix skew; one oversized task still runs on a single core.
6.4 Understand the features of Liquid Clustering and predictive optimizationAutomatic data layout and maintenance for Delta tables.
- What it means
- Liquid Clustering arranges a Delta table's data by chosen clustering keys so queries that filter on those keys read fewer files. Predictive optimization lets Databricks decide when to run maintenance operations on Unity Catalog managed tables.
- What to understand
- Enable Liquid Clustering with CLUSTER BY (columns) when creating or altering a table, or CLUSTER BY AUTO to let Databricks choose keys from query patterns. Keys can be changed without rewriting existing data, and clustering is applied incrementally by OPTIMIZE. It replaces Hive-style partitioning and ZORDER for new tables and cannot be combined with them. Predictive optimization runs OPTIMIZE, VACUUM and statistics collection automatically for Unity Catalog managed tables, and can be enabled or inherited at the account, catalog or schema level. It does not apply to external tables.
- How it shows up
- Expect questions on choosing clustering over partitioning and on what predictive optimization does for you.
- Common confusion
- Expecting predictive optimization to maintain external tables.
6.5 Diagnose cluster startup failures, library conflicts, and out-of-memory issuesCommon compute failures and how to fix them.
- What it means
- Some failures have nothing to do with your transformation logic. A cluster may fail to start, a library may clash with another package or the runtime, and a process may run out of memory. Each leaves different clues in the event log, driver logs or error message.
- What to understand
- Startup failures often come from cloud capacity or quota limits, unavailable instance types, network settings or a failing init script; the cluster event log and the init script logs show which. Library conflicts appear as install errors or import errors and are solved by pinning compatible versions, removing duplicates, or using notebook-scoped %pip installs. Driver out-of-memory errors commonly follow collect or toPandas on large data, so keep results distributed or write them out. Executor out-of-memory errors often come from skew or oversized partitions.
- How it shows up
- Scenario items give an error message or symptom and ask for the most likely cause or fix.
- Common confusion
- Raising executor memory to fix a driver out-of-memory error caused by collect.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Run history view
Matrix of past runs and task durations.
- Remember
- Shows trends against the baseline.
- Tested as
- Regression questions.
- Trap
- Judging from one run.
-
Upstream failed
Status of a task whose dependency failed.
- Remember
- The root cause is earlier in the graph.
- Tested as
- Health monitoring items.
- Trap
- Debugging the downstream task.
-
Duration warning
Threshold that notifies when a run is slow.
- Remember
- Set per job or task.
- Tested as
- Proactive monitoring.
- Trap
- Relying only on failure alerts.
-
Skew signal
Max task time or shuffle read far above median.
- Remember
- Fix with AQE skew handling or salting.
- Tested as
- Spark UI items.
- Trap
- Adding workers instead.
-
Spill signal
Non-zero spill memory and spill disk.
- Remember
- Partitions too big for memory.
- Tested as
- Spark UI items.
- Trap
- Confusing spill with shuffle.
-
Liquid Clustering
CLUSTER BY keys for data layout.
- Remember
- Keys can change without a rewrite.
- Tested as
- Layout choice questions.
- Trap
- Combining it with partitioning or ZORDER.
-
CLUSTER BY AUTO
Databricks picks clustering keys.
- Remember
- Based on observed query patterns.
- Tested as
- Low-maintenance layout.
- Trap
- Thinking keys must always be chosen by hand.
-
Predictive optimization
Automatic OPTIMIZE, VACUUM and statistics.
- Remember
- Unity Catalog managed tables only.
- Tested as
- Maintenance questions.
- Trap
- Applying it to external tables.
-
Driver OOM
Driver runs out of memory.
- Remember
- Often caused by collect or toPandas.
- Tested as
- Failure diagnosis.
- Trap
- Increasing executor memory.
-
Cluster event log
Timeline of cluster lifecycle events.
- Remember
- First stop for startup failures.
- Tested as
- Startup troubleshooting.
- Trap
- Searching notebook output instead.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Spark UI bottlenecks
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Data skew | Few tasks process far more data. | Max far above median for one stage. | Scaling out the cluster. |
| Heavy shuffle | Large data movement between stages. | High shuffle read and write totals. | Ignoring broadcast or earlier filtering. |
| Disk spill | Data written to disk when memory fills. | Non-zero spill metrics. | Mistaking it for slow storage reads. |
Layout and maintenance
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Liquid Clustering | Clusters data on chosen or automatic keys. | Flexible keys, incremental. | Mixing with partitions. |
| Partitioning | Splits a table into folders by column values. | Fixed and easy to over-partition. | Choosing it for new tables by habit. |
| Predictive optimization | Automates OPTIMIZE, VACUUM and statistics. | Managed tables only. | Expecting it on external tables. |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
More nodes for skew
- What candidates pick
- Add workers to speed up the slow stage
- Why it looks right
- More compute usually helps.
- Why it's wrong
- Skew leaves one huge task on one core; spreading the hot key with AQE or salting is what helps.
Clue to watch for Max task time far above median.
-
Downstream root cause
- What candidates pick
- Fix the task marked Upstream failed
- Why it looks right
- It is the one that shows an error state.
- Why it's wrong
- That status means a dependency failed first; the root cause is the earliest failed task.
Clue to watch for Upstream failed or skipped statuses.
-
Partition by habit
- What candidates pick
- PARTITIONED BY on a new large table
- Why it looks right
- Partitioning is a familiar technique.
- Why it's wrong
- Databricks recommends Liquid Clustering for new tables because keys can change without rewrites.
Clue to watch for New table with evolving query patterns.
-
Predictive optimization everywhere
- What candidates pick
- Enable it for an external table
- Why it looks right
- It sounds like a general setting.
- Why it's wrong
- Predictive optimization only maintains Unity Catalog managed tables.
Clue to watch for External LOCATION in the table definition.
-
Executor memory for driver errors
- What candidates pick
- Increase executor memory
- Why it looks right
- Memory errors seem to need more memory anywhere.
- Why it's wrong
- A driver out-of-memory error after collect needs code that keeps data distributed, or more driver memory.
Clue to watch for collect or toPandas in the code.
Answer framework
Use this sequence to diagnose symptoms in Section 6 questions.
- Classify the symptom as a duration trend, a failed run, a slow stage, a table layout issue or a compute failure.
- For trends and failures, compare against run history and follow the task graph back to the earliest failed task.
- For slow stages, read median versus maximum task time, shuffle totals and spill values to separate skew, shuffle and memory pressure.
- For table layout, prefer Liquid Clustering for new tables and predictive optimization for maintenance of managed tables.
- For compute failures, check the cluster event log, init script logs, library versions and whether memory errors come from the driver or executors.
Practice question
Want more like this? The Databricks DE Assoc practice test has questions for every domain.
Checklist
0 of 11
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Budget six to eight hours, split between monitoring and diagnosis. Run a small job daily for a few days, or reuse an existing one, and study the run history view and task graph. Make one task fail on purpose and observe how downstream tasks report Upstream failed. Set a duration warning and confirm a notification arrives.
For the Spark UI, create a skewed dataset where one key holds most rows, join it to another table, and look at the stage summary: compare median and maximum task time and shuffle read. Then run an aggregation on a small cluster with a large shuffle to produce spill, and find the spill columns. Try the fixes from the objective and measure each change.
Create a table with CLUSTER BY, change the keys with ALTER TABLE, and run OPTIMIZE. Check whether predictive optimization is enabled for your catalog. Finally, read the cluster event log on a cluster that fails due to a bad init script or library pin. Questions here describe symptoms, so practise naming the cause before choosing a fix.
FAQ
How much weight does Troubleshooting, Monitoring, and Optimization carry?
Troubleshooting, Monitoring, and Optimization is Section 6 and carries 10% of the exam, about 5 of the 45 scored questions, across five official objectives.
Do I need to read the full Spark UI?
Focus on stage pages and their summary metrics: task duration percentiles, shuffle read and write, and spill. Those cover the skew, shuffle and spill bottlenecks named in the guide.
Does Liquid Clustering replace ZORDER?
For tables that use it, yes. Liquid Clustering replaces both partitioning and ZORDER, and it cannot be combined with them on the same table.
Is predictive optimization on by default?
Databricks enables it by default for many newer accounts, and it can be enabled or disabled at account, catalog and schema level. It only applies to Unity Catalog managed tables.
Where do I look when a cluster will not start?
Start with the cluster event log, which records lifecycle events and errors such as cloud capacity limits. If an init script is involved, check its logs as well.
Are system tables part of this section?
The guide names the Lakeflow Jobs UI and run history. System tables for jobs are useful background for long-term analysis but are not named in the objectives.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 6 knowledge
Practise turning Spark UI metrics, run history and job statuses into a diagnosis and a fix under exam time pressure.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Databricks DE Assoc AI tutor Get any concept explained your way