Overview
This section has four objectives that sit at the end of the delivery cycle. Diagnosis means knowing which evidence answers which question: the Spark UI for stages, tasks, skew and spill on classic compute, driver and executor logs for exceptions and out-of-memory errors, system tables for history across runs, and query profiles for SQL and serverless work. Remediation means using repair runs to rerun only failed and dependent tasks, with parameter overrides when an input was wrong, rather than rerunning a whole job.
Deployment centres on Declarative Automation Bundles, formerly Databricks Asset Bundles: validate, deploy and run a project to dev, staging and prod targets from the CLI or a CI pipeline. The last objective connects Databricks Git folders, formerly Repos, to Git providers so code moves through branches, pull requests and automated deployment. The previous outline listed five objectives here; the current one has four.
How it's tested
Items describe a failed or slow run with a symptom, or a deployment process with a weakness, and ask for the next step. Correct answers favour targeted evidence and minimal reruns, and automated, versioned deployment over manual changes in the production workspace.
-
Find the evidence
A symptom such as one slow task or an out-of-memory error is given, and you pick the UI, log or table to check.
-
Recover the run
Part of a multi-task job failed, and you choose a repair run, parameter override or full rerun.
-
Ship the change
Code must move from development to production, and you choose bundle targets, Git folders and CI steps.
A nightly job with twelve tasks failed at task nine because a date parameter was wrong. Rerunning the whole job repeats eight successful tasks and their writes. A repair run with a parameter override reruns task nine and its dependants with the corrected date, keeping earlier results. The repair run is the right answer.
Objectives 4
Official objectives from the Databricks outline, in outline order.
8.1 Identify pertinent diagnostic information using Spark UI, cluster logs, system tables and query profiles to troubleshoot errorsChoosing the right evidence for a failure or slowdown.
- What it means
- Each diagnostic tool answers different questions. The Spark UI shows jobs, stages, tasks, storage and SQL plans for classic compute. Driver and executor logs hold stack traces, print output and memory errors. Cluster event logs record starts, resizes and terminations. System tables give history across runs, and query profiles explain SQL and serverless query execution.
- What to understand
- Skew appears in the Spark UI as a few tasks taking far longer than the median in one stage. Spill and shuffle sizes indicate memory pressure. An executor lost or out-of-memory error points to partition size or memory settings, while a driver error often comes from collecting too much data. Serverless compute does not expose the Spark UI, so use query profiles and run output there.
- How it shows up
- Expect a symptom and a choice of where to look or what it indicates.
- Common confusion
- Looking for a Spark UI on serverless compute, where query profiles serve that role instead.
8.2 Analyze the errors and remediate the failed job runs with job repairs and parameter overridesFixing failed runs without starting over.
- What it means
- A repair run reruns only the failed, canceled or skipped tasks of a job run and the tasks that depend on them, reusing results of tasks that succeeded. You can override job parameters for the repair, which helps when a run failed due to a wrong input value.
- What to understand
- Read the failed task's error and logs first to separate code bugs, data problems and transient infrastructure errors. Transient failures suit task retries; code bugs need a fix before repair. Repairs appear in the run history alongside the original attempt. Tasks should be idempotent so a rerun does not duplicate data.
- How it shows up
- A partly failed run is described, and you choose repair, parameter override, retry policy or full rerun.
- Common confusion
- Rerunning the whole job when only one task failed and earlier tasks wrote data.
8.3 Deploy Databricks resources using Declarative Automation Bundles (formerly Databricks Asset Bundles)Versioned, repeatable deployment of jobs and pipelines.
- What it means
- Declarative Automation Bundles describe jobs, pipelines and other resources as code in a databricks.yml project. The CLI commands bundle validate, bundle deploy and bundle run check, deploy and start them in a chosen target, such as dev or prod.
- What to understand
- Targets set the workspace host, root path, run-as identity, mode and variable overrides. Development mode prefixes resource names with the user and pauses schedules; production mode checks for issues such as personal identities. Deploying the same bundle to each target keeps environments consistent, and destroy removes deployed resources. CI systems run these commands with service principal credentials.
- How it shows up
- Expect a deployment requirement and a choice of bundle command, target setting or identity.
- Common confusion
- Deploying production with a personal identity instead of a service principal set in the target.
8.4 Configure Git-based CI/CD workflows using Databricks Git folders (formerly Repos) to deploy notebooks and codeSource control and automated promotion of code.
- What it means
- Databricks Git folders connect a workspace folder to a remote Git repository so developers can clone, branch, commit, pull and push from Databricks. CI/CD tools such as GitHub Actions or Azure DevOps then test and deploy merged code automatically.
- What to understand
- Developers work on feature branches in their own Git folders, open pull requests, and let CI run tests and a bundle deploy to staging and production. A production Git folder can also be updated to a branch or tag through the API, but bundles give fuller resource deployment. Avoid editing production code directly in the workspace. Authenticate CI with workload identity federation or OAuth for a service principal.
- How it shows up
- A team process is described, and you choose the step that makes deployment controlled and repeatable.
- Common confusion
- Letting developers edit the production folder directly instead of deploying merged commits through CI.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Task skew in Spark UI
A few tasks in a stage run much longer than the rest.
- Remember
- Check task duration distribution in the stage view.
- Tested as
- Slow stage questions.
- Trap
- Adding nodes, which does not fix skew.
-
Driver vs. executor errors
Driver failures often come from collect or broadcast; executor failures from partition size.
- Remember
- Read the stack trace location.
- Tested as
- OOM diagnosis.
- Trap
- Raising executor memory for a driver problem.
-
Cluster event log
Records starts, resizes, terminations and failures of compute.
- Remember
- Explains lost nodes and spot reclaim.
- Tested as
- Infrastructure failure questions.
- Trap
- Looking for it in the Spark UI.
-
Repair run
Reruns failed tasks and dependants only.
- Remember
- Keeps successful task results.
- Tested as
- Partial failure scenarios.
- Trap
- Rerunning the full job.
-
Parameter override on repair
Change job parameter values for the repair.
- Remember
- Fixes runs started with wrong inputs.
- Tested as
- Wrong-input failures.
- Trap
- Editing the job definition instead.
-
Task retries
Automatic reattempts for transient failures.
- Remember
- Not a fix for code bugs.
- Tested as
- Reliability settings.
- Trap
- Using retries to mask bad code.
-
bundle validate / deploy / run
CLI commands to check, deploy and start bundle resources.
- Remember
- Run with -t to pick a target.
- Tested as
- Deployment questions.
- Trap
- Skipping validate in CI.
-
Development vs. production mode
Dev mode prefixes names and pauses schedules; production mode enforces safeguards.
- Remember
- Set per target.
- Tested as
- Environment questions.
- Trap
- Deploying prod in development mode.
-
Databricks Git folders
Workspace folders linked to remote Git repositories.
- Remember
- Formerly called Repos.
- Tested as
- CI/CD questions.
- Trap
- Editing production code in place.
-
Service principal for CI
Non-human identity for deployments and runs.
- Remember
- Use OAuth or workload identity federation.
- Tested as
- Security of pipelines.
- Trap
- Personal access tokens in CI secrets.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Diagnostic sources
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Spark UI | Stages, tasks, shuffle and spill on classic compute | Best for skew and memory issues | Expected on serverless |
| Driver and executor logs | Exceptions and print output | Find the actual error message | Ignored in favour of guessing |
| System tables | History across runs and workspaces | Trends and recurring failures | Used for a live single failure |
| Query profile | Operator metrics for SQL and serverless | Pruning, joins and spill per operator | Confused with the Spark UI |
Repair run vs. retry vs. full rerun
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Repair run | Rerun failed tasks and dependants | Keeps successful results; allows overrides | Run before fixing the code bug |
| Task retry | Automatic reattempt in the same run | For transient errors | Hides permanent failures |
| Full rerun | Start the job from the beginning | Repeats successful tasks | Duplicates non-idempotent writes |
Deployment approaches
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Declarative Automation Bundles | Resources and code as a versioned project | Same definition across targets | Personal identity in prod target |
| Git folder update via API | Moves a workspace folder to a branch or tag | Updates code, not job definitions | Assumed to deploy jobs too |
| Manual UI edits | Changing jobs directly in production | Fast but untracked | Seen as acceptable for hotfixes |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Full rerun
- What candidates pick
- Click run now after one task fails.
- Why it looks right
- It is the quickest action.
- Why it's wrong
- It repeats successful tasks and can duplicate writes; a repair run reruns only what failed.
Clue to watch for Several upstream tasks already succeeded.
-
Scale to fix skew
- What candidates pick
- Add workers because one stage is slow.
- Why it looks right
- More compute often speeds jobs.
- Why it's wrong
- Skewed partitions run on single tasks regardless of cluster size; fix the key distribution or enable skew handling.
Clue to watch for One or two tasks take far longer than others.
-
Retries for bugs
- What candidates pick
- Raise retry count after a schema mismatch error.
- Why it looks right
- Retries improve reliability.
- Why it's wrong
- A deterministic error fails on every attempt; the code or data must change.
Clue to watch for The error message is the same on each attempt.
-
Spark UI on serverless
- What candidates pick
- Open the Spark UI to debug a serverless job.
- Why it looks right
- It is the classic debugging tool.
- Why it's wrong
- Serverless does not expose the Spark UI; use query profiles, run output and system tables.
Clue to watch for The job runs on serverless compute.
-
Git folder deploys jobs
- What candidates pick
- Update the prod Git folder and expect job settings to change.
- Why it looks right
- Code and jobs feel linked.
- Why it's wrong
- Git folders update files only; job and pipeline definitions deploy through bundles or the API.
Clue to watch for The change affects schedules, clusters or parameters.
-
Personal identity in CI
- What candidates pick
- Store a developer's token in the CI system.
- Why it looks right
- It works for a quick setup.
- Why it's wrong
- Deployments should use a service principal so access is auditable and survives staff changes.
Clue to watch for The scenario mentions production or compliance.
Answer framework
Use these steps when a Section 8 item asks how to diagnose, recover or deploy.
- Locate the symptom: a failed task, a slow stage, a memory error, or a deployment that drifted from source control.
- Choose evidence by compute type: Spark UI and logs on classic compute, query profiles and run output on serverless, and system tables for patterns across runs.
- Classify the cause as transient, data, configuration or code, since only transient errors are solved by retries.
- Recover with a repair run, adding parameter overrides when inputs were wrong, after fixing code if needed, and keep tasks idempotent.
- Ship changes by merging in Git, then validating and deploying the bundle to each target from CI under a service principal.
Practice question
Want more like this? The Databricks DE Pro practice test has questions for every domain.
Checklist
0 of 10
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Divide this section into diagnosis and delivery. For diagnosis, run a job on classic compute with a deliberately skewed join and a collect on a large DataFrame. Open the Spark UI, find the slow stage and the outlier tasks, then read the driver log for the collect error. Repeat a similar query on serverless and open its query profile, so the switch between tools is natural.
Next, build a multi-task job whose middle task fails because of a bad parameter. Repair it with a parameter override and look at how the run history shows both attempts. Add a retry policy and note that it does not help with the bad parameter.
For delivery, reuse the bundle from Section 1. Add dev and prod targets, set a service principal as run-as for prod, and deploy both from the CLI. Connect a Git folder to a repository, make a change on a branch, and sketch the CI steps that would run tests and bundle deploy after merge. Keep a list of old and new names, because questions use the current ones.
FAQ
How much of the exam is Debugging and Deploying?
Debugging and Deploying is 10% of the scored content, about six of the 60 scored questions.
What is a repair run?
A rerun of only the failed, canceled or skipped tasks in a job run plus their dependants, optionally with overridden parameters, keeping earlier successful results.
Are Declarative Automation Bundles the same as Databricks Asset Bundles?
Yes. The exam guide uses Declarative Automation Bundles and notes the former name Databricks Asset Bundles. The databricks.yml project format and bundle CLI commands are the tested skills.
Are Databricks Git folders the same as Repos?
Yes. Git folders is the current name for the feature formerly called Repos, which links workspace folders to remote Git repositories.
How do I debug serverless jobs without the Spark UI?
Use the query profile, run output and logs available in the job run, and system tables for history across runs.
Should CI use a personal token?
No. Use a service principal with OAuth or workload identity federation so deployments are auditable and do not depend on one person.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 8 knowledge
You have covered diagnostics, repair runs, bundle deployment and Git folder CI/CD. Practise recovering failed runs and shipping changes under exam timing.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Databricks DE Pro AI tutor Get any concept explained your way