Overview
Section 4 covers Lakeflow Jobs, the orchestrator built into Databricks, formerly known as Databricks Workflows. The first objective is control flow: task retries, If/else condition tasks for branching and For each tasks for looping over a list of inputs. The second is task configuration: notebook, SQL query, dashboard and pipeline tasks, and how dependencies between them form a directed acyclic graph that decides run order.
The last two objectives are about when a job runs. You need to know the trigger types named in the guide, which are scheduled, file arrival and table update, and to choose between running on a clock and running when data actually arrives or changes. Questions blend these ideas, for example a job that should start when files land in a volume, run a pipeline, branch on a row count and retry a flaky API task.
How it's tested
Items usually describe a pipeline with timing or reliability requirements and ask how to configure the job. Expect options that differ in one setting: the trigger type, a Run if condition, a retry policy, or whether to use a For each task rather than duplicated tasks. Reading a described DAG to find run order is also common.
-
Select the trigger
Data arrival or freshness needs are described and you choose a schedule, file arrival or table update trigger.
-
Shape the DAG
Tasks with dependencies, branches or loops are described and you choose how to wire them with conditions and Run if settings.
-
Handle transient failure
A task fails occasionally for external reasons and you choose retries, timeouts or a repair run rather than a full rerun.
A job must start whenever a partner drops files into a Unity Catalog volume, which happens at unpredictable times. A schedule every five minutes would often run with nothing to process and still add latency. A file arrival trigger on that volume path starts the job only when new files appear, so it is the better choice.
Objectives 4
Official objectives from the Databricks outline, in outline order.
4.1 Implement control flows (retries and conditional tasks such as branching and looping) using Lakeflow Jobs for pipeline orchestrationRetries, If/else branching and For each loops inside a job.
- What it means
- Control flow decides what a job does when things succeed, fail or depend on a value. Lakeflow Jobs provides task-level retries, If/else condition tasks that branch on a comparison, For each tasks that run a nested task once per input, and Run if settings that control when a dependent task runs.
- What to understand
- Retries are set per task with a maximum count and an interval, and can apply on timeout. An If/else task compares two operands, often a job parameter or a task value set upstream with dbutils.jobs.taskValues.set, and dependent tasks attach to its true or false outcome. A For each task takes a JSON array or a reference to one and runs its nested task for each element, with a concurrency setting for parallel runs. Run if options include All succeeded, At least one succeeded, None failed, All done, At least one failed and All failed, which allow cleanup or alert tasks.
- How it shows up
- Expect questions on looping over a list, branching on an upstream value, and running a cleanup task even after failures.
- Common confusion
- Thinking a failed task needs the whole job rerun; a repair run reruns only failed tasks and their dependents.
4.2 Configure common tasks (notebook, SQL query, dashboard, and pipeline tasks) and their dependencies using Lakeflow Jobs and its DAG-based task graphTask types and how dependencies decide run order.
- What it means
- A job is a set of tasks connected by dependencies. Each task has a type that says what it runs: a notebook, a SQL query, a dashboard refresh, a Lakeflow pipeline update and others. Dependencies make a directed acyclic graph, so a task starts only when the tasks it depends on finish under its Run if rule.
- What to understand
- Notebook tasks take a path from the workspace or a Git provider plus parameters. SQL tasks run a query, a file, an alert or a dashboard update on a SQL warehouse. Pipeline tasks trigger an update of an existing Lakeflow pipeline. Tasks without dependencies run in parallel. Each task can use different compute: serverless, a shared job cluster, or a SQL warehouse for SQL tasks. Job parameters pass to all tasks, and dynamic value references such as {{job.start_time.iso_date}} insert run context. Task values let one task pass small results to later tasks.
- How it shows up
- Questions ask which task type fits a step, how to pass values, or which tasks run in parallel.
- Common confusion
- Running a SQL task on a jobs cluster; SQL tasks require a SQL warehouse.
4.3 Implement job schedules using Lakeflow Jobs with an understanding of trigger types (scheduled, file arrival, and table update)Setting up scheduled, file arrival and table update triggers.
- What it means
- A trigger decides when a job run starts. Scheduled triggers use a time pattern. File arrival triggers watch a storage location. Table update triggers watch one or more tables. Jobs can also run manually, through the API, or continuously.
- What to understand
- Schedules use a simple interval or Quartz cron syntax with a time zone and can be paused. File arrival triggers monitor a Unity Catalog external location or volume path and start a run when new files appear, with settings for minimum time between triggers and a wait after the last change so a batch of files lands first. Table update triggers watch Unity Catalog tables and start the job when any or all of them are updated, which suits jobs downstream of other pipelines. Only one trigger type is active on a job at a time.
- How it shows up
- Expect questions matching a data-arrival story to the right trigger and its settings.
- Common confusion
- Pointing a file arrival trigger at a path not governed by Unity Catalog.
4.4 Choose between time-based and data-driven triggers based on data availability and pipeline dependenciesDeciding whether a job runs on a clock or on data events.
- What it means
- Time-based triggers suit work tied to business calendars, such as a report due at 7 a.m. Data-driven triggers suit work that should run as soon as inputs are ready, whenever that is. The choice depends on how predictable data arrival is and on what downstream users expect.
- What to understand
- If upstream data arrives at irregular times, a schedule either runs too early and finds nothing or runs too late and adds delay, so a file arrival or table update trigger is better. If several upstream tables must all be fresh, a table update trigger set to wait for all of them avoids partial results. If the business needs output at a fixed time regardless of arrival, a schedule with a check task or Run if conditions fits. Data-driven triggers also avoid wasted compute from empty runs.
- How it shows up
- Scenario items describe arrival patterns and freshness needs and ask which trigger approach fits.
- Common confusion
- Shortening a cron interval to approximate an event-driven trigger.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
For each task
Runs a nested task once per input element.
- Remember
- Has a concurrency setting.
- Tested as
- Looping questions.
- Trap
- Copying the same task many times.
-
If/else condition task
Branches on a comparison of two values.
- Remember
- Often compares task values or parameters.
- Tested as
- Branching questions.
- Trap
- Writing branch logic inside one notebook.
-
Run if
Rule for when a dependent task runs.
- Remember
- All done runs regardless of outcome.
- Tested as
- Cleanup and alert tasks.
- Trap
- Assuming downstream tasks never run after a failure.
-
Task values
Small values passed between tasks.
- Remember
- Set with dbutils.jobs.taskValues.set.
- Tested as
- Passing results downstream.
- Trap
- Using temp files for small values.
-
Repair run
Reruns failed tasks and dependents only.
- Remember
- Keeps successful task results.
- Tested as
- Recovery questions.
- Trap
- Rerunning the full job.
-
File arrival trigger
Starts a job when files land in a UC location or volume.
- Remember
- Has wait and minimum interval settings.
- Tested as
- Irregular file drops.
- Trap
- Polling with a frequent schedule.
-
Table update trigger
Starts a job when watched UC tables change.
- Remember
- Can wait for any or all tables.
- Tested as
- Downstream of other pipelines.
- Trap
- Scheduling after an estimated upstream finish time.
-
SQL task compute
Runs on a SQL warehouse.
- Remember
- Covers queries, files, alerts and dashboards.
- Tested as
- Task configuration items.
- Trap
- Assigning a jobs cluster.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Trigger types
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Scheduled | Runs on an interval or cron pattern. | Predictable times, ignores data arrival. | Running before upstream data is ready. |
| File arrival | Runs when new files appear in a location. | Event-driven for file sources. | Watching a path outside Unity Catalog. |
| Table update | Runs when watched tables are updated. | Event-driven for table dependencies. | Forgetting to choose any versus all tables. |
| Continuous | Starts a new run as soon as the last ends. | Always-on processing. | Using it for daily batch work. |
Control flow features
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Retries | Rerun a failed task automatically. | Handles transient failures. | Retrying deterministic code bugs. |
| If/else task | Branch on a condition. | Chooses between paths. | Expecting it to loop. |
| For each task | Loop over a list of inputs. | Runs a nested task per element. | Using If/else to simulate loops. |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Faster cron for events
- What candidates pick
- A schedule every minute to catch new files
- Why it looks right
- It reacts quickly to arrivals.
- Why it's wrong
- Frequent schedules waste compute on empty runs; a file arrival trigger starts runs only when files appear.
Clue to watch for Unpredictable file drops.
-
Loop by duplication
- What candidates pick
- One task per input value
- Why it looks right
- It is easy to see each task in the graph.
- Why it's wrong
- A For each task loops over inputs with one definition and a concurrency limit, which is easier to maintain.
Clue to watch for A list of inputs that may change.
-
Full rerun after failure
- What candidates pick
- Run the job again from the start
- Why it looks right
- It guarantees a clean state.
- Why it's wrong
- A repair run reruns only failed tasks and their dependents, saving time and compute.
Clue to watch for Most tasks already succeeded.
-
Cleanup skipped on failure
- What candidates pick
- Leave Run if at All succeeded for cleanup
- Why it looks right
- It is the default setting.
- Why it's wrong
- With All succeeded, cleanup never runs after an upstream failure; All done runs it either way.
Clue to watch for Must run regardless of outcome.
-
Wrong compute for SQL task
- What candidates pick
- A jobs cluster for a SQL query task
- Why it looks right
- Jobs clusters run most task types.
- Why it's wrong
- SQL query, alert and dashboard tasks run on a SQL warehouse.
Clue to watch for SQL task type in the scenario.
Answer framework
Use these steps when a question asks how to orchestrate or trigger work with Lakeflow Jobs.
- Decide what should start the job: a fixed business time, new files arriving in a Unity Catalog location, or updates to upstream tables.
- Map each step to a task type, such as notebook, SQL, dashboard or pipeline, and give SQL tasks a SQL warehouse.
- Draw dependencies so independent tasks run in parallel and set Run if rules for cleanup or alert tasks.
- Add control flow where needed: For each for lists of inputs, If/else for branches, and task values to pass results.
- Plan for failure with retries for transient errors, timeouts for hung tasks, and repair runs rather than full reruns.
Practice question
Want more like this? The Databricks DE Assoc practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Allow eight to ten hours, mostly building jobs. Create a job with three tasks: a notebook that sets a task value, an If/else task that checks it, and two branches. Add a For each task that loops over a short list of table names with concurrency set to two. Force one branch to fail and practise a repair run, then add a cleanup task with Run if set to All done and confirm it runs.
Next, explore triggers. Point a file arrival trigger at a volume, upload a file, and watch the run start. Set up a table update trigger on two tables and update them one at a time to see when the job fires. Compare that with a cron schedule. Add a SQL task and a pipeline task so you see how compute is assigned to each.
To finish, write short scenario cards: an arrival pattern on one side, the trigger and settings on the other. Questions in this section are mostly configuration choices, so rehearsing the reason each wrong option fails is the fastest way to improve.
FAQ
How much of the exam covers Working with Lakeflow Jobs?
Working with Lakeflow Jobs is Section 4 and carries 16% of the exam, about 7 of the 45 scored questions. It has four official objectives covering control flow, tasks and triggers.
Is Lakeflow Jobs the same as Databricks Workflows?
Yes. Databricks renamed Workflows jobs to Lakeflow Jobs. Older study material may use the Workflows name for the same orchestrator and features.
Can a job have more than one trigger?
A job has one trigger type at a time, such as a schedule or a file arrival trigger. It can still be started manually or through the API in addition.
What is the difference between a task value and a job parameter?
Job parameters are set when a run starts and passed to all tasks. Task values are set by a task during the run and read by later tasks.
Do I need to know Quartz cron syntax?
Know that schedules can use Quartz cron expressions with a time zone and that simple intervals are also available. Writing complex expressions from memory is unlikely to be tested in depth.
Can a job trigger a Lakeflow pipeline?
Yes. A pipeline task starts an update of an existing pipeline, so ingestion, transformation and downstream notebooks can be orchestrated in one job.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 4 knowledge
Practise configuring triggers, task dependencies, retries and control flow tasks in timed Lakeflow Jobs scenarios like those on the exam.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Databricks DE Assoc AI tutor Get any concept explained your way