Overview
Section 1 is the engineering core of the Professional exam and carries 12 objectives. It starts with code organisation: structuring a Python project so Declarative Automation Bundles (formerly Databricks Asset Bundles) can build and deploy it, and resolving library conflicts on serverless, pipeline and bundle environments. It then covers UDFs in pandas, Python and SQL, including Unity Catalog functions. The pipeline objectives ask you to build streaming pipelines with Lakeflow Pipelines and Auto Loader, pick a streaming table or a materialized view, write AUTO CDC flows for SCD Type 1 and Type 2, and decide between hand-written Structured Streaming and Apache Spark Declarative Pipelines. Orchestration covers Lakeflow Jobs created through the UI, API and CLI, with If/Else and For Each tasks. The section closes with compute selection, unit and integration tests, and stateful streaming semantics.
This page follows the exam version that went live on 9 October 2026, which replaced the earlier ten-section outline.
How it's tested
Expect scenario items with short code fragments in Python or SQL. A question describes a pipeline requirement, such as late data, history tracking or a per-file loop, and offers four plausible implementations. Usually two options would run, and the correct one is the option that meets the stated latency, cost or correctness constraint.
-
Pick the pipeline object
A latency and refresh requirement is given and you choose a streaming table, a materialized view, or a plain Structured Streaming query.
-
Fix the CDC flow
Change records arrive out of order or include deletes, and you choose the AUTO CDC settings such as keys, sequence column and SCD type.
-
Bound the streaming state
A stateful query keeps growing or restarts badly, and you identify the missing watermark, wrong output mode or checkpoint problem.
A team must keep full history of customer address changes from a CDC feed where events can arrive late. A MERGE in foreachBatch would work but needs custom ordering logic. AUTO CDC in Lakeflow Pipelines with KEYS on customer_id, SEQUENCE BY on the source change timestamp and SCD Type 2 handles ordering and history for you, so that is the answer.
Objectives 12
Official objectives from the Databricks outline, in outline order.
1.1 Implement scalable Python project structures for Declarative Automation Bundles that support modular development and CI/CDLaying out bundle projects so code, resources and targets scale.
- What it means
- A bundle is a project folder with a databricks.yml file that declares resources such as jobs and pipelines, the artifacts to build, and deployment targets like dev and prod. A scalable layout keeps reusable Python in a src package, resource definitions in separate YAML or Python files, and tests beside them.
- What to understand
- Use include to split resource files, variables to avoid hard-coded catalogs or paths, and targets to change workspace, run-as identity and settings per environment. Python code is usually built into a wheel through the artifacts block and attached to tasks, so notebooks import shared modules instead of copying them. Development mode prefixes resource names and pauses schedules so engineers do not collide.
- How it shows up
- Questions show a growing repo and ask how to share code across jobs or deploy the same project to several environments.
- Common confusion
- Copying notebooks per environment instead of using one bundle with several targets and variables.
1.2 Apply troubleshooting techniques to dependency conflicts and installation failures for external libraries across serverless, pipeline and bundle environmentsDiagnosing and fixing library installation and version problems.
- What it means
- External libraries can be installed as notebook-scoped packages with %pip, as job task dependencies, in a serverless environment specification, in a pipeline's environment settings, or as bundle artifacts. Each place has its own rules, and failures show up as install errors or as import errors at run time.
- What to understand
- Pin versions in a requirements file so runs are reproducible, and check whether a pinned package clashes with a library already in the runtime image. Serverless compute uses an environment version plus a dependency list rather than cluster libraries or init scripts. Wheels and requirement files can be stored in Unity Catalog volumes or workspace files. Restart the Python process after %pip changes in a notebook.
- How it shows up
- You may see an error message after a runtime upgrade and must pick the change that restores a stable install.
- Common confusion
- Assuming init scripts or cluster libraries apply to serverless compute; serverless relies on environment dependencies instead.
1.3 Develop User-Defined Functions (UDFs) using Pandas, Python and SQL, including Unity Catalog functions, for given constraintsChoosing and writing the right kind of UDF.
- What it means
- UDFs add custom logic where built-in functions fall short. Python UDFs process one row at a time, pandas UDFs process Arrow batches as pandas Series, SQL UDFs wrap SQL expressions, and Unity Catalog functions are registered in a catalog and schema so they are governed and reusable across workspaces.
- What to understand
- Prefer a built-in function first, then a SQL UDF, then a pandas UDF, and use row-at-a-time Python UDFs last because of serialization cost. Pandas UDFs are vectorized and much faster for numeric work. A function created with CREATE FUNCTION in Unity Catalog needs EXECUTE permission and can also serve as a row filter or column mask. Temporary functions disappear when the session ends.
- How it shows up
- Expect a requirement such as reuse by analysts in SQL or high-volume numeric scoring, and you pick the UDF type.
- Common confusion
- Picking a plain Python UDF for heavy numeric work where a pandas UDF or built-in function is far faster.
1.4 Create production-ready data pipelines for streaming data using Lakeflow Pipelines and Auto LoaderIncremental file ingestion inside declarative pipelines.
- What it means
- Lakeflow Pipelines (earlier called Delta Live Tables and Lakeflow Declarative Pipelines) let you declare tables and flows while the platform manages orchestration, retries and checkpoints. Auto Loader, the cloudFiles source, discovers new files in cloud storage incrementally and is the usual source for a bronze streaming table.
- What to understand
- Auto Loader tracks processed files so each file is ingested once, infers and evolves schema using a schema location, and puts unexpected fields into the _rescued_data column. In a pipeline the checkpoint and schema location are managed for you. Production readiness means triggered or continuous mode chosen deliberately, expectations on data quality, and Unity Catalog as the publishing target.
- How it shows up
- Questions ask which source or option handles new files, schema drift or reprocessing in a pipeline.
- Common confusion
- Using a batch read of the whole folder each run instead of Auto Loader's incremental file tracking.
1.5 Create and automate ETL workloads using Lakeflow Jobs via the UI, API and CLIBuilding and scheduling multi-task jobs in every interface.
- What it means
- Lakeflow Jobs (formerly Workflows) orchestrate tasks such as notebooks, Python wheels, SQL, pipelines and dbt into a dependency graph with schedules, file arrival or table update triggers, parameters and notifications. The same job can be defined in the UI, through the Jobs REST API, with the Databricks CLI, or in a bundle.
- What to understand
- Know that tasks share job parameters and can pass small values through task values. Triggers include cron schedules, continuous runs, file arrival and table updates. The CLI and API accept the same JSON job specification the UI produces, so a job can be exported and recreated. Use a service principal as the run-as identity for production jobs.
- How it shows up
- A scenario asks how to automate creation or triggering of a job from an external tool or script.
- Common confusion
- Thinking the UI is the only place to define task dependencies; the API and CLI use the same JSON specification.
1.6 Choose between a streaming table and a materialized view for a given latency, cost and refresh requirementMatching declarative table types to workload needs.
- What it means
- A streaming table processes each new source row once and appends results, which suits append-only ingestion and low-latency incremental loads. A materialized view stores the result of a query and refreshes it, incrementally when possible, so it always reflects the current source including updates and deletes.
- What to understand
- Streaming tables are cheap for high-volume append sources but do not recompute when earlier rows change. Materialized views handle joins and aggregations over changing data and give correct results after source corrections, at the cost of refresh work. Both can be defined in pipelines or standalone in Databricks SQL. Freshness needs and whether sources are append-only usually decide the choice.
- How it shows up
- A table definition and freshness target are given, and you decide which object gives correct results at lowest cost.
- Common confusion
- Using a streaming table over a source that receives updates and expecting old aggregates to correct themselves.
1.7 Implement CDC pipelines using AUTO CDC APIs (formerly APPLY CHANGES) in Lakeflow Pipelines, including SCD Type 1 and Type 2 with stored_as_scd_typeApplying change feeds as current-state or history tables.
- What it means
- AUTO CDC reads a stream of change records and applies inserts, updates and deletes to a target streaming table. You specify keys to identify rows, a sequence column to order changes, optional delete conditions, and stored_as_scd_type to keep only current values (Type 1) or full history (Type 2).
- What to understand
- With SCD Type 2 the target gains __START_AT and __END_AT columns and each change closes the previous version. The sequence column solves out-of-order arrival, which a naive MERGE does not. apply_as_deletes marks delete events, and track_history columns limit which changes create new history rows. AUTO CDC FROM SNAPSHOT handles sources that only provide periodic full snapshots.
- How it shows up
- Questions show a CDC requirement, often with late or duplicate events, and ask for the right keys, sequence and SCD setting.
- Common confusion
- Sequencing by ingestion time instead of the source change timestamp, which reorders updates incorrectly.
1.8 Choose between Spark Structured Streaming and Apache Spark Declarative Pipelines for scalable ETL based on operational constraintsDeciding between hand-coded streams and declarative pipelines.
- What it means
- Structured Streaming gives full control over sources, sinks, triggers and checkpoints in your own code. Apache Spark Declarative Pipelines, the engine behind Lakeflow Pipelines, let you declare datasets and flows while the framework infers dependencies, manages checkpoints, retries and data quality.
- What to understand
- Declarative pipelines suit medallion ETL with many dependent tables, expectations and CDC, and reduce operational code. Hand-written Structured Streaming suits custom sinks, fine-grained trigger and state control, or logic that does not fit the declarative model. Operational constraints such as team skills, need for lineage, and custom foreachBatch writes often decide the question.
- How it shows up
- Expect a list of constraints and a choice between the two approaches.
- Common confusion
- Assuming declarative pipelines cannot stream; they run streaming tables continuously or in triggered mode.
1.9 Create Lakeflow Jobs that use control-flow operators such as If/Else conditions and For Each loopsBranching and looping inside a job graph.
- What it means
- Control-flow tasks let a job make decisions at run time. The If/else condition task compares values such as parameters or task values and routes to true or false branches. The For each task runs a nested task once per item in a list, with a configurable concurrency limit.
- What to understand
- Run if conditions such as All succeeded, At least one failed or All done control whether downstream tasks run after upstream results. For each inputs can come from a static list or a task value set by an earlier task. These features replace duplicated tasks and external orchestrators for simple branching.
- How it shows up
- A scenario describes a repeated task or a conditional step and asks which task type or dependency setting fits.
- Common confusion
- Creating one task per item manually when a For each task with a parameter list does the same work.
1.10 Choose appropriate compute and configuration for environments and dependencies, including serverless options, high-memory notebook tasks and auto-optimization settingsPicking compute types and settings for each workload.
- What it means
- Workloads can run on serverless compute for jobs, notebooks and pipelines, on classic jobs compute, or on SQL warehouses. Serverless offers environment versions, a performance-optimized or standard mode, and a higher memory option for memory-hungry notebook tasks. Classic clusters give control over instance types, Spark configuration and init scripts.
- What to understand
- Use jobs compute or serverless rather than all-purpose clusters for scheduled work because it costs less and isolates runs. Serverless jobs can retry failed tasks automatically, and you may disable that when a task is not safe to rerun. Choose classic compute when you need GPUs, custom images or settings serverless does not expose.
- How it shows up
- Questions give cost, start-up or dependency constraints and ask for the compute option or setting.
- Common confusion
- Running production jobs on an all-purpose cluster, which costs more and mixes interactive and scheduled work.
1.11 Develop unit and integration tests using assertDataFrameEqual, assertSchemaEqual, DataFrame.transform and other testing frameworksTesting PySpark logic before deployment.
- What it means
- PySpark ships testing utilities in pyspark.testing: assertDataFrameEqual compares rows (optionally ignoring order) and assertSchemaEqual compares schemas. DataFrame.transform chains functions that take and return DataFrames, which makes each step testable in isolation with frameworks such as pytest.
- What to understand
- Unit tests run small transformation functions against tiny in-memory DataFrames. Integration tests run the deployed job or pipeline against test data in a dev target, often triggered from CI after a bundle deploy. Keep business logic in importable modules rather than notebook cells so tests can import it. Check both data and schema to catch silent type changes.
- How it shows up
- Expect a choice of assertion or code structure that makes a transformation testable.
- Common confusion
- Comparing DataFrames with collect and equality, which fails on row order and hides schema differences.
1.12 Apply Structured Streaming stateful-processing semantics, including watermarks, output modes, foreachBatch and checkpoints for exactly-once state recoveryControlling state and guarantees in streaming queries.
- What it means
- Stateful operations such as aggregations, deduplication and stream-stream joins keep state between micro-batches. Watermarks tell Spark how late data may arrive so old state can be dropped. Output modes define what is written each trigger, and checkpoints store offsets and state so a restarted query resumes exactly once.
- What to understand
- Append mode emits a window only after the watermark passes it; update mode emits changed rows each trigger; complete mode rewrites the whole result. foreachBatch lets you run MERGE or write to several sinks per micro-batch, and should be idempotent using the batch ID. Each query needs its own checkpoint, and changing stateful logic may require a new checkpoint.
- How it shows up
- A streaming query runs out of memory, duplicates output or emits nothing, and you choose the fix.
- Common confusion
- Expecting append mode to emit aggregates immediately; it waits until the watermark closes the window.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Bundle targets
Named environments in databricks.yml with their own workspace, identity and variable values.
- Remember
- One bundle deploys to dev, staging and prod.
- Tested as
- Project structure and CI/CD questions.
- Trap
- Copying the project per environment.
-
Serverless environment dependencies
Package lists attached to serverless notebooks, jobs or pipelines.
- Remember
- Serverless does not use init scripts or cluster libraries.
- Tested as
- Library troubleshooting questions.
- Trap
- Trying to fix serverless with an init script.
-
Pandas UDF
Vectorized UDF that processes Arrow batches as pandas objects.
- Remember
- Much faster than row-at-a-time Python UDFs.
- Tested as
- UDF selection items.
- Trap
- Choosing a plain Python UDF for numeric scoring.
-
Unity Catalog function
A SQL or Python function registered in a catalog and schema.
- Remember
- Governed by EXECUTE and reusable across workspaces.
- Tested as
- Reuse and governance scenarios.
- Trap
- Using a temporary function for shared logic.
-
Auto Loader
The cloudFiles source that ingests new files incrementally.
- Remember
- Uses a schema location and the _rescued_data column.
- Tested as
- Streaming ingestion in pipelines.
- Trap
- Re-reading the whole directory every run.
-
Streaming table
A table fed by a stream that processes each input row once.
- Remember
- Best for append-only sources.
- Tested as
- Object choice questions.
- Trap
- Expecting it to recompute after source updates.
-
Materialized view
A stored query result refreshed incrementally where possible.
- Remember
- Reflects updates and deletes in its sources.
- Tested as
- Object choice questions.
- Trap
- Assuming it is always fully recomputed.
-
AUTO CDC with SEQUENCE BY
Applies change records in source order to a target table.
- Remember
- Sequence on the source change time, not arrival time.
- Tested as
- CDC pipeline questions.
- Trap
- Leaving out the sequence column.
-
SCD Type 2 columns
History rows carry __START_AT and __END_AT validity columns.
- Remember
- Set with stored_as_scd_type = 2.
- Tested as
- History tracking scenarios.
- Trap
- Confusing it with Type 1, which overwrites.
-
For each task
Runs a nested task once per item with set concurrency.
- Remember
- Inputs can come from task values.
- Tested as
- Job control-flow questions.
- Trap
- Duplicating tasks by hand.
-
Watermark
A threshold on event time that bounds late data and state.
- Remember
- Required to drop old state in aggregations and joins.
- Tested as
- Stateful streaming questions.
- Trap
- Thinking it filters data on ingest time.
-
foreachBatch
Runs batch logic, such as MERGE, on each micro-batch.
- Remember
- Make it idempotent with the batch ID.
- Tested as
- Upsert and multi-sink streaming.
- Trap
- Assuming it is exactly once without idempotent writes.
-
assertDataFrameEqual
PySpark testing helper comparing two DataFrames.
- Remember
- Pairs with assertSchemaEqual for type checks.
- Tested as
- Testing questions.
- Trap
- Comparing collected lists with ==.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Streaming table vs. materialized view vs. Structured Streaming query
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Streaming table | Incremental, append-style table in a pipeline or Databricks SQL | Processes each source row once | Used where sources update earlier rows |
| Materialized view | Precomputed query result with managed refresh | Correct after source updates and deletes | Assumed to be costlier in every case |
| Structured Streaming query | Hand-written readStream and writeStream code | Full control of sinks, triggers and checkpoints | Chosen when the declarative pipeline would remove operational work |
SCD Type 1 vs. SCD Type 2 in AUTO CDC
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| SCD Type 1 | Target keeps only the latest value per key | No history rows | Picked when auditors need prior values |
| SCD Type 2 | Each change creates a new versioned row | Adds __START_AT and __END_AT | Forgetting track_history limits which columns create versions |
Structured Streaming output modes
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Append | Writes only new final rows | With aggregations, waits for the watermark | Expecting immediate window results |
| Update | Writes rows changed since last trigger | Good for dashboards and upserts | Using it with file sinks that do not support it |
| Complete | Rewrites the entire result each trigger | State and output grow with all keys | Using it for unbounded key spaces |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
MERGE is always enough
- What candidates pick
- Write a custom MERGE in foreachBatch for every CDC feed.
- Why it looks right
- MERGE is familiar from the Associate exam.
- Why it's wrong
- A plain MERGE does not handle out-of-order events or history; AUTO CDC handles sequencing and SCD Type 2 for you.
Clue to watch for The scenario mentions late or out-of-order change records.
-
Streaming table for corrections
- What candidates pick
- Use a streaming table for an aggregate over a source that gets updates.
- Why it looks right
- Streaming tables are incremental and cheap.
- Why it's wrong
- Streaming tables do not revisit processed rows, so corrected source rows never fix the aggregate.
Clue to watch for The source receives updates or deletes after first load.
-
More memory fixes state
- What candidates pick
- Increase memory for a stateful query that slowly fails.
- Why it looks right
- Bigger compute seems to fix memory errors.
- Why it's wrong
- Without a watermark, state is never dropped, so more memory only delays the failure.
Clue to watch for A windowed aggregation or dedup with no withWatermark call.
-
Python UDF for speed
- What candidates pick
- Wrap numeric logic in a row-level Python UDF.
- Why it looks right
- It is the simplest UDF to write.
- Why it's wrong
- Row-at-a-time UDFs serialize each row; pandas UDFs or built-in functions are much faster.
Clue to watch for The requirement stresses large volume or performance.
-
All-purpose for production
- What candidates pick
- Schedule nightly jobs on a shared interactive cluster.
- Why it looks right
- The cluster is already running.
- Why it's wrong
- Jobs compute or serverless is cheaper and isolates production runs from interactive users.
Clue to watch for The question mentions cost and a scheduled workload.
-
Shared checkpoint
- What candidates pick
- Point two streaming queries at one checkpoint folder.
- Why it looks right
- It seems tidy to keep one location.
- Why it's wrong
- Each query must have its own checkpoint; sharing one corrupts offsets and state.
Clue to watch for Two writeStream calls with the same checkpointLocation.
Answer framework
Use these steps when a Section 1 scenario asks how to build or run a pipeline.
- Classify the source: append-only files or events, a change feed with updates and deletes, or periodic full snapshots, because that decides streaming table, AUTO CDC or snapshot CDC.
- Check the freshness and correctness requirement, then choose a streaming table for append workloads or a materialized view when results must reflect source changes.
- Decide whether a declarative pipeline covers the logic or whether custom sinks, triggers or state control call for hand-written Structured Streaming.
- Pick orchestration features: job triggers, parameters, If/else branches and For each loops, and run production work on jobs compute or serverless under a service principal.
- Confirm packaging and testing: shared code in a bundle wheel, dependencies pinned for the target compute, and tests using pyspark.testing assertions before deployment.
Practice question
Want more like this? The Databricks DE Pro practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Give this section the largest share of your hours, and study it hands-on. Build one small project as a Declarative Automation Bundle: a src package, a pipeline, a job and dev and prod targets. Deploy it with the CLI, break a dependency on purpose, and fix it on serverless so you see the real error messages.
Inside that pipeline, add an Auto Loader bronze table, a streaming silver table and a materialized view gold table, then add an AUTO CDC flow fed by sample change events that arrive out of order. Compare Type 1 and Type 2 outputs and note the validity columns.
Next, write a plain Structured Streaming query with a windowed aggregation. Run it without a watermark, then with one, and try each output mode so the difference in emitted rows is concrete. Add a foreachBatch MERGE and make it idempotent.
Finish with orchestration and testing: build a job with an If/else branch and a For each loop, and write pytest tests using pyspark.testing helpers. Keep a list of renamed features, because questions use the new names.
FAQ
How much of the exam is Developing Code for Data Processing using Python and SQL?
It is 23% of the scored content, the largest section, which works out to about 14 of the 60 scored questions.
Do I need to write code in the exam?
No. All items are multiple choice, but many show Python or SQL fragments and ask which one meets a requirement, so reading code fluently matters.
Are APPLY CHANGES and AUTO CDC the same thing?
AUTO CDC is the current name for the APPLY CHANGES APIs. The behaviour is the same: keys, a sequence column and an SCD type, with new function names in Python and SQL.
What changed in this section with the October 2026 exam?
It gained a stateful streaming objective covering watermarks, output modes, foreachBatch and checkpoints, and now names Apache Spark Declarative Pipelines alongside Lakeflow Pipelines.
When should I choose a materialized view over a streaming table?
Choose a materialized view when results must reflect updates or deletes in sources, or involve joins and aggregates over changing data. Choose a streaming table for append-only ingestion.
Which testing tools does the guide name?
assertDataFrameEqual, assertSchemaEqual and DataFrame.transform, plus other frameworks. In practice that means pytest with small in-memory DataFrames for unit tests.
Is serverless compute tested?
Yes. You should know serverless environments and dependencies, the higher memory option for notebook tasks, and when automatic retries should be turned off.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 1 knowledge
You have covered bundles, UDFs, Lakeflow Pipelines, AUTO CDC, job control flow, compute and stateful streaming. Check whether you can pick the right implementation under time pressure.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Databricks DE Pro AI tutor Get any concept explained your way