Databricks DE Pro · Domain 2 of 9

Databricks DE Pro Domain 2: Data Ingestion & Acquisition

Worth 12%, this section asks how data gets into the lakehouse or is reached in place: files and message buses, CDC into Delta or Iceberg, Lakeflow Connect database connectors, OpenSharing and Clean Rooms, and governed Lakehouse Federation.

Exam weight
12%
Official objectives
5
Estimated questions estimate
~7
Suggested study time estimate
8–11 hrs

About 13 min read Reviewed October 2026

Overview

Section 2 has five objectives that cover every route data takes into, or across, the platform. The first is ingesting many formats, including Delta Lake, Parquet, JSON, Iceberg, CSV and binary files, from cloud storage and from message buses such as Kafka, Kinesis and Pub/Sub. The second is configuring incremental CDC pipelines in Lakeflow Pipelines that write to Delta or Iceberg tables.

The third is Lakeflow Connect for change capture from SQL Server, MySQL and PostgreSQL. The last two moved here from the retired Data Sharing and Federation section: OpenSharing, in both Databricks-to-Databricks and Databricks-to-Open forms, together with Clean Rooms for privacy-preserving joint analysis, and Lakehouse Federation, where Unity Catalog permissions and connection credentials govern queries against external systems. The thread running through all five is choosing the right mechanism: copy, stream, share or query in place.

How it's tested

Items here describe a source system, a consumer and constraints such as latency, privacy or ownership, and ask which mechanism fits. Distractors are usually real features used in the wrong place, such as federation for a heavy nightly load or open sharing for a recipient that already runs Unity Catalog.

  • Copy, share or federate

    A consumer needs data from another system or organisation, and you decide between ingestion, OpenSharing, Clean Rooms and Lakehouse Federation.

  • Read the message bus

    A Kafka, Kinesis or Pub/Sub stream must land in a table, and you pick reader options, value decoding and checkpointing.

  • Database CDC source

    An operational database must replicate changes, and you choose Lakeflow Connect and the source-side prerequisites.

Reasoning it through

A retailer and a supplier want to measure overlap between their customer lists without either side seeing the other's records. OpenSharing would hand raw tables to the other party. Federation only reaches systems you hold credentials for. A Clean Room lets both parties contribute tables and run approved notebooks that return only agreed outputs, so Clean Rooms is correct.

Objectives 5

Official objectives from the Databricks outline, in outline order.

2.1 Develop data ingestion pipelines to ingest a variety of data formats, including Delta Lake, Parquet, JSON, Iceberg, CSV and Binary, from diverse sources such as message buses (Kafka, Kinesis, Pub/Sub) and cloud storageReading files and event streams of many formats.
What it means
This objective covers reading every common format into tables. Files in cloud storage are read with Auto Loader or read_files, choosing the format and options such as headers, multiline JSON or schema hints. Event streams are read with the Kafka, Kinesis or Pub/Sub connectors in Structured Streaming or in a pipeline.
What to understand
Message bus records arrive with a binary value and metadata such as key, topic, partition and offset, so you cast or parse the value, often with from_json or a VARIANT. Starting offsets and checkpoints control replay. The binaryFile format loads images or PDFs as bytes plus path and length. Iceberg and Delta tables can be read directly, while CSV and JSON benefit from explicit schemas.
How it shows up
Expect a source description and a choice of reader, format option or parsing step.
Common confusion
Forgetting that Kafka values are bytes that must be cast or parsed before use.
2.2 Configure incremental CDC pipelines using Lakeflow Pipelines with Delta or Iceberg as the target table formatLanding change data incrementally into open table formats.
What it means
This covers pipelines that read change events and keep a target table in sync incrementally, writing either Delta tables or Iceberg tables managed in Unity Catalog. The CDC logic uses AUTO CDC flows, while the target format decides which engines can read the output.
What to understand
Iceberg targets let external engines that speak Iceberg read the results through Unity Catalog's Iceberg REST interface. Delta remains the default with the widest platform feature support. Whichever format you write, the incremental contract is the same: process only new change events, apply them by key and sequence, and keep checkpoints managed by the pipeline.
How it shows up
A scenario requires interoperability with another engine, and you choose the target format and CDC configuration.
Common confusion
Assuming Iceberg output needs a separate copy job; the pipeline can target the format directly.
2.3 Configure CDC ingestion pipelines from relational database sources, including SQL Server, MySQL and PostgreSQL, using Lakeflow ConnectManaged change capture from operational databases.
What it means
Lakeflow Connect provides managed connectors that replicate tables from databases and SaaS applications into Unity Catalog. Database connectors use an ingestion gateway that reads changes from the source and an ingestion pipeline that applies them to streaming tables, optionally as SCD Type 2 history.
What to understand
Each database needs change capture enabled at the source: change tracking or CDC in SQL Server, binary logging in MySQL and logical replication in PostgreSQL. Credentials live in a Unity Catalog connection. The gateway runs continuously to avoid missing changes, while the ingestion pipeline can run on a schedule. This replaces custom JDBC polling scripts that miss deletes.
How it shows up
Questions ask what must be configured on the source or which component captures changes.
Common confusion
Using a scheduled JDBC full read and calling it CDC; it misses deletes and wastes source capacity.
2.4 Implement OpenSharing (D2D and Databricks-to-Open) and Clean Rooms for privacy-preserving collaborationSharing live data and running joint analysis safely.
What it means
OpenSharing, the open protocol earlier called Delta Sharing, lets a provider share tables, views, volumes and models without copying them. Databricks-to-Databricks sharing targets recipients on Unity Catalog workspaces; Databricks-to-Open sharing serves any client through credentials or token-based access. Clean Rooms let several parties analyze combined data through approved code without exposing raw rows.
What to understand
Shares are read-only for recipients and update as the provider's data changes. D2D sharing needs only the recipient's metastore identifier, while open sharing requires distributing credentials securely. History sharing enables time travel and change data feed for recipients. In Clean Rooms, collaborators add assets and approve notebooks, and only the outputs those notebooks produce leave the room.
How it shows up
Expect a partner collaboration scenario and a choice between D2D, open sharing and a Clean Room.
Common confusion
Using a share when partners must not see each other's raw records; that calls for a Clean Room.
2.5 Configure Lakehouse Federation with governance across supported source systems, applying UC permissions and connection-level credentialsQuerying external databases in place under Unity Catalog.
What it means
Lakehouse Federation registers an external system, such as PostgreSQL, MySQL, SQL Server, Snowflake, Redshift or BigQuery, through a Unity Catalog connection that stores credentials, and a foreign catalog that mirrors its databases and tables. Users then query those tables with Unity Catalog permissions applied.
What to understand
The connection holds one set of credentials, so access control should happen through Unity Catalog grants on the foreign catalog rather than many database logins. Filters and some operations are pushed down to the source, but heavy, repeated workloads are better ingested. Federation suits exploration, occasional joins and migration periods. Lineage and audit are recorded like other Unity Catalog objects.
How it shows up
A team needs occasional access to an external database without building a pipeline, and you choose federation and its permissions.
Common confusion
Granting users direct database credentials instead of Unity Catalog privileges on the foreign catalog.

High-yield concepts

The terms, rules and formulas this domain turns into questions most often.

  • Kafka value decoding

    Kafka records expose key and value as binary columns.

    Remember
    Cast or parse value with from_json or parse_json.
    Tested as
    Message bus ingestion items.
    Trap
    Querying value as if it were text.

  • binaryFile format

    Reads whole files as bytes with path, length and modification time.

    Remember
    Used for images, PDFs and other unstructured files.
    Tested as
    Format selection questions.
    Trap
    Using text format for binary content.

  • read_files

    SQL table-valued function that reads files with format options.

    Remember
    Works in Databricks SQL and streaming tables.
    Tested as
    SQL-based ingestion questions.
    Trap
    Thinking only Python can read raw files.

  • Iceberg target tables

    Pipeline outputs written as Unity Catalog managed Iceberg tables.

    Remember
    Readable by external Iceberg engines.
    Tested as
    Interoperability scenarios.
    Trap
    Adding a separate copy job to convert formats.

  • Ingestion gateway

    Lakeflow Connect component that captures database changes continuously.

    Remember
    Paired with an ingestion pipeline that applies them.
    Tested as
    Database CDC questions.
    Trap
    Scheduling the gateway and missing changes.

  • Source-side CDC prerequisites

    Change tracking or CDC, binlog, or logical replication on the source.

    Remember
    Must be enabled before the connector works.
    Tested as
    Configuration questions.
    Trap
    Assuming the connector enables it for you.

  • D2D sharing

    OpenSharing between Unity Catalog metastores.

    Remember
    Recipient identified by metastore ID; no tokens to manage.
    Tested as
    Partner sharing questions.
    Trap
    Using open sharing for Databricks recipients.

  • Databricks-to-Open sharing

    OpenSharing for any client, such as pandas, Spark or BI tools.

    Remember
    Credentials must be distributed securely.
    Tested as
    Non-Databricks recipient scenarios.
    Trap
    Assuming recipients can write.

  • Clean Rooms

    Shared, governed space where parties run approved notebooks on combined data.

    Remember
    Raw rows are not exposed to other collaborators.
    Tested as
    Privacy-preserving collaboration.
    Trap
    Choosing a plain share for sensitive joint analysis.

  • Foreign catalog

    Unity Catalog mirror of an external database through a connection.

    Remember
    Governed with normal UC grants.
    Tested as
    Federation questions.
    Trap
    Handing out source database logins.

  • Federation vs. ingestion

    Query in place versus copying into Delta or Iceberg.

    Remember
    Federate for occasional access; ingest for heavy repeated use.
    Tested as
    Mechanism choice items.
    Trap
    Federating a large nightly transformation.

Side by side

Concepts candidates mix up, compared on the point that decides the answer.

Moving or reaching external data

ConceptMeaningKey differenceCommon trap
Ingestion (Auto Loader, connectors) Copies data into lakehouse tables Best for repeated heavy processing Used when the data is needed only occasionally
Lakehouse Federation Queries the source in place No copy, but limited by source performance Used for large recurring workloads
OpenSharing Provider exposes live read-only data to recipients Crosses organisations without copying Expecting recipients to write back
Clean Rooms Joint analysis through approved code Partners never see each other's raw rows Replaced by a share when privacy is required

D2D vs. Databricks-to-Open sharing

ConceptMeaningKey differenceCommon trap
Databricks-to-Databricks Recipient runs Unity Catalog Identity by metastore ID Issuing bearer credentials unnecessarily
Databricks-to-Open Recipient uses any OpenSharing client Requires credential distribution Choosing it for a Databricks recipient

Database CDC prerequisites for Lakeflow Connect

ConceptMeaningKey differenceCommon trap
SQL Server Change tracking or CDC enabled Per database and table settings Forgetting to enable it on new tables
MySQL Binary logging enabled in row format Retention must cover gateway downtime Short binlog retention losing changes
PostgreSQL Logical replication with a replication slot Slot holds changes until consumed Ignoring slot growth when the gateway stops

Exam traps

Each trap is an answer that looks right. Learn the clue that should change your mind.

  1. Federation for ETL

    What candidates pick
    Point a nightly heavy transformation at a federated catalog.
    Why it looks right
    It avoids building a pipeline.
    Why it's wrong
    Every run re-reads the source over the network and loads it; repeated heavy work should be ingested.

    Clue to watch for The workload is large and recurring.

  2. Open sharing for Databricks recipient

    What candidates pick
    Create token-based open sharing for a partner on Unity Catalog.
    Why it looks right
    Open sharing works for everyone.
    Why it's wrong
    D2D sharing is simpler and avoids managing credentials when both sides use Unity Catalog.

    Clue to watch for The recipient has a Databricks workspace with Unity Catalog.

  3. Share for private overlap

    What candidates pick
    Share both customer tables to compute overlap.
    Why it looks right
    Sharing is the familiar collaboration tool.
    Why it's wrong
    Shares expose rows to the recipient; a Clean Room returns only approved outputs.

    Clue to watch for Neither party may see the other's records.

  4. JDBC polling as CDC

    What candidates pick
    Schedule a JDBC query on updated_at to capture changes.
    Why it looks right
    It is quick to build.
    Why it's wrong
    Polling misses hard deletes and loads the source; Lakeflow Connect reads the database's change log.

    Clue to watch for The requirement mentions deletes or minimal source impact.

  5. Direct database logins

    What candidates pick
    Give analysts the external database password.
    Why it looks right
    It seems the fastest way to grant access.
    Why it's wrong
    Federation centralises credentials in a connection and controls access through Unity Catalog grants and audit.

    Clue to watch for The scenario stresses governance or auditing.

Answer framework

Apply these steps when a Section 2 question asks how data should arrive or be reached.

  1. Identify the source type: files in storage, a message bus, an operational database, another Databricks account, an outside partner or an external warehouse.
  2. Decide whether the data must be copied for heavy repeated use or can stay in place for occasional queries, which separates ingestion from federation and sharing.
  3. For files and streams, pick Auto Loader, read_files or the bus connector, set format options, and plan how binary values or semi-structured payloads are parsed.
  4. For databases, choose Lakeflow Connect, confirm the source change-capture prerequisite, and pick the target table format and SCD type.
  5. For collaboration, use D2D or Databricks-to-Open sharing for read access, and a Clean Room when no party may see the other's raw records.

Practice question

Analysts need to query a PostgreSQL reporting database a few times a week and join it with lakehouse tables. Security requires central access control and auditing, and the data team does not want to maintain another pipeline. What should the data engineer configure?

Want more like this? The Databricks DE Pro practice test has questions for every domain.

Checklist

0 of 10

Tick each one you can do without notes. Your ticks stay in this browser.

Study strategy

Study this section as a set of mechanism choices. Write a one-line rule for each: Auto Loader for new files, bus connectors for event streams, Lakeflow Connect for database change capture, federation for occasional in-place queries, OpenSharing for read access across organisations, and Clean Rooms when nobody may see the other side's raw data.

Then get hands-on where you can. Read a sample Kafka or Kinesis stream, or simulate one, and practise casting the binary value and parsing JSON. Load a folder of mixed CSV and JSON files with explicit schemas and a folder of images with the binary format. Configure a CDC pipeline that writes to an Iceberg managed table and query it.

For Lakeflow Connect, read the setup docs for each database and memorise the source prerequisite, because exam items often hinge on it. For sharing, create a share and a D2D recipient if you have two metastores, and read how Clean Room notebooks are approved. Finish by creating a federated connection and granting SELECT on the foreign catalog to a group.

FAQ

How much of the exam is Data Ingestion & Acquisition?

Data Ingestion & Acquisition is 12% of the scored content, about seven of 60 questions, and now includes sharing and federation.

Is Delta Sharing still on the exam?

The October 2026 guide uses the name OpenSharing, with D2D and Databricks-to-Open forms. Study material that says Delta Sharing describes the same protocol under its earlier name.

What does Lakeflow Connect need on a SQL Server source?

Change tracking or change data capture must be enabled on the database and tables, plus a connection with suitable credentials and an ingestion gateway.

When is federation the wrong answer?

When the workload is heavy and repeated, such as a large nightly transformation. Copying the data in through ingestion is cheaper and faster for those cases.

What does a Clean Room protect?

Each collaborator's raw data. Parties add tables and run approved notebooks, and only the agreed outputs leave the room.

Can pipelines write Iceberg tables?

Yes. The objective expects you to configure incremental CDC pipelines with either Delta or Iceberg as the target table format.

Other Data & Analytics certifications

Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.

Test your Domain 2 knowledge

You have covered multi-format ingestion, CDC targets, Lakeflow Connect, OpenSharing, Clean Rooms and federation. Practise picking the right mechanism from short scenarios.

Take the Databricks DE Pro practice test Check your Databricks DE Pro readiness

Keep going