Overview
Ingesting and processing the data is the heaviest section of the Professional Data Engineer exam at about 25%. Google lists three objectives. Objective 2.1 is planning: sources and sinks, transformation and orchestration logic, networking and encryption. Objective 2.2 is building: cleansing, choosing among Dataflow, Apache Beam, Dataproc, Cloud Data Fusion, BigQuery, Pub/Sub, Spark, Hadoop and Kafka, writing batch and streaming transformations with windowing and late data, enriching data with AI and integrating new sources.
Objective 2.3 is deployment: automating jobs with Cloud Composer or Workflows and running CI/CD for pipeline code. Google has since renamed Cloud Composer to Managed Service for Apache Airflow and Dataproc to Managed Service for Apache Spark, but the exam guide still prints the older names, so both appear below. Plan for roughly a quarter of your questions to come from this section.
How it's tested
Domain 2 questions give a data source, a latency target, an existing skill set and sometimes existing code, then ask which service or pipeline design fits. The decisive clues are words like real time, existing Spark jobs, no-code, late arriving events, exactly once and minimal operations.
-
Engine selection
Choose Dataflow, Managed Service for Apache Spark, Cloud Data Fusion or BigQuery SQL based on code reuse, latency and team skills.
-
Streaming semantics
Pick window type, trigger, watermark or allowed lateness so aggregations stay correct when events arrive late or out of order.
-
Orchestrate and ship
Decide between Airflow DAGs, Workflows or scheduled queries, and how to test and promote pipeline code safely.
A logistics firm streams truck sensor readings through Pub/Sub and needs five-minute average speeds per truck, but cellular gaps delay some readings by up to ten minutes. A Dataflow pipeline with fixed five-minute windows, a watermark-based trigger and allowed lateness of ten minutes, plus late-firing panes, updates results when delayed data lands. Processing time windows would put delayed readings in the wrong interval.
Objectives 3
Official objectives from the Google Cloud outline, in outline order.
2.1 Planning the data pipelinesDefining sources and sinks, transformation and orchestration logic, networking fundamentals and data encryption.
- What it means
- Before writing code you decide where data comes from, where it lands, what happens in between and how steps are coordinated. This objective also covers how pipeline workers reach private sources across networks and how data stays encrypted in transit and at rest.
- What to understand
- Sources include Pub/Sub topics, Cloud Storage files, operational databases through Datastream and external APIs; sinks include BigQuery, Bigtable, Cloud Storage and Spanner. Decide whether logic belongs in ELT inside BigQuery or ETL in Dataflow before load. Networking matters when Dataflow or Spark workers need private IPs, Private Google Access, Shared VPC subnets or Cloud VPN and Interconnect to reach on-premises systems. Firewall rules must allow worker-to-worker traffic. Data is encrypted in transit by default, and CMEK can protect pipeline state and temporary files.
- How it shows up
- A pipeline cannot reach an on-premises database or must avoid public IPs; look for private networking settings.
- Common confusion
- Assuming Dataflow workers need external IP addresses; with Private Google Access they can run on internal IPs only.
2.2 Building the pipelinesCleansing, identifying services, batch and streaming transformations with windowing and late data, AI enrichment, data acquisition and new sources.
- What it means
- This is the build step. You clean incoming data, choose the processing service, write batch and streaming transformations that handle windowing and late arrivals, enrich records with AI, acquire and import data, and connect new sources without breaking existing flows.
- What to understand
- Dataflow runs Apache Beam pipelines serverlessly with autoscaling and exactly-once processing, and suits new batch or streaming work. Managed Service for Apache Spark (Dataproc in the guide) runs existing Spark and Hadoop code with little change. Cloud Data Fusion builds pipelines visually. Pub/Sub decouples producers and consumers; Managed Service for Apache Kafka fits teams already using Kafka APIs. Beam windows are fixed, sliding or session; watermarks estimate event-time completeness and allowed lateness keeps windows open. Dead-letter outputs capture bad records. AI enrichment can call Gemini models through Agent Platform or BigQuery AI functions.
- How it shows up
- Existing Spark code points to Managed Spark; new streaming with late data points to Dataflow; drag-and-drop points to Data Fusion.
- Common confusion
- Mixing session windows, which group by activity gaps per key, with sliding windows, which overlap on a fixed schedule.
2.3 Deploying and operationalizing the pipelinesJob automation and orchestration with Cloud Composer and Workflows, plus CI/CD for pipeline code.
- What it means
- After a pipeline works, it has to run on schedule, in the right order, and be updated safely. This objective covers orchestrating jobs with Managed Service for Apache Airflow (Cloud Composer in the guide) or Workflows and building CI/CD for pipeline code and templates.
- What to understand
- Managed Airflow suits multi-step workflows with dependencies, retries, backfills and many operators for Google Cloud services. Workflows is a lighter serverless orchestrator for chaining API calls and short service sequences. BigQuery scheduled queries and Dataform schedules handle SQL-only jobs. Dataflow Flex Templates package pipelines so CI/CD can build a container image, run tests and launch the same template in each environment. Cloud Build with Artifact Registry automates build and deploy, and streaming jobs can be updated in place or drained and replaced.
- How it shows up
- Many dependent steps across services with retries suggest Managed Airflow; a short API chain suggests Workflows.
- Common confusion
- Using Managed Airflow to process data itself; it should orchestrate jobs, not perform heavy transformations in its workers.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Dataflow
Serverless Apache Beam runner for batch and streaming.
- Remember
- Autoscaling and exactly-once processing.
- Tested as
- Default for new pipelines with low operations effort.
- Trap
- Picking it when existing Spark code must be reused.
-
Managed Service for Apache Spark
Managed Spark and Hadoop, formerly Dataproc, with a serverless option.
- Remember
- Lift existing Spark jobs with minimal rewrite.
- Tested as
- Code reuse and open-source ecosystem cases.
- Trap
- Running long-lived clusters for occasional jobs.
-
Pub/Sub
Global messaging service for event ingestion.
- Remember
- At-least-once by default; exactly-once delivery is optional per subscription.
- Tested as
- Decoupling producers from streaming consumers.
- Trap
- Expecting ordering without ordering keys.
-
Watermark
Estimate of how complete event-time data is.
- Remember
- Triggers fire relative to it.
- Tested as
- Late data and correctness questions.
- Trap
- Treating it as a guarantee no late data will arrive.
-
Allowed lateness
How long a window accepts late elements after the watermark.
- Remember
- Late data beyond it is dropped.
- Tested as
- Delayed mobile or IoT events.
- Trap
- Leaving it at zero with known delays.
-
Window types
Fixed, sliding and session windows in Beam.
- Remember
- Session windows depend on gaps per key.
- Tested as
- Aggregation design questions.
- Trap
- Using fixed windows for user sessions.
-
Cloud Data Fusion
Visual, code-free pipeline builder based on CDAP.
- Remember
- Many prebuilt connectors.
- Tested as
- Teams without coding skills.
- Trap
- Choosing it for low-latency custom streaming logic.
-
Managed Service for Apache Airflow
Managed Airflow for DAG orchestration (Cloud Composer in the guide).
- Remember
- Dependencies, retries, backfills.
- Tested as
- Multi-step cross-service workflows.
- Trap
- Doing heavy processing inside Airflow tasks.
-
Dataflow Flex Templates
Container-packaged reusable Dataflow pipelines.
- Remember
- Supports CI/CD and parameterized launches.
- Tested as
- Deployment and promotion questions.
- Trap
- Rebuilding pipelines by hand per environment.
-
Dead-letter queue
Side output or topic for records that fail processing.
- Remember
- Keeps the pipeline running while preserving bad data.
- Tested as
- Cleansing and error handling.
- Trap
- Silently dropping malformed records.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Processing engines
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Dataflow | Serverless Beam pipelines. | Unified batch and streaming, no clusters. | Rewriting working Spark code without need. |
| Managed Service for Apache Spark | Managed Spark and Hadoop. | Best for existing open-source jobs. | Keeping clusters running between jobs. |
| Cloud Data Fusion | Visual pipeline design. | No-code with connectors. | Expecting fine-grained streaming control. |
| BigQuery SQL (ELT) | Transform after loading. | No separate engine needed. | Forcing an external engine for SQL-friendly work. |
Orchestration choices
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Managed Service for Apache Airflow | Python DAGs with operators. | Rich dependencies and backfills. | Overkill for one scheduled query. |
| Workflows | Serverless YAML or JSON service chaining. | Light, pay per step. | Using it for complex data DAGs with backfills. |
| Scheduled queries / Dataform | SQL-only scheduling in BigQuery. | Simplest for warehouse transforms. | Expecting cross-service orchestration. |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Kafka by habit
- What candidates pick
- Self-manage Kafka on Compute Engine
- Why it looks right
- Kafka is familiar and widely used.
- Why it's wrong
- Self-managed clusters add operations work; Pub/Sub or Managed Service for Apache Kafka meets the need with less effort.
Clue to watch for Minimal operations, new ingestion design.
-
Rewrite for serverless
- What candidates pick
- Port all Spark jobs to Dataflow
- Why it looks right
- Serverless sounds more modern.
- Why it's wrong
- The scenario values reuse and speed; Managed Spark runs the code nearly unchanged.
Clue to watch for Existing Spark or Hadoop code, short timeline.
-
Drop late data
- What candidates pick
- Use default allowed lateness
- Why it looks right
- Defaults seem safe.
- Why it's wrong
- Default lateness discards events that arrive after the watermark, skewing aggregates when delays are known.
Clue to watch for Devices go offline or events arrive late.
-
Airflow does the work
- What candidates pick
- Run transformations inside Airflow Python tasks
- Why it looks right
- Everything stays in one tool.
- Why it's wrong
- Airflow workers are for coordination; heavy processing belongs in Dataflow, Spark or BigQuery.
Clue to watch for Large data volumes in an orchestration question.
-
Public workers
- What candidates pick
- Give pipeline workers external IPs to reach sources
- Why it looks right
- It makes connectivity easy.
- Why it's wrong
- Security rules often forbid public IPs; Private Google Access and VPN or Interconnect solve reachability privately.
Clue to watch for No public IP allowed, on-premises source.
-
Ordering for free
- What candidates pick
- Assume Pub/Sub delivers messages in order
- Why it looks right
- Messages are published in order.
- Why it's wrong
- Ordering needs ordering keys and only applies per key within a region.
Clue to watch for Per-entity order matters.
Answer framework
Apply these steps when a Domain 2 question asks which ingestion or processing design to choose.
- Identify the latency need first: batch on a schedule, micro-batch, or true streaming with results in seconds, because that narrows the service list quickly.
- Check for existing code or skills: Spark and Hadoop jobs favor Managed Spark, SQL teams favor BigQuery ELT, and no-code teams favor Cloud Data Fusion.
- For streaming, decide event time versus processing time, pick fixed, sliding or session windows, and set allowed lateness to the known delay.
- Plan error handling and networking: dead-letter outputs for bad records, private worker IPs, and VPN or Interconnect for on-premises sources.
- Choose orchestration by complexity, from scheduled queries to Workflows to Managed Airflow, and package pipelines as templates for CI/CD promotion.
Practice question
Want more like this? The GCP PDE practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Give this section the largest share of your hours because it carries the largest weight. Begin with the Apache Beam model: run the word count example on Dataflow, then build a streaming pipeline that reads from Pub/Sub, applies fixed windows and writes to BigQuery. Publish some messages with old timestamps and watch how allowed lateness changes the output.
Next, run a PySpark job on Managed Service for Apache Spark, once on an ephemeral cluster and once as a serverless batch, and compare startup time and cost. Open Cloud Data Fusion and build one pipeline visually so you recognize its strengths. Then write a small Airflow DAG that triggers a Dataflow template and a BigQuery query with a dependency between them. Keep a decision table of engines and orchestrators with the clue words that select each. Finish with timed practice questions, and after each one, write down which keyword pointed to the correct service.
FAQ
How much of the GCP PDE exam is Ingesting and processing the data?
Ingesting and processing the data is the largest section at about 25% of the exam, which is roughly 10 to 13 questions on a 40-50 question paper.
Do I need to write Apache Beam code on the exam?
No coding is required, but you should recognize Beam concepts such as PCollections, ParDo, windows, triggers and side outputs, and know what each controls.
Is Dataproc still on the exam?
Yes. The exam guide names Dataproc, which Google now calls Managed Service for Apache Spark. Questions may use either name until the guide is refreshed.
When is Cloud Data Fusion the right answer?
When the team needs a visual, low-code pipeline builder with prebuilt connectors and does not want to write Beam or Spark code.
How deep does streaming go?
Expect questions on event time, windows, watermarks, late data, exactly-once processing and Pub/Sub delivery behavior. These are among the most tested ideas in this section.
Should I learn Workflows as well as Airflow?
Yes. The guide names both. Know that Workflows suits short serverless service chains and Airflow suits complex DAGs with dependencies and backfills.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 2 knowledge
You have reviewed pipeline planning, engines, streaming semantics and deployment. See how quickly you can match clue words to the right service in timed questions.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the GCP PDE AI tutor Get any concept explained your way