Overview
Domain 1 is the conceptual base of the exam and carries 20% of the scored questions across seven official objectives. It asks you to explain where Spark helps and where it struggles, name the job of the driver, executors, cores and memory, and describe DataFrames, Datasets and the SparkSession lifecycle. It also covers caching, storage levels and garbage collection, the job-stage-task hierarchy, partitioning and shuffles, lazy evaluation, and the purpose of each Spark module.
These ideas are tested through short PySpark snippets as well as plain definitions. A question may show three transformations followed by count() and ask how many jobs run, or show a groupBy and ask where a stage boundary falls. Knowing what actually executes, and when, is what separates correct answers from plausible ones across the rest of the exam.
How it's tested
Questions in this section are either definitional or trace-the-code. Definitional items describe a component's behaviour and ask you to name it. Trace items show a few lines of PySpark and ask what triggers execution, how many stages appear, or what storage level applies. Read every line before judging, because one action changes the answer.
-
Name the component
A responsibility such as scheduling tasks or holding cached blocks is described, and you choose driver, executor, cluster manager or core.
-
Count the work
A short snippet is shown and you identify which call starts a job or where a shuffle splits the plan into stages.
-
Pick the storage behaviour
You decide what cache(), persist() or unpersist() does and when the data is actually materialised in executor memory.
A snippet reads a CSV file, filters rows, adds a column, calls cache(), then calls count(). Nothing before count() runs, because filter, withColumn and cache are lazy. The count() action launches the job, and while it runs the executors store the filtered partitions. A second action on the same DataFrame then reads from the cache instead of rescanning the file, which is the answer the question wants.
Objectives 7
Official objectives from the Databricks outline, in outline order.
1.1 Identify the advantages and challenges of implementing SparkWhy teams adopt Spark and where it causes problems.
- What it means
- This objective covers why Spark is chosen for large-scale processing: in-memory computation, a unified engine for batch, SQL, streaming and machine learning, horizontal scaling and APIs in Python, SQL, Scala and Java. It also covers the costs, such as shuffle overhead, memory pressure, small-file problems and the skill needed to tune distributed jobs.
- What to understand
- Spark speeds up iterative and multi-step work by keeping intermediate data in memory rather than writing it to disk between steps. Its weaknesses show up with tiny datasets where cluster overhead dominates, with skewed keys that leave one task running long after the others, and with operations that collect large results to the driver.
- How it shows up
- A scenario describes a workload, and you judge whether Spark's strengths apply or whether its distributed overhead outweighs them.
- Common confusion
- Assuming Spark is always faster; for small data a single-machine library can finish first.
1.2 Identify the role of core components of Apache Spark's Architecture, including cluster, driver node, worker nodes/executors, CPU cores, and memoryDriver, executors, worker nodes, cores, memory and the cluster manager.
- What it means
- This objective covers who does what in a running application. The driver runs your main program, creates the SparkSession, builds the plan and schedules tasks. Worker nodes host executor processes that run tasks and store cached data. Each executor core runs one task at a time, and executor memory is shared between execution and storage.
- What to understand
- The cluster manager (standalone, YARN or Kubernetes) allocates resources but does not run your logic. Parallelism is bounded by total executor cores, so ten executors with four cores each can run forty tasks at once. Driver memory matters when results are collected; executor memory matters for joins, aggregations and caching.
- How it shows up
- Expect a described responsibility, such as splitting work into tasks, with options naming different components.
- Common confusion
- Thinking the driver processes the data; it coordinates, while executors do the row-level work.
1.3 Describe the architecture of Apache Spark, including DataFrame and Dataset concepts, SparkSession lifecycle, caching, storage levels, and garbage collectionDataFrames vs Datasets, SparkSession, cache and persist, storage levels and GC.
- What it means
- This objective covers the main abstractions and their lifecycle. A DataFrame is a distributed, immutable collection of rows with a schema. A Dataset adds compile-time typing and exists only in Scala and Java; in PySpark you work with DataFrames. SparkSession is the single entry point, created with SparkSession.builder.getOrCreate() and ended with stop().
- What to understand
- cache() stores a DataFrame with the default memory-and-disk storage level, while persist() accepts an explicit level such as MEMORY_ONLY or DISK_ONLY. Both are lazy and take effect when an action runs; unpersist() releases the blocks. Heavy caching of many objects increases JVM garbage collection, which shows up as GC time in the Executors tab and slows tasks.
- How it shows up
- Look for snippets where cache() appears without an action, or options that claim Datasets are available in Python.
- Common confusion
- Believing cache() materialises data immediately; it only marks the DataFrame for storage.
1.4 Explain the Apache Spark Architecture execution hierarchyApplication, job, stage and task.
- What it means
- This objective covers how work is broken down. An application is one SparkSession and its executors. Each action triggers a job. The scheduler splits a job into stages at shuffle boundaries, and each stage runs one task per partition on the executors.
- What to understand
- Narrow transformations such as select, filter and withColumn stay within a stage because each output partition depends on one input partition. Wide transformations such as groupBy, distinct, orderBy and most joins need data from many partitions, so they introduce a shuffle and a new stage. The number of tasks in a stage equals the number of partitions it processes.
- How it shows up
- Questions show a chain of transformations and an action, then ask how many jobs or stages run, or what defines a task.
- Common confusion
- Confusing stages with jobs; a single job can contain several stages separated by shuffles.
1.5 Configure Spark partitioning in distributed data processing, including shuffles and partitionsPartition counts, shuffle partitions, repartition and coalesce.
- What it means
- This objective covers how data is divided and how that division changes. File reads produce partitions sized by spark.sql.files.maxPartitionBytes, which defaults to 128 MB. After a shuffle, the partition count comes from spark.sql.shuffle.partitions, which defaults to 200. You can change partitioning with repartition() or coalesce().
- What to understand
- repartition(n) or repartition(col) performs a full shuffle and can increase or decrease partitions, producing evenly sized output. coalesce(n) merges existing partitions without a full shuffle, so it is cheaper but can only reduce the count and may leave partitions uneven. getNumPartitions() on df.rdd reports the current number in classic Spark.
- How it shows up
- A scenario asks how to raise parallelism, or how to reduce output files cheaply before a write.
- Common confusion
- Using coalesce to increase partitions; it cannot add partitions without a shuffle.
1.6 Describe the execution patterns of the Apache Spark engine, including actions, transformations, and lazy evaluationTransformations build plans; actions run them.
- What it means
- This objective covers the split between transformations, which return a new DataFrame and only extend the logical plan, and actions, which return results to the driver or write output and so trigger execution. Lazy evaluation lets the Catalyst optimizer see the whole plan before running anything.
- What to understand
- Common actions are count(), collect(), take(), first(), show(), and write operations such as save() or saveAsTable(). Transformations include select, filter, withColumn, join, groupBy().agg() and orderBy. Because nothing runs until an action, Catalyst can push filters down to the data source, prune columns and reorder operations for efficiency.
- How it shows up
- Expect a list of calls where you must spot which ones are actions, or a question on why lazy evaluation helps optimisation.
- Common confusion
- Treating show() or write as transformations; both trigger execution.
1.7 Identify the features of the Apache Spark Modules, including Core, Spark SQL, DataFrames, Pandas API on Spark, Structured Streaming, and MLibWhat each Spark library is for.
- What it means
- This objective covers the libraries layered on Spark Core. Core provides task scheduling, memory management and the RDD abstraction. Spark SQL and the DataFrame API provide structured processing with the Catalyst optimizer. Pandas API on Spark offers pandas syntax at scale. Structured Streaming processes unbounded data. MLlib (written MLib in the guide) supplies distributed machine learning.
- What to understand
- All the higher-level modules compile down to the same engine, so a DataFrame query and a SQL query on the same data produce equivalent plans. MLlib's DataFrame-based spark.ml package is the primary API; the older RDD-based API is in maintenance mode. Structured Streaming reuses the DataFrame API, which is why most batch transformations also work on streams.
- How it shows up
- A requirement is described, such as pandas-style code on large data, and you choose the module that fits.
- Common confusion
- Thinking SQL queries run on a different engine from DataFrame code; both share Catalyst.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Driver
The process that runs main(), holds the SparkSession, plans queries and schedules tasks.
- Remember
- It coordinates; it does not process partitions.
- Tested as
- Component identification questions.
- Trap
- Assuming collect() results go to executors.
-
Executor
A process on a worker node that runs tasks and stores cached partitions.
- Remember
- Cores per executor set how many tasks it runs at once.
- Tested as
- Architecture and tuning items.
- Trap
- Confusing executors with worker machines.
-
Job, stage, task
An action creates a job; shuffles split it into stages; each stage runs one task per partition.
- Remember
- Stage count rises with each wide transformation.
- Tested as
- Trace-the-code questions.
- Trap
- Counting transformations as jobs.
-
Narrow vs wide transformation
Narrow needs one input partition per output partition; wide needs many and shuffles.
- Remember
- groupBy, join, distinct and orderBy are wide.
- Tested as
- Stage boundary questions.
- Trap
- Calling filter a wide transformation.
-
spark.sql.shuffle.partitions
Number of partitions produced after a DataFrame shuffle.
- Remember
- Default is 200.
- Tested as
- Partitioning and tuning items.
- Trap
- Expecting it to control file-read partitions.
-
repartition vs coalesce
repartition fully shuffles to any count; coalesce merges partitions to a lower count without a full shuffle.
- Remember
- Only repartition can increase partitions.
- Tested as
- Partition configuration questions.
- Trap
- Picking coalesce to raise parallelism.
-
cache() and persist()
Mark a DataFrame for reuse; persist() takes an explicit StorageLevel.
- Remember
- Both are lazy until an action runs.
- Tested as
- Caching snippets.
- Trap
- Thinking cache() runs a job.
-
Storage levels
Options such as MEMORY_ONLY, MEMORY_AND_DISK and DISK_ONLY control where cached blocks live.
- Remember
- Memory-and-disk spills instead of recomputing.
- Tested as
- Storage level questions.
- Trap
- Assuming MEMORY_ONLY spills to disk.
-
Lazy evaluation
Transformations only extend the plan; execution waits for an action.
- Remember
- Enables Catalyst to optimise the whole plan.
- Tested as
- Execution pattern questions.
- Trap
- Believing errors in a transformation appear immediately.
-
DataFrame vs Dataset
DataFrame is untyped rows with a schema; Dataset is typed and JVM-only.
- Remember
- PySpark exposes DataFrames only.
- Tested as
- Architecture definitions.
- Trap
- Choosing a Dataset answer for Python code.
-
Catalyst optimizer
Spark SQL's optimiser that rewrites logical plans and chooses physical plans.
- Remember
- Applies to SQL and DataFrame code alike.
- Tested as
- Module and lazy evaluation items.
- Trap
- Thinking Python UDFs are optimised by it.
-
Garbage collection
JVM reclaiming memory from unused objects on driver and executors.
- Remember
- High GC time signals memory pressure.
- Tested as
- Architecture and monitoring questions.
- Trap
- Blaming GC on the Python process.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Driver vs executor vs cluster manager
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Driver | Runs the program, plans and schedules | Coordinates the application | Thinking it processes all rows |
| Executor | Runs tasks and caches data | Does the distributed work | Confusing it with a worker node |
| Cluster manager | Allocates CPU and memory to the app | Resource broker only | Thinking it optimises queries |
repartition vs coalesce
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| repartition(n) | Full shuffle to n partitions | Can increase or decrease; balanced output | Using it just to shrink output cheaply |
| coalesce(n) | Merges partitions without a full shuffle | Can only decrease; may be uneven | Using it to increase parallelism |
cache vs persist vs unpersist
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| cache() | Store with the default storage level | No level argument | Expecting immediate storage |
| persist(level) | Store with a chosen StorageLevel | Explicit control of memory, disk, replication | Forgetting it is also lazy |
| unpersist() | Release cached blocks | Frees executor storage memory | Assuming cached data clears automatically when no longer needed |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Every line runs
- What candidates pick
- Assume filter or withColumn executes as soon as it is called.
- Why it looks right
- Python normally runs statements in order.
- Why it's wrong
- Transformations are lazy and only build the plan; execution starts with an action such as count or write.
Clue to watch for The snippet has several transformations and only one action at the end.
-
Coalesce to scale up
- What candidates pick
- Use coalesce(400) on a 50-partition DataFrame to add parallelism.
- Why it looks right
- coalesce is described as the cheaper option.
- Why it's wrong
- Without a shuffle, coalesce cannot create new partitions, so the count stays at 50.
Clue to watch for The requirement is to increase the number of partitions.
-
Driver crunches data
- What candidates pick
- Say the driver executes the filter on all rows.
- Why it looks right
- The driver runs the user program.
- Why it's wrong
- The driver plans and schedules; executors apply the filter to their partitions in parallel.
Clue to watch for The question asks which component performs row-level work.
-
Cache is instant
- What candidates pick
- Pick an answer saying the data is in memory right after cache().
- Why it looks right
- The method name sounds immediate.
- Why it's wrong
- cache() only marks the DataFrame; blocks are stored as the first action computes each partition.
Clue to watch for No action appears between cache() and the question's checkpoint.
-
Python Datasets
- What candidates pick
- Choose a typed Dataset approach for a PySpark job.
- Why it looks right
- The objective mentions Datasets.
- Why it's wrong
- Datasets with compile-time types exist in Scala and Java; Python uses untyped DataFrames.
Clue to watch for All code in the exam is Python.
-
Stage per transformation
- What candidates pick
- Count one stage for each select, filter and withColumn.
- Why it looks right
- Each call looks like a step.
- Why it's wrong
- Narrow transformations are pipelined into one stage; only shuffles create stage boundaries.
Clue to watch for The chain contains no groupBy, join, distinct or orderBy.
Answer framework
Use these steps to trace any Domain 1 snippet or architecture scenario toward the answer Databricks expects.
- Find the actions in the snippet first, because each action starts a job and nothing before it has executed yet.
- Mark every wide transformation, such as groupBy, join, distinct or orderBy, since each one adds a shuffle and a new stage.
- Work out partition counts at each step using file split size, shuffle partitions and any repartition or coalesce call.
- Decide which component owns the described responsibility: driver for planning and results, executors for tasks and cached blocks.
- For caching questions, confirm that an action has run, check which storage level applies, and look for unpersist calls that release memory.
Practice question
Want more like this? The Spark Dev Assoc practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Start by running a small notebook on Databricks Free Edition and opening the Spark UI after each action. Watch how one action becomes one job, how a groupBy splits that job into two stages, and how the task count matches the partition count. Seeing this once makes the hierarchy questions straightforward.
Next, experiment with partitioning. Read a file, check the partition count, run a groupBy and check again, then try repartition and coalesce and compare the results. Change spark.sql.shuffle.partitions and observe the effect on task counts. Then practise caching: call cache() with and without a following action and confirm in the Storage tab when blocks appear.
Finish with definitions. Write one line for each component and each module, note that Python uses DataFrames rather than Datasets, and list the common actions from memory. Timed practice questions will expose any remaining gaps, especially snippet questions where a single action changes the answer. This section is 20% of the exam and supports Domains 3 and 4, so do it first.
FAQ
How much of the exam is Apache Spark Architecture and Components?
Databricks weights Apache Spark Architecture and Components at 20% of the scored exam, about nine of the 45 questions, across seven objectives.
Do I need to know Scala for this section?
No. All exam code is Python. You only need to recognise that typed Datasets are a Scala and Java feature, while PySpark works with DataFrames.
What creates a new stage?
A shuffle. Wide transformations such as groupBy, join, distinct and orderBy redistribute data across partitions, so the scheduler ends one stage and begins another at that point.
Is cache() the same as persist()?
cache() is persist() with the default storage level. persist() lets you pick a level such as MEMORY_ONLY or DISK_ONLY. Both are lazy until an action runs.
How many tasks run at once?
At most one per available executor core. A stage with more partitions than cores runs its tasks in several waves.
Which Spark version should I study?
The current guide does not name a Spark version. Study the Spark 3.5 and 4.x DataFrame API as documented by Databricks and avoid deprecated features.
Why does lazy evaluation matter for performance?
Because Spark sees the whole plan before running it, Catalyst can push filters to the source, drop unused columns and combine narrow steps into a single stage.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 1 knowledge
Check whether you can count jobs and stages, choose the right partitioning call and predict caching behaviour under timed conditions.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Spark Dev Assoc AI tutor Get any concept explained your way