Overview
Domain 3 is the heaviest section at 30% of the exam, with ten official objectives. It tests whether you can write correct PySpark: adding, renaming, splitting and dropping columns, filtering rows and exploding arrays; removing duplicates and validating values; aggregating with count, approx_count_distinct, mean and summary; converting Unix epochs and extracting date parts; joining and unioning DataFrames; and reading and writing with explicit schemas.
The section also covers sorting, iterating and converting DataFrames to Python lists, user-defined functions including stateful processing backed by state stores, broadcast variables and accumulators, and broadcast joins. Questions usually show a snippet with one missing or wrong call, so precise knowledge of function names, arguments and return types matters more than broad theory.
How it's tested
Most items present PySpark code and ask which line completes it, which output it produces, or which option contains an error. Distractors are usually real methods used incorrectly, such as the wrong join type string or a function applied to the wrong data type. Read argument order and return types carefully.
-
Fill the blank
A snippet is missing one function or argument, and you choose the call that produces the stated output DataFrame.
-
Spot the bug
Several code options look similar, and you identify the one that fails or returns the wrong columns or rows.
-
Choose the join or union
Two DataFrames and a desired result are shown, and you pick the join type, keys or union method that matches.
Two DataFrames have the same columns in different orders, and the task is to stack them. df1.union(df2) would combine them by position and put values under the wrong headers without any error. df1.unionByName(df2) aligns columns by name, giving the correct result. If df2 lacked a column, allowMissingColumns=True would fill it with nulls. So unionByName is the answer.
Objectives 10
Official objectives from the Databricks outline, in outline order.
3.1 Manipulate columns, rows, and table structures by adding, dropping, splitting, renaming column names, applying filters, and exploding arrayswithColumn, drop, split, withColumnRenamed, filter and explode.
- What it means
- This objective covers reshaping DataFrames: adding or replacing columns with withColumn, removing them with drop, renaming with withColumnRenamed or withColumnsRenamed, splitting strings into arrays with split, filtering rows with filter or where, and turning array elements into rows with explode.
- What to understand
- withColumn returns a new DataFrame, so you must assign the result. split(col, pattern) takes a regular expression and returns an array you can index with getItem. explode drops rows whose array is null or empty; explode_outer keeps them with a null. filter accepts a column expression or a SQL string, and conditions are combined with & and |, each wrapped in parentheses.
- How it shows up
- Code-completion items ask which call adds, renames or flattens a column to match a sample output.
- Common confusion
- Using Python and/or instead of & and | in filter conditions, which raises an error.
3.2 Perform data deduplication and validation operations on DataFramesdistinct, dropDuplicates, null handling and value checks.
- What it means
- This objective covers removing duplicate rows and checking data quality. distinct() removes rows identical across all columns, while dropDuplicates(['col']) keeps one row per value of the listed columns. Validation uses filters such as isNull, isNotNull, isin and rlike, plus dropna and fillna to handle missing values.
- What to understand
- dropDuplicates on a subset keeps an arbitrary row from each group, not necessarily the first or latest, so ordering logic needs a window function with row_number instead. dropna(how='any') removes rows with any null, how='all' only fully null rows, and subset limits the check. Counting invalid rows before and after a rule is a common validation pattern.
- How it shows up
- Expect a requirement such as one row per customer, with options that differ in subset or method.
- Common confusion
- Expecting dropDuplicates to keep the most recent record; it does not guarantee which row survives.
3.3 Perform aggregate operations on DataFrames such as count, approximate count distinct, and mean, summarygroupBy and agg, approx_count_distinct, describe and summary.
- What it means
- This objective covers computing totals and statistics: df.count() for row counts, groupBy().agg() with functions such as count, sum, avg or mean, countDistinct for exact distinct counts, approx_count_distinct for fast estimates, and describe() or summary() for column statistics.
- What to understand
- approx_count_distinct uses HyperLogLog++ with a default maximum relative standard deviation of 0.05, trading a little accuracy for much less memory on large data. count('col') skips nulls while count('*') counts all rows. describe() returns count, mean, stddev, min and max; summary() adds percentiles such as 25%, 50% and 75%.
- How it shows up
- Questions ask which function gives a fast distinct estimate or which method outputs quartiles.
- Common confusion
- Mixing up df.count(), an action returning an integer, with F.count inside agg, which is an expression.
3.4 Manipulate and utilize Date data type, such as Unix epoch to date string, and extract date componentfrom_unixtime, to_date, date_format and date part functions.
- What it means
- This objective covers converting and decomposing dates and timestamps. from_unixtime(col, fmt) turns epoch seconds into a formatted string; unix_timestamp does the reverse. to_date and to_timestamp parse strings with a pattern, date_format renders them, and year, month, dayofmonth, dayofweek and hour extract components.
- What to understand
- Epoch values in milliseconds must be divided by 1000 before from_unixtime, which expects seconds. Patterns follow Spark's datetime syntax, so yyyy is year, MM is month and mm is minutes. from_unixtime returns a string, so wrap it in to_date or cast it when a real date type is required. dayofweek returns 1 for Sunday.
- How it shows up
- Snippets convert an epoch column and ask for the resulting value or type, or which pattern letter is wrong.
- Common confusion
- Using mm for month, which gives minutes instead.
3.5 Combine DataFrames with operations such as Inner join, left join, broadcast join, multiple keys, cross join, union, and union allJoin types, multi-key joins, cross joins and unions.
- What it means
- This objective covers combining DataFrames: df.join(other, on, how) with how values such as inner (the default), left, right, outer, left_semi and left_anti; joins on several keys; crossJoin for Cartesian products; and union, unionAll and unionByName for stacking rows.
- What to understand
- Passing on as a list of column names, such as ['id', 'date'], keeps one copy of each key, while an expression like df1.id == df2.id keeps both. In the DataFrame API, union and unionAll both keep duplicates and match by position; add distinct() to mimic SQL UNION. left_semi returns left rows with a match and only left columns; left_anti returns left rows without a match.
- How it shows up
- A target output is shown, and you choose the join type, key form or union method that produces it.
- Common confusion
- Thinking DataFrame union removes duplicates like SQL UNION; it behaves like UNION ALL.
3.6 Manage input and output operations by writing, overwriting, and reading DataFrames with schemasStructType or DDL schemas, read modes and overwrite.
- What it means
- This objective covers controlling structure at the I/O boundary. You can pass a StructType or a DDL string such as 'id INT, name STRING' to spark.read.schema() instead of inferring it, and write results with mode('overwrite') or other save modes. Reader modes PERMISSIVE, DROPMALFORMED and FAILFAST decide how corrupt records are handled.
- What to understand
- Explicit schemas avoid an extra pass over the data that inferSchema needs for CSV and JSON, and they keep types stable across runs. PERMISSIVE, the default, sets bad fields to null and can capture raw text in a corrupt-record column; DROPMALFORMED discards bad rows; FAILFAST stops at the first one. Overwrite replaces the target's existing data.
- How it shows up
- Questions ask how to read a file with fixed types quickly or how to stop a job when malformed rows appear.
- Common confusion
- Choosing inferSchema for speed; it is slower because it scans data first.
3.7 Perform operations on DataFrames such as sorting, iterating, printing schema, and conversion between DataFrame and sequence/list formatsorderBy, printSchema, collect, toLocalIterator and createDataFrame.
- What it means
- This objective covers inspecting and converting DataFrames. orderBy or sort with asc, desc and null-ordering variants sorts rows; printSchema shows the tree of column types; collect returns a list of Row objects to the driver; toLocalIterator streams them one partition at a time; and spark.createDataFrame builds a DataFrame from a Python list.
- What to understand
- collect() and toPandas() bring everything to the driver and can exhaust its memory, so prefer take(n), limit(n) or toLocalIterator() for large results. Row values are accessed by attribute or key. createDataFrame accepts a list of tuples with column names or a schema. desc_nulls_last and asc_nulls_first control where nulls appear.
- How it shows up
- Expect a list-to-DataFrame conversion or a question about which method risks driver memory.
- Common confusion
- Assuming printSchema returns a value; it prints and returns None.
3.8 Create and invoke user-defined functions with or without stateful operators, including StateStoresPython UDFs, SQL registration and stateful processing with state stores.
- What it means
- This objective covers writing custom logic. A row-at-a-time Python UDF is created with udf(func, returnType) or the @udf decorator and used in DataFrame expressions; spark.udf.register makes it callable from SQL. Stateful operators, such as streaming aggregations or custom state functions like applyInPandasWithState, keep per-key state between micro-batches in a state store.
- What to understand
- The default UDF return type is StringType, so specify the real type or results will be strings. Python UDFs serialise data between the JVM and Python and are opaque to Catalyst, so built-in functions are preferred. State stores are versioned key-value stores on executors, checkpointed for recovery; RocksDB is an alternative provider to the default HDFS-backed store.
- How it shows up
- A snippet registers a UDF, and you choose the correct return type or call; or you identify where streaming state is kept.
- Common confusion
- Forgetting the return type and getting string output from a numeric function.
3.9 Describe different types of variables in Spark, including broadcast variables and accumulatorsRead-only broadcast values and add-only accumulators.
- What it means
- This objective covers shared variables. A broadcast variable, created with spark.sparkContext.broadcast(value), ships a read-only copy of a lookup object to each executor once, accessed through .value. An accumulator, created with sparkContext.accumulator(0), lets tasks add to a counter that only the driver reads.
- What to understand
- Broadcasting avoids sending the same large object with every task. Accumulator updates made inside actions such as foreach are applied once per task, while updates inside transformations may be counted again if a task is retried or a stage recomputed. Both rely on SparkContext, which is not available through Spark Connect.
- How it shows up
- A scenario needs a lookup map on every executor or a count of bad records, and you choose the variable type.
- Common confusion
- Reading an accumulator's value inside a task; only the driver can read it reliably.
3.10 Describe the purpose and implementation of broadcast joinsSending a small table to every executor to avoid shuffling the large one.
- What it means
- This objective covers broadcast hash joins. When one side of a join is small, Spark copies it to every executor so the large side can be joined in place without a shuffle. You request it with the broadcast() function from pyspark.sql.functions or a SQL hint, and Spark chooses it automatically below spark.sql.autoBroadcastJoinThreshold.
- What to understand
- The automatic threshold defaults to 10 MB, and setting it to -1 disables automatic broadcasting. The broadcast side must fit in driver and executor memory, or the job may fail. Broadcast joins work for inner and most outer joins as long as the broadcast side is not the preserved side of the outer join.
- How it shows up
- A large fact table is joined to a small dimension and runs slowly; you choose the code that broadcasts the small side.
- Common confusion
- Broadcasting the large table instead of the small one.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
withColumn
Adds or replaces one column and returns a new DataFrame.
- Remember
- DataFrames are immutable; assign the result.
- Tested as
- Column manipulation snippets.
- Trap
- Calling it without reassigning.
-
explode vs explode_outer
Both turn array elements into rows; explode_outer keeps null or empty arrays.
- Remember
- explode drops those rows.
- Tested as
- Array flattening questions.
- Trap
- Expecting explode to preserve every input row.
-
dropDuplicates(subset)
Keeps one row per distinct combination of the listed columns.
- Remember
- Which row survives is arbitrary.
- Tested as
- Deduplication questions.
- Trap
- Relying on it for latest-record logic.
-
approx_count_distinct
HyperLogLog++ estimate of distinct values.
- Remember
- Default rsd is 0.05.
- Tested as
- Aggregation questions.
- Trap
- Choosing countDistinct when speed is required.
-
describe vs summary
Both return column statistics; summary also returns percentiles.
- Remember
- summary includes 25%, 50%, 75%.
- Tested as
- Summary statistics items.
- Trap
- Expecting quartiles from describe.
-
from_unixtime
Converts epoch seconds to a formatted string.
- Remember
- Divide milliseconds by 1000 first.
- Tested as
- Date conversion snippets.
- Trap
- Expecting a date type as output.
-
Join on a list of keys
on=['a','b'] joins on equal names and keeps one copy of each key.
- Remember
- Default join type is inner.
- Tested as
- Multi-key join questions.
- Trap
- Getting ambiguous duplicate columns from expression joins.
-
left_semi and left_anti
Filter left rows by existence or absence of a match.
- Remember
- Only left columns are returned.
- Tested as
- Join type selection.
- Trap
- Confusing left_anti with left outer.
-
union vs unionByName
union matches by position; unionByName matches by column name.
- Remember
- Neither removes duplicates.
- Tested as
- Union questions.
- Trap
- Assuming union aligns by name.
-
Read modes
PERMISSIVE, DROPMALFORMED and FAILFAST handle corrupt records.
- Remember
- PERMISSIVE is the default.
- Tested as
- Schema-driven read questions.
- Trap
- Picking FAILFAST when bad rows should be skipped.
-
collect() risk
Returns all rows to the driver as a list.
- Remember
- Can cause driver out-of-memory.
- Tested as
- Iteration and conversion items.
- Trap
- Using it on very large DataFrames.
-
UDF return type
udf defaults to StringType unless a type is given.
- Remember
- Specify the type explicitly.
- Tested as
- UDF snippets.
- Trap
- Numeric functions returning strings.
-
Accumulator
Add-only shared variable read on the driver.
- Remember
- Updates count once per task only inside actions.
- Tested as
- Shared variable questions.
- Trap
- Updating it in a transformation and trusting the count.
-
broadcast() hint
Marks a DataFrame to be copied to all executors for a join.
- Remember
- Auto threshold is 10 MB.
- Tested as
- Broadcast join questions.
- Trap
- Broadcasting the larger table.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Join types
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| inner | Rows with matches on both sides | Default join type | Expecting unmatched rows |
| left | All left rows plus matching right columns | Nulls where no match | Confusing with left_semi |
| left_semi | Left rows that have a match | Returns left columns only | Expecting right columns |
| left_anti | Left rows without a match | Returns left columns only | Confusing with left outer |
| cross | Every left row with every right row | Cartesian product | Running it on large tables |
union vs unionByName vs SQL UNION
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| union / unionAll | Stack rows by column position | Keeps duplicates | Assuming name matching |
| unionByName | Stack rows by column name | Optional allowMissingColumns | Forgetting it also keeps duplicates |
| SQL UNION | Stack and remove duplicates | Equivalent to union().distinct() | Equating it with DataFrame union |
Broadcast variable vs accumulator
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Broadcast variable | Read-only value cached on each executor | Driver writes, tasks read | Trying to modify it in tasks |
| Accumulator | Counter tasks add to | Tasks write, driver reads | Reading it inside tasks |
Python UDF vs built-in function
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Built-in function | Native expression in pyspark.sql.functions | Optimised by Catalyst, runs in the JVM | Writing a UDF for something that already exists |
| Python UDF | Custom row-at-a-time Python logic | Serialisation overhead, opaque to optimiser | Omitting the return type |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Union aligns names
- What candidates pick
- Pick union for DataFrames with reordered columns.
- Why it looks right
- union is the most familiar method.
- Why it's wrong
- union combines by position, so values land under the wrong column without any error.
Clue to watch for The column order differs between the two inputs.
-
Latest via dropDuplicates
- What candidates pick
- Use dropDuplicates(['id']) to keep each customer's newest record.
- Why it looks right
- It returns one row per id.
- Why it's wrong
- The surviving row is arbitrary; use a window with row_number ordered by timestamp descending.
Clue to watch for The requirement says latest, newest or most recent.
-
Month as mm
- What candidates pick
- Choose a pattern like 'yyyy-mm-dd' for dates.
- Why it looks right
- Lowercase looks consistent.
- Why it's wrong
- In Spark datetime patterns mm means minutes; months use uppercase MM.
Clue to watch for Output shows unexpected minute values in the month position.
-
Explode keeps all rows
- What candidates pick
- Expect explode to output a row for records with empty arrays.
- Why it looks right
- Every input row seems to need an output.
- Why it's wrong
- explode drops rows whose array is null or empty; explode_outer keeps them with null.
Clue to watch for Row counts before and after differ in the question.
-
UDF type defaults
- What candidates pick
- Register udf(lambda x: x * 2) and expect integers.
- Why it looks right
- Python infers types normally.
- Why it's wrong
- Without a returnType the UDF returns StringType, so the column holds strings.
Clue to watch for No return type appears in the udf call.
-
Broadcast the big side
- What candidates pick
- Wrap the larger DataFrame in broadcast().
- Why it looks right
- It seems the expensive side needs help.
- Why it's wrong
- Only the small table should be copied to every executor; broadcasting a large one can cause out-of-memory failures.
Clue to watch for One table is described as small or a lookup table.
-
Union removes duplicates
- What candidates pick
- Expect df1.union(df2) to return distinct rows.
- Why it looks right
- SQL UNION removes duplicates.
- Why it's wrong
- The DataFrame union keeps every row, like UNION ALL; add distinct() to remove duplicates.
Clue to watch for The expected output count equals the sum of both inputs.
Answer framework
For a Domain 3 code question, work through these checks before choosing an option.
- Compare the input and expected output to see which columns, rows or types change, and name the operation needed.
- Check function names, argument order and data types, such as seconds for from_unixtime or a list for multi-key joins.
- For joins and unions, decide whether matching is by key, position or name and whether duplicates or unmatched rows should remain.
- For custom logic, prefer a built-in function, and when a UDF is used confirm its declared return type and registration for SQL.
- Scan each option for driver-side risks or wrong scope, such as collect on large data, broadcasting a large table or reading accumulators in tasks.
Practice question
Want more like this? The Spark Dev Assoc practice test has questions for every domain.
Checklist
0 of 14
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Give this section the most time, because it carries 30% of the marks and almost every question contains code. Work in a notebook with pyspark.sql.functions imported as F and build small DataFrames with createDataFrame so you can see every result. For each objective, write the operation yourself, then change one argument and predict the new output before running it.
Group practice by confusion pairs: explode and explode_outer, union and unionByName, describe and summary, dropDuplicates and distinct, left and left_semi joins, broadcast variables and accumulators. Keep a short list of defaults, such as inner joins, StringType for UDFs, PERMISSIVE read mode and a 10 MB broadcast threshold.
When you can write these from memory, switch to timed snippet questions. Review every mistake by rerunning the code, since seeing the real output fixes the rule faster than rereading notes. Expect date patterns and union behaviour to reappear in Spark SQL and streaming questions too.
FAQ
How much weight does Developing Apache Spark DataFrame/DataSet API Applications carry?
It is the largest section at 30% of the scored exam, about 14 of 45 questions, with ten official objectives covering everyday PySpark DataFrame work.
Does the exam use the Dataset API?
Code is in Python, so questions use DataFrames. The section name keeps the Dataset term, but typed Datasets are only available in Scala and Java.
Does DataFrame union remove duplicates?
No. union and unionAll keep all rows and match columns by position. Use distinct() afterwards to remove duplicates, or unionByName to match by column name.
When should I use approx_count_distinct?
When an estimate is acceptable and the data is large. It uses far less memory than an exact countDistinct, with a default relative standard deviation of 5%.
Why are Python UDFs discouraged?
They move data between the JVM and Python row by row and cannot be optimised by Catalyst. Built-in functions or pandas UDFs are usually faster.
What happens if a broadcast table is too large?
Spark must hold it in driver and executor memory, so an oversized broadcast can fail with out-of-memory errors or slow the job considerably.
How do I keep the newest row per key?
Use a window partitioned by the key and ordered by timestamp descending, assign row_number, and keep rows where it equals 1.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 3 knowledge
Try timed PySpark snippet questions on joins, unions, dates, aggregations and UDFs to see how well you handle the exam's largest section.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Spark Dev Assoc AI tutor Get any concept explained your way