Spark Dev Assoc Tests Reading PySpark Code Precisely — Not Databricks Platform Features
Built around Apache Spark itself (DataFrame API, Spark SQL, Structured Streaming, Spark Connect) in Python. Most items show code; one argument, default or missing action usually decides the answer, so trace every line.
Check Your Readiness →Most candidates understand Databricks Certified Associate Developer for Apache Spark concepts — and still fail. This exam tests how you apply knowledge under pressure.
The guide does not state a Spark version; content follows the Spark 3.5 and 4.x Python DataFrame API. Unlike the Databricks engineer exams, platform services barely appear: questions ask what code returns, which line completes it, how many stages it creates, or which component does the work.
Leave it; append mode emits windows as data arrives
Append mode with an aggregation needs a watermark, declared on the event-time column before groupBy, so Spark knows when a window is final and can drop old state. Without it Spark rejects the query.
Increase spark.executor.memory and rerun the job unchanged
collect brings every row to the driver, so the driver ran out of memory. Keep work distributed, write results out or take a sample; spark.driver.maxResultSize caps returned data. Executor OOM instead points to large or skewed partitions.
Expect it to work exactly like a classic session
Spark Connect sends unresolved logical plans over gRPC and supports only DataFrame and SQL APIs. SparkContext, RDDs, broadcast variables and accumulators need a classic session; the server, not the client, analyses and executes plans.
Declare it with pd.Series -> pd.Series type hints
Series-to-Series UDFs must return output the same length as their input. A per-group aggregate needs Series-to-scalar hints used with groupBy().agg(), and the return type must still be given in the pandas_udf decorator.
Each action launches a job; wide transformations such as groupBy, join, distinct and orderBy add stage boundaries; narrow ones like filter, select and withColumn pipeline into one stage. Each stage then runs one task per partition.
coalesce merges partitions without a full shuffle, so coalesce(400) on 50 partitions still yields 50 and a hot key stays in one partition. repartition shuffles to any count; skew needs AQE skew join handling or salting both join sides.
The default save mode is errorifexists, so df.write.parquet(path) fails when data exists. append adds rows, overwrite replaces them, and ignore silently skips the write. bucketBy and sortBy only work with saveAsTable, never a path-based save.
udf(lambda x: x * 2) returns StringType by default, so numeric results become strings. Pass returnType explicitly, and prefer built-in functions or a pandas_udf: Python UDFs serialise row by row and are opaque to the Catalyst optimizer.
In Spark datetime patterns, mm is minutes and MM is months, so 'yyyy-mm-dd' puts minute values in the month slot. Also divide millisecond epochs by 1000 before from_unixtime, which returns a formatted string rather than a date type.
Strengthen weak areas with exam-style practice questions and detailed explanations.
Simulate the real exam experience and assess your readiness under timed conditions.
Review key concepts, objectives, and exam topics in one place.
Get personalized explanations, learning recommendations, and instant answers.
Follow a structured learning path designed to help you prepare efficiently.
Trace PySpark snippets under time pressure and see which defaults and edge cases still catch you.