Overview
Distributed Systems has three topics. The distributed cloud computing review covers cloud fundamentals, parallel versus distributed computing, scalability and fault tolerance, distributed file systems and distributed processing frameworks. The Apache Spark introduction explains Spark's architecture (driver, cluster manager, executors), how it processes data with lazy transformations and actions, and machine learning with Spark MLlib. Spark deployment covers cluster management systems, infrastructure sizing, monitoring and debugging, scaling and deployment practices.
The module is what separates the EBDS scientist from a single-laptop data scientist. Every model from Modules 3 to 5 must eventually run on data that does not fit on one machine, and the exam asks you to recognize where distribution changes the answer: shuffles, data skew, memory limits, partitioning and cluster managers. It also links back to the EBDF architecture element reviewed in Module 1.
How it's tested
Expect architecture and troubleshooting scenarios: a job is slow, runs out of memory or produces no output, and you identify the Spark mechanism responsible. Other items ask which storage, scaling approach or cluster manager fits a requirement. The right answer usually names a specific mechanism, such as a shuffle, skew or lazy evaluation, rather than more hardware.
-
Explain the behavior
A Spark job behaves unexpectedly, such as running instantly or failing at the last step, and you explain it through lazy evaluation, actions, shuffles or the driver's memory.
-
Fix the slow job
A join or aggregation is slow and you choose the targeted remedy, such as a broadcast join, salting a skewed key, repartitioning or using a columnar format.
-
Choose the platform piece
A requirement such as fault-tolerant storage, elastic compute or running on Kubernetes is given, and you choose the storage, scaling model or cluster manager.
A notebook applies filter, select and groupBy to 2 TB of events and every cell finishes in under a second, then a final write takes forty minutes. Transformations in Spark are lazy: they only build a logical plan. The write is an action, so all the work, including the shuffle that groupBy requires, runs there. Nothing is wrong with the earlier cells; they simply did no computation.
Topics 3
Official topics from the Enterprise Big Data Framework Alliance outline, in outline order.
7.1 Distributed Cloud Computing ReviewCloud fundamentals, parallel and distributed computing, scalability, fault tolerance, distributed file systems and processing frameworks.
- What it means
- Cloud computing delivers on-demand compute and storage as IaaS, PaaS or SaaS with elastic scaling. Parallel computing splits work across processors that may share memory; distributed computing splits work across networked machines that coordinate by message. Distributed file systems such as HDFS, and cloud object stores, spread data across nodes, and frameworks such as MapReduce and Spark process it where it lives.
- What to understand
- Horizontal scaling (adding nodes) suits big data better than vertical scaling (a bigger machine), which has a ceiling. HDFS splits files into large blocks (128 MB by default) replicated three times by default across DataNodes, with a NameNode holding metadata. The CAP theorem says that during a network partition a distributed system must give up either consistency or availability. MapReduce writes intermediate results to disk between stages; Spark keeps them in memory where possible.
- How it shows up
- A cluster loses a node mid-job without data loss and you attribute it to block replication, or you choose horizontal scaling for unpredictable growth.
- Common confusion
- Reading the CAP theorem as 'pick any two at all times'; the tradeoff between consistency and availability applies when a partition occurs.
7.2 Apache Spark IntroductionSpark architecture, Spark data processing and machine learning with Spark.
- What it means
- A Spark application has a driver that builds an execution plan, a cluster manager that allocates resources, and executors that run tasks and cache data. Data is processed as resilient distributed datasets (RDDs) or, more commonly, DataFrames optimized by the Catalyst optimizer. MLlib provides distributed algorithms and a Pipeline API of transformers and estimators.
- What to understand
- Transformations (select, filter, join) are lazy and build a plan; actions (count, collect, write) trigger execution. Narrow transformations work within a partition; wide ones such as groupBy and join require a shuffle, which splits a job into stages. Each stage runs one task per partition. RDD lineage lets lost partitions be recomputed. In MLlib, an estimator's fit() returns a model, which is a transformer applied with transform().
- How it shows up
- A job is described with its operations and you identify where the shuffle and stage boundary occur, or which call actually triggers computation.
- Common confusion
- Thinking collect() is a harmless way to view results; it pulls the whole dataset to the driver and can crash it.
7.3 Spark DeploymentCluster management systems, infrastructure, monitoring and debugging, scaling and deployment practices.
- What it means
- Spark runs on a cluster manager such as Spark standalone, Hadoop YARN or Kubernetes, in client mode (driver on the submitting machine) or cluster mode (driver inside the cluster). Deployment involves sizing executors, choosing partition counts, monitoring with the Spark UI and scaling with dynamic allocation.
- What to understand
- The Spark UI shows jobs, stages, tasks, storage and executors; one stage with a few very slow tasks indicates data skew, and spill to disk indicates memory pressure. Fixes include broadcast joins for small tables, salting skewed keys, repartitioning to a few tasks per core, columnar formats such as Parquet with partition pruning, and the DataFrame API over Python UDFs. Dynamic allocation adds and removes executors with workload.
- How it shows up
- A stage shows 199 tasks finishing in seconds and one running for an hour, and you diagnose a skewed key rather than too little memory.
- Common confusion
- Treating every slow job as a resource shortage when the cause is skew or an avoidable shuffle.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Horizontal vs. vertical scaling
Horizontal scaling adds machines to a cluster; vertical scaling adds CPU or memory to one machine.
- Remember
- Big data platforms scale horizontally because single machines have hard limits and are single points of failure.
- Tested as
- Architecture choices for growing data.
- Trap
- Choosing a bigger server for data that will keep growing.
-
HDFS blocks and replication
HDFS splits files into large blocks, 128 MB by default, and stores three replicas by default on different DataNodes, tracked by the NameNode.
- Remember
- Replication gives fault tolerance and lets computation move to where data sits (data locality).
- Tested as
- Fault tolerance and storage questions.
- Trap
- Thinking the NameNode stores the data itself rather than metadata.
-
CAP theorem
A distributed data store cannot guarantee consistency, availability and partition tolerance all at once when a network partition happens.
- Remember
- Partitions are unavoidable in practice, so the real choice during one is consistency or availability.
- Tested as
- Design tradeoff questions for distributed stores.
- Trap
- Believing a system can choose to drop partition tolerance.
-
MapReduce vs. Spark
MapReduce writes intermediate data to disk between map and reduce stages; Spark keeps intermediate data in memory where possible and plans multi-stage jobs as a DAG.
- Remember
- Spark's advantage is greatest for iterative workloads such as machine learning.
- Tested as
- Framework-choice items.
- Trap
- Assuming Spark never uses disk; it spills when memory is short.
-
Driver, cluster manager and executors
The driver runs the main program and builds the plan, the cluster manager allocates resources, and executors run tasks and cache data.
- Remember
- collect() and toPandas() bring results to the driver, which has limited memory.
- Tested as
- Architecture and out-of-memory scenarios.
- Trap
- Increasing executor memory to fix a driver out-of-memory error.
-
Lazy evaluation
Transformations only add steps to a plan; nothing runs until an action such as count, collect or write.
- Remember
- Laziness lets the Catalyst optimizer reorder and prune the whole plan before execution.
- Tested as
- Why cells run instantly and the action is slow.
- Trap
- Timing transformations and concluding they are free.
-
Narrow vs. wide transformations
Narrow transformations (map, filter) need data from one partition; wide transformations (groupBy, join, distinct) need data from many partitions, causing a shuffle.
- Remember
- Shuffles create stage boundaries and are the most expensive part of most jobs.
- Tested as
- Stage counting and performance items.
- Trap
- Treating a join as cheap because it is one line of code.
-
RDD vs. DataFrame
RDDs are low-level immutable distributed collections; DataFrames add a schema and are optimized by Catalyst and the Tungsten engine.
- Remember
- Prefer DataFrames and built-in functions; Python UDFs bypass much of the optimization.
- Tested as
- API-choice and performance questions.
- Trap
- Assuming RDD code is faster because it is lower level.
-
MLlib Pipeline
A sequence of transformers (such as VectorAssembler) and estimators (such as LogisticRegression) fit together as one unit.
- Remember
- Calling fit() on an estimator returns a model, which is a transformer; pipelines keep preprocessing consistent between training and scoring.
- Tested as
- Machine learning with Spark items.
- Trap
- Fitting preprocessing outside the pipeline and leaking information.
-
Data skew and salting
Skew occurs when a few keys hold most of the data, so a few tasks do most of the work; salting adds a random suffix to spread a hot key across partitions.
- Remember
- Skew shows as a few straggler tasks in one stage of the Spark UI.
- Tested as
- Troubleshooting scenarios.
- Trap
- Adding executors, which leaves the single hot task just as slow.
-
Broadcast join
Sends a copy of a small table to every executor so the large table is joined without being shuffled.
- Remember
- Spark broadcasts automatically below a size threshold (10 MB by default) and can be hinted to broadcast larger small tables.
- Tested as
- Slow-join scenarios with one small table.
- Trap
- Broadcasting a table too large to fit in executor memory.
-
Client vs. cluster deploy mode
In client mode the driver runs on the submitting machine; in cluster mode it runs inside the cluster.
- Remember
- Cluster mode suits production jobs because the driver does not depend on a user's laptop or session.
- Tested as
- Deployment questions.
- Trap
- Running scheduled production jobs in client mode from a workstation.
-
Parquet and partition pruning
Parquet is a columnar, compressed format; partitioned layouts let Spark skip folders that a filter excludes.
- Remember
- Columnar storage reads only needed columns, and predicate pushdown skips irrelevant data.
- Tested as
- Storage-format and performance questions.
- Trap
- Storing large analytical datasets as CSV.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Transformations vs. actions
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Transformation | Defines a new DataFrame or RDD, such as select, filter, join or groupBy | Lazy: builds the plan, runs nothing | Expecting errors or costs to appear when it is defined |
| Action | Returns a result or writes output, such as count, collect, show or write | Triggers execution of the whole plan | Using collect on large data and overloading the driver |
Narrow vs. wide transformations
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Narrow | Each output partition depends on one input partition, such as map, filter or select | No shuffle; pipelined within a stage | Assuming every transformation is equally cheap |
| Wide | Output depends on many input partitions, such as groupBy, join or distinct | Requires a shuffle and creates a stage boundary | Missing that a join on large tables shuffles both sides |
Spark standalone vs. YARN vs. Kubernetes
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Standalone | Spark's built-in simple cluster manager | Easy to set up; Spark-only clusters | Choosing it for a shared multi-framework cluster |
| YARN | Hadoop's resource manager | Shares resources with other Hadoop workloads near HDFS data | Assuming it is required to run Spark |
| Kubernetes | Container orchestration platform running Spark drivers and executors as pods | Fits containerized, cloud-native environments | Forgetting images and resource limits must be configured |
MapReduce vs. Spark
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| MapReduce | Two-phase map and reduce model writing intermediate data to disk | Disk-based between stages; slow for iterative jobs | Choosing it for iterative machine learning |
| Spark | DAG-based engine with in-memory processing | Fast iterative and interactive work; unified batch, SQL, streaming and ML | Assuming everything must fit in memory |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
More executors for skew
- What candidates pick
- Double the number of executors because one stage takes an hour.
- Why it looks right
- More parallelism usually speeds up Spark jobs.
- Why it's wrong
- If one key holds most of the data, a single task processes it regardless of executor count; salting the key or handling it separately spreads the work.
Clue to watch for Most tasks in the stage finish quickly while one or a few run far longer.
-
Executor memory for a driver crash
- What candidates pick
- Increase executor memory after an out-of-memory error during collect().
- Why it looks right
- Executors do most of the work, so they seem the likely bottleneck.
- Why it's wrong
- collect() moves the entire result to the driver; the fix is to avoid collecting large data, write it out or sample it.
Clue to watch for The failure happens at collect(), toPandas() or another call that returns data to the driver.
-
Shuffle the large table
- What candidates pick
- Repartition a 3 TB table before joining it to a 5 MB lookup table.
- Why it looks right
- Repartitioning is a standard performance step.
- Why it's wrong
- Repartitioning adds a shuffle; broadcasting the small table to every executor avoids shuffling the large one entirely.
Clue to watch for One side of the join is small enough to fit comfortably in memory.
-
Instant cells mean fast code
- What candidates pick
- Conclude the transformations are efficient because each cell runs in milliseconds.
- Why it looks right
- The timing appears to measure the work.
- Why it's wrong
- Transformations are lazy; the cost appears at the first action, where the whole plan, including shuffles, executes.
Clue to watch for Only defining cells are fast and a later count, show or write is slow.
-
Vertical scaling for growth
- What candidates pick
- Buy the largest available server for data growing 50% a year.
- Why it looks right
- It avoids distributed complexity.
- Why it's wrong
- A single machine has a ceiling and is a single point of failure; horizontal scaling on a distributed platform grows with the data.
Clue to watch for Sustained, open-ended data growth or a need for fault tolerance.
-
Python UDF for everything
- What candidates pick
- Rewrite column logic as Python UDFs for readability.
- Why it looks right
- Python code is familiar and flexible.
- Why it's wrong
- Row-at-a-time Python UDFs move data between the JVM and Python and bypass Catalyst optimization; built-in functions or vectorized pandas UDFs are faster.
Clue to watch for A built-in Spark SQL function could express the same logic.
Answer framework
Use these steps to diagnose distributed processing scenarios on the exam.
- Identify where the work actually runs: which call is an action, which operations shuffle data, and what returns results to the driver's limited memory.
- Locate the symptom in the Spark UI terms the scenario gives you, such as one slow task, spill to disk, many tiny tasks or a driver error.
- Match the symptom to its mechanism: data skew, an avoidable shuffle, driver overload, too few or too many partitions, or an inefficient storage format.
- Choose the targeted fix, such as salting a hot key, a broadcast join, avoiding collect(), repartitioning or switching to Parquet with partition pruning.
- Add hardware or executors only when the work is evenly spread and still limited by available resources.
Practice question
Want more like this? The EBDS practice test has questions for every domain.
Checklist
0 of 13
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Start with topic 7.1 to fix the vocabulary: draw HDFS with a NameNode, DataNodes and replicated blocks, and write one sentence each on scaling, fault tolerance and the CAP theorem. This review is quick for most candidates but supplies the reasoning for later questions.
Spend most of your time on topic 7.2 with a hands-on Spark environment, even a local one. Chain transformations, time them, then trigger an action and watch the Spark UI create jobs, stages and tasks. Run a groupBy and a join and find the shuffle. Build a small MLlib pipeline so the estimator and transformer roles become concrete.
Topic 7.3 is the most underestimated area because it sounds operational. Deliberately create skew with one hot key, see the straggler task, then fix it with salting; compare a broadcast join against a shuffle join; and note what a driver out-of-memory error looks like. Afterwards, practice troubleshooting scenarios under time pressure, then move to Module 8, where deep learning adds its own compute demands.
FAQ
Does a data scientist really need Spark for the EBDS exam?
Yes. The outline gives a full module to distributed systems, including Spark architecture, data processing, machine learning with Spark and deployment.
How many questions cover distributed systems?
No module weights are published. An even split of the 80 questions suggests about nine, as a planning estimate only.
What is the difference between a transformation and an action?
A transformation such as filter or join defines a new dataset lazily. An action such as count, collect or write triggers Spark to execute the plan.
Why are shuffles expensive?
A shuffle redistributes data across the network so that records with the same key end up together, which involves serialization, disk and network I/O, and it splits the job into stages.
How do I recognize data skew?
In the Spark UI, a stage where most tasks finish quickly but a few take much longer usually indicates that some keys hold far more data than others.
Which cluster manager should I know?
Know Spark standalone, Hadoop YARN and Kubernetes at a conceptual level: what each is, when it fits, and how client and cluster deploy modes differ.
How does this module connect to the rest of the course?
It is where the models from Modules 3 to 5 run on big data, and it reflects the architecture element of the Enterprise Big Data Framework reviewed in Module 1.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 7 knowledge
You have covered distributed cloud computing, Spark's architecture and processing model, MLlib and Spark deployment. Check whether you can diagnose distributed processing scenarios accurately under exam timing.