Spark Dev Assoc · Domain 6 of 7

Spark Dev Assoc Domain 6: Using Spark Connect to Deploy Applications

How Spark Connect separates client applications from the Spark driver, what it supports and leaves out, and how client, cluster and local deployment modes decide where the driver runs.

Exam weight
5%
Official objectives
2
Estimated questions estimate
~2
Suggested study time estimate
3–5 hrs

About 10 min read Reviewed October 2026

Overview

Domain 6 is a small section, 5% of the exam, with two official objectives. The first asks you to describe the features of Spark Connect, the client-server protocol introduced in Spark 3.4 that lets a thin client send DataFrame operations to a remote Spark server. The second asks you to describe the deployment modes in Apache Spark: client, cluster and local.

Expect only a couple of questions, mostly definitional. You should know how a client connects with a remote URL, what travels over the connection, what the decoupled design improves, and which classic APIs are unavailable. For deployment modes, the key fact is where the driver process runs in each case and what that means for logs, failures and the machine submitting the job.

How it's tested

Questions here usually describe a requirement or behaviour and ask you to name the feature or mode. For Spark Connect, focus on the client-server split, the remote connection string and unsupported APIs. For deployment modes, focus on where the driver runs and what happens when the submitting machine disconnects.

  • Identify the Connect benefit

    A scenario mentions remote IDE development, isolation or independent upgrades, and you recognise Spark Connect as the enabling feature.

  • Spot the unsupported call

    Code using a Connect session calls an RDD or SparkContext method, and you identify why it fails.

  • Place the driver

    A deployment description is given, and you decide whether it is client, cluster or local mode.

Reasoning it through

A team submits a nightly job from a laptop, and the job fails whenever the laptop sleeps. In client mode the driver runs on the submitting machine, so losing that machine ends the application. Submitting with cluster deploy mode places the driver on a node inside the cluster, so the job keeps running after the laptop disconnects. Cluster mode is therefore the correct answer.

Objectives 2

Official objectives from the Databricks outline, in outline order.

6.1 Describe the features of Spark ConnectDecoupled client-server architecture for Spark applications.
What it means
This objective covers Spark Connect, which separates the client from the Spark driver. The client builds unresolved logical plans from DataFrame code and sends them over gRPC, encoded as protocol buffers, to a Spark Connect server. The server analyses, optimises and runs the plan, then streams results back as Apache Arrow batches.
What to understand
You connect with SparkSession.builder.remote('sc://host:15002').getOrCreate(). Benefits include stability, since a client crash cannot take down the shared driver; independent upgrades of client and server; lightweight clients for IDEs and notebooks; and support for several languages. The DataFrame and SQL APIs are supported, but RDD operations and direct SparkContext access are not. Databricks Connect is built on Spark Connect.
How it shows up
Questions ask what Spark Connect sends, how to open a remote session, or which API cannot be used through it.
Common confusion
Thinking the client runs the query locally; execution happens on the remote server.
6.2 Describe the different deployment mode types (Client, Cluster, Local) in the Apache Spark environmentWhere the driver runs in client, cluster and local modes.
What it means
This objective covers how an application is launched. In client mode the driver runs on the machine that submitted the application, while executors run in the cluster. In cluster mode the driver is launched on a node inside the cluster. In local mode the driver and executor work run in a single JVM on one machine, set with a master such as local[*].
What to understand
Client mode, the default for spark-submit, suits interactive work because output appears on the submitting machine, but the job dies if that machine disconnects. Cluster mode suits production jobs and keeps driver traffic close to executors. Local mode is for development and tests; local[4] uses four threads and local[*] uses all available cores. Standalone clusters do not support cluster mode for Python applications.
How it shows up
A scenario describes where code runs or a failure caused by a disconnect, and you identify the mode.
Common confusion
Confusing local mode with client mode; client mode still uses remote executors.

High-yield concepts

The terms, rules and formulas this domain turns into questions most often.

  • Spark Connect

    Client-server protocol that sends DataFrame plans to a remote Spark server.

    Remember
    Introduced in Spark 3.4.
    Tested as
    Feature identification questions.
    Trap
    Assuming it is a Databricks-only product.

  • sc:// connection string

    URL scheme used by SparkSession.builder.remote().

    Remember
    Default port is 15002.
    Tested as
    Session creation snippets.
    Trap
    Using http:// or spark:// instead.

  • Unresolved logical plan

    What the Connect client sends; the server resolves and optimises it.

    Remember
    Analysis errors surface on the server when the plan is analysed.
    Tested as
    Architecture questions.
    Trap
    Thinking the client optimises the query.

  • gRPC and Arrow

    Transport and result format used by Spark Connect.

    Remember
    Plans go out as protobuf; results return as Arrow batches.
    Tested as
    Protocol questions.
    Trap
    Expecting JDBC as the transport.

  • No RDD or SparkContext

    Spark Connect sessions expose DataFrame and SQL APIs only.

    Remember
    Broadcast variables and accumulators need classic mode.
    Tested as
    Unsupported API questions.
    Trap
    Calling spark.sparkContext on a remote session.

  • Client deploy mode

    Driver runs on the submitting machine.

    Remember
    Default for spark-submit.
    Tested as
    Deployment questions.
    Trap
    Using it for jobs submitted from laptops.

  • Cluster deploy mode

    Driver runs inside the cluster.

    Remember
    Survives the submitter disconnecting.
    Tested as
    Deployment questions.
    Trap
    Assuming it is available for Python on standalone clusters.

  • Local mode

    Driver and tasks run in one JVM on one machine.

    Remember
    local[*] uses all cores.
    Tested as
    Deployment questions.
    Trap
    Treating it as a production option.

  • Databricks Connect

    Databricks client library for running code from IDEs on Databricks compute.

    Remember
    Built on Spark Connect.
    Tested as
    Context for Connect features.
    Trap
    Thinking it embeds a full local cluster.

Side by side

Concepts candidates mix up, compared on the point that decides the answer.

Classic Spark session vs Spark Connect session

ConceptMeaningKey differenceCommon trap
Classic session Client code runs in the driver process Full API including RDDs and SparkContext Assuming remote IDE access is built in
Spark Connect session Thin client sends plans to a remote server DataFrame and SQL only; isolated from driver Calling RDD methods

Deployment modes

ConceptMeaningKey differenceCommon trap
Client Driver on submitting machine Interactive output; fails if submitter disconnects Using for unattended production jobs
Cluster Driver on a cluster node Resilient to submitter loss Expecting console output locally
Local Everything in one JVM No cluster needed Confusing it with client mode

Exam traps

Each trap is an answer that looks right. Learn the clue that should change your mind.

  1. Connect runs locally

    What candidates pick
    Say a Spark Connect client executes the DataFrame operations.
    Why it looks right
    The code is written and run on the client machine.
    Why it's wrong
    The client only builds and sends plans; the remote server analyses, optimises and executes them.

    Clue to watch for The question asks where execution happens.

  2. RDDs over Connect

    What candidates pick
    Use spark.sparkContext.parallelize with a remote session.
    Why it looks right
    It works in classic notebooks.
    Why it's wrong
    Spark Connect exposes DataFrame and SQL APIs only, so SparkContext and RDD calls are unsupported.

    Clue to watch for The session was created with builder.remote.

  3. Client mode is local

    What candidates pick
    Describe client mode as running everything on one machine.
    Why it looks right
    The driver is on the submitting machine.
    Why it's wrong
    In client mode executors still run on cluster nodes; only the driver is local to the submitter.

    Clue to watch for Executors are described on worker nodes.

  4. Cluster mode for console output

    What candidates pick
    Choose cluster mode to see printed results in the submitting terminal.
    Why it looks right
    Cluster mode sounds more capable.
    Why it's wrong
    In cluster mode the driver runs remotely, so its output goes to the cluster's driver logs.

    Clue to watch for The requirement is interactive output.

  5. Connect is a new language

    What candidates pick
    Describe Spark Connect as a separate API to learn.
    Why it looks right
    It has a new name and protocol.
    Why it's wrong
    Developers keep using the familiar DataFrame API; Spark Connect changes where and how it executes.

    Clue to watch for Options describe new syntax versus a new architecture.

Answer framework

Apply these steps to Spark Connect and deployment mode questions to land on the expected answer.

  1. Decide first whether the question is about the Spark Connect client-server protocol or about where the driver process runs in a deployment.
  2. For Spark Connect, remember the client builds unresolved plans, sends them over gRPC and receives Arrow results from the server.
  3. Check any code for RDD, SparkContext, broadcast variable or accumulator calls, since these are unsupported in Connect sessions.
  4. For deployment questions, locate the driver: submitting machine for client, a cluster node for cluster, one JVM for local.
  5. Match the scenario's needs, such as interactive output, resilience to disconnects or quick testing, to the mode or feature that provides them.

Practice question

A developer working in a local IDE creates a session with SparkSession.builder.remote('sc://spark-host:15002').getOrCreate() and then runs df.groupBy('country').count().show() on a large table. Which statement correctly describes what happens?

Want more like this? The Spark Dev Assoc practice test has questions for every domain.

Checklist

0 of 10

Tick each one you can do without notes. Your ticks stay in this browser.

Study strategy

Treat this section as a short set of facts to learn precisely rather than a large practical topic. Read the Apache Spark documentation overview of Spark Connect and sketch the flow: client code, unresolved plan, gRPC, server analysis and execution, Arrow results. Note the connection string format and the list of unsupported APIs.

If you can, try Databricks Connect or a local Spark Connect server and run a simple DataFrame query from an IDE, then try calling spark.sparkContext to see the error. That single quick experiment makes the main exam point about unsupported APIs memorable.

For deployment modes, write a three-row table showing where the driver runs, where executors run and the typical use for client, cluster and local. Add the note that standalone clusters do not offer cluster mode for Python applications. Review the table before mock exams, and spend most of your remaining time on the heavier DataFrame and architecture sections.

FAQ

How much of the exam is Using Spark Connect to Deploy Applications?

Using Spark Connect to Deploy Applications is 5% of the scored exam, about two questions, from two official objectives.

What is Spark Connect?

A decoupled client-server architecture, added in Spark 3.4, where a thin client sends DataFrame plans to a remote Spark server that executes them and returns results.

Can I use RDDs with Spark Connect?

No. Spark Connect supports the DataFrame and SQL APIs. RDD operations and direct SparkContext access require a classic Spark session.

How do I start a Spark Connect session in Python?

Call SparkSession.builder.remote('sc://host:port').getOrCreate(). The default server port is 15002.

What is the difference between client and cluster deploy mode?

In client mode the driver runs on the submitting machine. In cluster mode it runs on a node inside the cluster, so the job continues if the submitter disconnects.

When should I use local mode?

For development, unit tests and small experiments on one machine, where driver and tasks share one JVM without a cluster.

Other Data & Analytics certifications

Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.

Test your Domain 6 knowledge

See whether you can explain Spark Connect's client-server flow and pick the right deployment mode for a scenario under exam timing.

Take the Spark Dev Assoc practice test Check your Spark Dev Assoc readiness

Keep going