Overview
Data Processing covers the preparation work that happens between loading a table and fitting a model. Its nine objectives run in a natural order. First you profile the data, computing summary statistics with DataFrame.summary() or dbutils.data.summarize and plotting categorical and continuous features. Then you compare pairs of features with the right method, such as correlation for two continuous columns or a contingency table and chi-square test for two categorical ones. Cleaning follows: removing outliers with a standard deviation or interquartile range rule, and filling missing values with the mean, median or mode, including when each choice is sensible. The last objectives cover encoding and scaling, namely one-hot encoding categorical columns, knowing which models need it, and spotting skewed targets or features where a log transform helps.
Most questions are code-aware but conceptual: which Spark ML class or DataFrame method fits, and which statistical choice suits the data described.
How it's tested
Section 2 questions usually describe a column or pair of columns, including type, distribution, missing values and cardinality, and ask which preparation step or statistic fits. Some show a short PySpark snippet using summary, approxQuantile, Imputer or OneHotEncoder. Read the data description closely, because skew, cardinality and model type decide the answer.
-
Match the statistic to the data
A distribution is described, and you choose mean or median imputation, a standard deviation or IQR rule, or a log transform to suit it.
-
Encoding fit
A model type and a categorical column are described, and you decide whether one-hot encoding is needed, harmful or unnecessary.
-
Right comparison method
Two features are named with their types, and you pick correlation, a chi-square test, a contingency table or a suitable plot.
A house_price column has a long right tail and 4% missing values. Mean imputation would pull fill values toward a few very expensive homes, so the median is the safer central value for skewed data. If price is also the regression target, a log transform makes its distribution closer to normal, and predictions must be exponentiated back before reporting errors in currency.
Objectives 9
Official objectives from the Databricks outline, in outline order.
2.1 Compute summary statistics on a Spark DataFrame using .summary() or dbutils data summariesProfiling a DataFrame with built-in statistics.
- What it means
- This covers producing descriptive statistics for Spark DataFrame columns with DataFrame.summary() or describe(), or with dbutils.data.summarize(df), which renders an interactive profile in a notebook.
- What to understand
- summary() returns count, mean, stddev, min, the 25%, 50% and 75% approximate percentiles, and max by default; describe() omits the percentiles. You can pass specific statistics, such as summary("count", "mean", "50%"). dbutils.data.summarize adds histograms, missing and zero counts, and frequent values per column. The notebook's Data Profile tab on a displayed DataFrame gives the same view.
- How it shows up
- A question asks which call returns quartiles, or which tool gives a visual per-column profile.
- Common confusion
- Expecting describe() to include percentiles, which only summary() provides by default.
2.2 Remove outliers from a Spark DataFrame based on standard deviation or IQRFiltering extreme values with two common rules.
- What it means
- Outlier removal filters rows whose values fall far from the bulk of the distribution, using either a standard deviation rule or the interquartile range.
- What to understand
- With the standard deviation rule, compute mean and stddev and keep rows within a band such as mean plus or minus three standard deviations; it suits roughly normal data. With IQR, get Q1 and Q3, for example via approxQuantile, compute IQR = Q3 - Q1, and keep values between Q1 - 1.5*IQR and Q3 + 1.5*IQR. IQR is more robust for skewed data because quartiles ignore extremes.
- How it shows up
- Expect a fragment that computes bounds with approxQuantile and asks which filter expression completes it.
- Common confusion
- Using the standard deviation rule on heavily skewed data, where the outliers inflate the deviation itself.
2.3 Create visualizations for categorical or continuous featuresChoosing plots by feature type.
- What it means
- This covers picking a suitable chart to explore a single feature, depending on whether it is categorical or continuous.
- What to understand
- For categorical features use bar charts of counts or frequencies. For continuous features use histograms or density plots to see shape and skew, and box plots to see spread and outliers. In Databricks, display() offers built-in chart types, and libraries such as matplotlib, seaborn or plotly work on a pandas sample. Large Spark DataFrames should be aggregated or sampled before plotting locally.
- How it shows up
- A feature type is given and you choose the chart that reveals its distribution.
- Common confusion
- Plotting a continuous variable as a bar chart of every distinct value.
2.4 Compare two categorical or two continuous features using the appropriate methodPairwise comparison methods by data type.
- What it means
- This objective asks how to examine the relationship between two features of the same type.
- What to understand
- Two continuous features are compared with a scatter plot and a correlation coefficient, Pearson for linear relationships or Spearman for monotonic ones; Spark offers df.stat.corr and pyspark.ml.stat.Correlation for matrices. Two categorical features are compared with a contingency table, using df.stat.crosstab, a stacked or grouped bar chart, and a chi-square test of independence through pyspark.ml.stat.ChiSquareTest.
- How it shows up
- Given the types of two columns, choose correlation or a chi-square test and the matching chart.
- Common confusion
- Computing Pearson correlation on label-encoded categories, which produces a meaningless number.
2.5 Compare and contrast imputing missing values with the mean or median or mode valueChoosing an imputation statistic.
- What it means
- Imputation replaces missing values with a representative value. This objective compares the three simple choices and when each is appropriate.
- What to understand
- The mean suits roughly symmetric numeric data without strong outliers but is pulled by extreme values. The median is robust to skew and outliers, so it suits income, prices and durations. The mode is the only one of the three that works for categorical columns and can also fit discrete numeric data. All three shrink variance, and adding a missing-indicator column can preserve the signal that a value was absent.
- How it shows up
- A described distribution or column type decides which statistic the question wants.
- Common confusion
- Using the mean for a skewed column, or trying to compute a mean for a categorical one.
2.6 Impute missing values with the mode, mean, or median valueApplying imputation in Spark.
- What it means
- This covers carrying out imputation in code, mainly with pyspark.ml.feature.Imputer or DataFrame methods such as fillna.
- What to understand
- Imputer takes inputCols, outputCols and a strategy of "mean", "median" or "mode"; it is an estimator, so you fit it on training data and then transform. It works on numeric columns. For categorical strings, compute the most frequent value with a groupBy count and fill with fillna. Fit imputation statistics on the training split only, to avoid leaking test information.
- How it shows up
- Look for the strategy parameter, or for fitting the imputer before splitting the data.
- Common confusion
- Fitting the Imputer on the full dataset, which leaks test-set statistics into training.
2.7 Use one-hot encoding for categorical featuresEncoding categories as binary vectors in Spark ML.
- What it means
- One-hot encoding converts a categorical column into binary indicators, one per category, so models that need numbers can use it without implying an order.
- What to understand
- In Spark ML, StringIndexer first maps strings to numeric indices, then OneHotEncoder turns indices into sparse vectors, usually followed by VectorAssembler. Both StringIndexer and OneHotEncoder are estimators. Set handleInvalid="keep" to cope with categories unseen during training. dropLast defaults to true, removing one redundant column. pandas users would use get_dummies, but the exam frames this in Spark.
- How it shows up
- A pipeline fragment is missing a stage, often StringIndexer before OneHotEncoder.
- Common confusion
- Feeding raw strings to OneHotEncoder, which expects numeric indices.
2.8 Identify and explain the model types or data sets for which one-hot encoding is or is not appropriate.When to one-hot encode and when not to.
- What it means
- This objective asks when one-hot encoding helps and when it hurts, depending on the algorithm and on the cardinality of the column.
- What to understand
- Linear and logistic regression, support vector machines, k-means and neural networks need numeric, unordered inputs, so one-hot encoding fits them. Tree-based models can split on indexed categories directly, and very high-cardinality columns such as ZIP codes or user IDs create huge sparse matrices that slow training and can harm tree performance. Alternatives include index encoding, target or frequency encoding, hashing, or grouping rare categories.
- How it shows up
- A tree model with a column of thousands of categories signals that one-hot encoding is a poor choice.
- Common confusion
- Assuming every categorical column should be one-hot encoded regardless of model type or cardinality.
2.9 Identify scenarios where log scale transformation is appropriateUsing log transforms on skewed values.
- What it means
- A log transform replaces a value x with log(x) or log(x+1) to compress large values and reduce right skew.
- What to understand
- It helps when a positive variable spans orders of magnitude or has a long right tail, such as prices, incomes or counts, and when relationships are multiplicative. Applied to a regression target, it can stabilise variance and reduce the influence of extreme values. log1p handles zeros. It cannot be applied to negative values without shifting. Predictions on a log target must be converted back before interpretation.
- How it shows up
- A right-skewed, strictly positive column or target is the trigger for a log transform.
- Common confusion
- Applying a log to data with zeros or negatives, or to a roughly symmetric distribution.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
DataFrame.summary()
Descriptive statistics including quartiles.
- Remember
- count, mean, stddev, min, 25%, 50%, 75%, max.
- Tested as
- Profiling questions.
- Trap
- Expecting quartiles from describe().
-
dbutils.data.summarize
Interactive per-column data profile.
- Remember
- Histograms, nulls and frequent values.
- Tested as
- Profiling tool questions.
- Trap
- Thinking it returns a DataFrame for modelling.
-
approxQuantile
Approximate percentiles on a Spark column.
- Remember
- Used to get Q1 and Q3 for IQR.
- Tested as
- Outlier code questions.
- Trap
- Forgetting it is approximate with a relative error setting.
-
IQR rule
Keep values within 1.5 IQR of the quartiles.
- Remember
- Robust to skew.
- Tested as
- Outlier questions.
- Trap
- Using stddev bounds on skewed data.
-
Correlation
Strength of association between continuous features.
- Remember
- Pearson linear, Spearman monotonic.
- Tested as
- Feature comparison questions.
- Trap
- Applying it to categorical codes.
-
Chi-square test
Tests independence of two categorical features.
- Remember
- Uses a contingency table.
- Tested as
- Feature comparison questions.
- Trap
- Using it for continuous pairs.
-
Median imputation
Fill missing values with the median.
- Remember
- Robust choice for skewed numeric data.
- Tested as
- Imputation questions.
- Trap
- Choosing the mean on skewed columns.
-
Mode imputation
Fill with the most frequent value.
- Remember
- Works for categorical columns.
- Tested as
- Imputation questions.
- Trap
- Trying mean on strings.
-
Imputer
Spark ML estimator for mean, median or mode.
- Remember
- Fit on training data only.
- Tested as
- Imputation code questions.
- Trap
- Fitting before the split.
-
StringIndexer + OneHotEncoder
Index categories, then encode as vectors.
- Remember
- Indexer must come first.
- Tested as
- Encoding pipeline questions.
- Trap
- Skipping the indexer.
-
High cardinality
A column with very many distinct values.
- Remember
- One-hot creates huge sparse vectors.
- Tested as
- Encoding suitability questions.
- Trap
- One-hot encoding IDs for trees.
-
Log transform
Compresses right-skewed positive values.
- Remember
- Use log1p when zeros exist.
- Tested as
- Transformation questions.
- Trap
- Forgetting to invert predictions.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Mean vs. median vs. mode imputation
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Mean | Arithmetic average | Fine for symmetric numeric data | Distorted by outliers |
| Median | Middle value | Robust for skewed numeric data | Assuming it suits categories |
| Mode | Most frequent value | Works for categorical data | Using it on continuous values with no repeats |
Standard deviation vs. IQR outlier rule
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Standard deviation | Mean plus or minus k stddev | Assumes roughly normal data | Outliers inflate the bounds |
| IQR | Q1 - 1.5 IQR to Q3 + 1.5 IQR | Quartile-based and robust | Forgetting approxQuantile is approximate |
Comparing two features
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Two continuous | Scatter plot and correlation | Pearson or Spearman coefficient | Ignoring non-linear patterns |
| Two categorical | Contingency table and chi-square | Tests independence | Correlating label-encoded codes |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Mean for skewed data
- What candidates pick
- Impute a right-skewed income column with the mean.
- Why it looks right
- The mean is the default statistic people reach for.
- Why it's wrong
- A few large incomes pull the mean upward, so fills overstate typical values; the median resists that skew.
Clue to watch for The column is described as skewed or containing extreme values.
-
Pearson on categories
- What candidates pick
- Compute correlation between two label-encoded categorical columns.
- Why it looks right
- It produces a tidy single number.
- Why it's wrong
- Index codes are arbitrary, so their correlation is meaningless; a contingency table and chi-square test compare categorical features properly.
Clue to watch for Both features are categorical.
-
Stddev rule on skew
- What candidates pick
- Drop rows beyond three standard deviations in a long-tailed column.
- Why it looks right
- It is a familiar rule of thumb.
- Why it's wrong
- Extreme values inflate the standard deviation, so the bounds become too wide; IQR bounds rely on quartiles and stay stable.
Clue to watch for The data is skewed or heavy-tailed.
-
One-hot everything
- What candidates pick
- One-hot encode a 20,000-value merchant ID for a random forest.
- Why it looks right
- Encoding categories is standard practice.
- Why it's wrong
- It creates a huge sparse feature space that slows training and dilutes splits; tree models can use indexed or grouped categories instead.
Clue to watch for High cardinality plus a tree-based model.
-
Encoder without indexer
- What candidates pick
- Pass a string column directly into OneHotEncoder.
- Why it looks right
- The name suggests it handles categories end to end.
- Why it's wrong
- Spark's OneHotEncoder expects numeric category indices, so StringIndexer must run first in the pipeline.
Clue to watch for The input column holds strings.
-
Fit before split
- What candidates pick
- Fit Imputer on the full dataset, then split into train and test.
- Why it looks right
- It seems efficient to clean everything once.
- Why it's wrong
- Test-set statistics leak into the fill values, making evaluation optimistic; fit on training data and apply to test.
Clue to watch for The order of split and fit in the code.
Answer framework
Apply these steps to any Section 2 data preparation scenario.
- Identify each column's type and shape: categorical or continuous, cardinality, skew, presence of zeros or negatives, and share of missing values.
- Profile first with summary() for quartiles or dbutils.data.summarize for a visual profile, and choose histograms, box plots or bar charts by feature type.
- For outliers, use the IQR rule on skewed data and the standard deviation rule only when the distribution is close to normal.
- For missing values, use the median for skewed numeric data, the mean for symmetric numeric data and the mode for categorical columns, fitting on training data only.
- For encoding and scaling, one-hot encode low-cardinality categories for linear models, avoid it for high-cardinality columns on trees, and log-transform positive right-skewed values.
Practice question
Want more like this? The Databricks ML Assoc practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
This section rewards a small set of clear rules, so learn them as decisions tied to data shapes. Load a public tabular dataset with skewed numeric columns, missing values and a few categorical fields into a Databricks notebook. Run summary() and describe() side by side to see which returns quartiles, then open dbutils.data.summarize and the Data Profile tab.
Next, implement both outlier rules on the same skewed column and compare how many rows each removes. Fit Imputer with mean, median and mode strategies, and note how the median changes less when you add a few extreme values. Practise the StringIndexer, OneHotEncoder and VectorAssembler sequence, setting handleInvalid, and confirm that OneHotEncoder fails on raw strings.
Then build a short cheat list: chart by feature type, comparison method by type pair, imputation statistic by distribution, encoding by model type and cardinality, and log transform for positive skewed data. Rehearse it with timed questions, then continue to Section 3, Model Development, where these choices affect metrics.
FAQ
How much of the Databricks ML Associate exam is Data Processing?
Data Processing is 19% of the exam, about nine of the 48 scored questions, and covers profiling, cleaning, imputation, encoding and log transforms.
What is the difference between summary() and describe()?
Both return count, mean, stddev, min and max. summary() also returns the 25%, 50% and 75% approximate percentiles by default, and lets you request specific statistics.
When should I use the median rather than the mean to impute?
When the numeric column is skewed or has outliers, such as income or price. The median is not pulled by extreme values, so fills stay representative.
Can Spark's Imputer fill categorical strings?
Imputer works on numeric columns. For string categories, find the most frequent value and fill it with fillna, or index the column first and then apply the mode strategy.
Do tree-based models need one-hot encoding?
Generally not. Trees can split on indexed categories, and one-hot encoding high-cardinality columns slows training and can weaken splits. Linear models and neural networks do need it.
Which outlier rule should I choose?
Use the IQR rule for skewed or heavy-tailed data and the standard deviation rule when the distribution is close to normal.
How does Data Processing connect to Model Development?
The imputation, encoding and log choices made here become pipeline stages in model development and change the evaluation metrics you interpret there.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 2 knowledge
Practise choosing statistics, outlier rules, imputation strategies and encodings for described columns in short, timed question sets.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Databricks ML Assoc AI tutor Get any concept explained your way