Overview
Model Development carries 31% of the exam and has 15 objectives, making it the section where general machine learning knowledge and Spark ML practice meet. It opens with choosing an algorithm for a problem and mitigating class imbalance. It then moves into Spark ML structure: the difference between estimators and transformers and how a Pipeline chains them. Tuning objectives cover Hyperopt's fmin with a search space and algorithm, the trade-offs between grid, random and Bayesian search, and parallelizing single-node models such as scikit-learn during tuning. Validation objectives compare cross-validation with a single train-validation split, perform cross-validation inside fitting, and ask you to count how many models a grid search with k folds trains. The final group covers classification and regression metrics, choosing one for a business goal, exponentiating predictions from a log-transformed target, and how model complexity drives bias and variance.
Expect both calculation-style and judgement-style questions in this section.
How it's tested
Section 3 mixes scenario judgement with small calculations. Scenarios describe a prediction problem, a dataset's balance, or a business cost of errors and ask for the algorithm, technique or metric. Calculation items give grid sizes and fold counts, or a log-scale model's output. Code items show Spark ML or Hyperopt fragments and ask what they produce.
-
Count the models
A parameter grid and fold count are given, and you calculate how many models are trained, including the final refit.
-
Pick the metric
A business objective and error costs are described, and you choose among F1, AUC, log loss, RMSE, MAE or R-squared.
-
Estimator or transformer
A Spark ML class is named, and you decide whether it learns from data with fit or simply maps a DataFrame with transform.
A fraud dataset has 0.5% positive cases, and a model reports 99.5% accuracy. That figure equals predicting no fraud for every row, so it says nothing useful. Because missed fraud is costly, the right measures are recall, precision and F1 on the positive class, or area under the precision-recall curve, and imbalance should be handled with class weights or resampling during training.
Objectives 15
Official objectives from the Databricks outline, in outline order.
3.1 Use ML foundations to select the appropriate algorithm for a given model scenarioMatching problem type and constraints to an algorithm.
- What it means
- This objective tests whether you can pick a suitable algorithm family from a description of the target, the data and the constraints.
- What to understand
- First identify the task: regression for a continuous target, classification for categories, clustering when there is no label. Then weigh constraints. Linear and logistic regression are fast and interpretable. Decision trees are interpretable and handle non-linearity. Random forests and gradient-boosted trees such as XGBoost or LightGBM usually lead on tabular data. K-means groups unlabeled rows. Interpretability, data size and latency requirements narrow the choice.
- How it shows up
- A scenario names the target type and one constraint, such as explainability, and you choose the algorithm.
- Common confusion
- Choosing a classifier for a numeric target, or clustering when labels exist.
3.2 Identify methods to mitigate data imbalance in training dataHandling rare classes.
- What it means
- Data imbalance occurs when one class is far rarer than another, so a model can score high accuracy by ignoring the minority class.
- What to understand
- Mitigations include undersampling the majority class, oversampling the minority class, synthetic oversampling such as SMOTE, class weights through parameters like weightCol in Spark ML or class_weight in scikit-learn, and stratified splits that keep class ratios in each fold. Resample only the training data. Evaluate with recall, precision, F1 or precision-recall AUC instead of plain accuracy.
- How it shows up
- A rare-event dataset with high accuracy but poor recall points to weighting or resampling.
- Common confusion
- Resampling before the train-test split, which leaks duplicated rows into evaluation.
3.3 Compare estimators and transformersThe two core Spark ML abstractions.
- What it means
- In Spark ML, a transformer converts one DataFrame into another with transform(), while an estimator learns from data with fit() and returns a transformer.
- What to understand
- Examples of estimators are StringIndexer, OneHotEncoder, Imputer, StandardScaler, LogisticRegression and Pipeline itself; fitting each yields a model, such as StringIndexerModel or PipelineModel, which is a transformer. Pure transformers such as VectorAssembler or SQLTransformer need no fitting. A fitted model applies learned parameters, which is why fit happens on training data and transform on both splits.
- How it shows up
- A class is named and you decide if it needs fit before transform.
- Common confusion
- Calling VectorAssembler an estimator, or forgetting that a fitted PipelineModel is a transformer.
3.4 Develop a training pipelineChaining preparation and model stages in Spark ML.
- What it means
- A training pipeline chains preparation steps and a model into one object so the same logic runs on training and new data.
- What to understand
- Build pyspark.ml.Pipeline(stages=[indexer, encoder, assembler, model]), call fit on the training DataFrame to get a PipelineModel, then call transform on test or new data. Each estimator stage is fitted in order on the output of earlier stages. Pipelines prevent leakage by fitting preparation only on training data, and the whole fitted pipeline can be logged to MLflow as one model.
- How it shows up
- A stage order is wrong or a stage is missing; choose the correct pipeline definition.
- Common confusion
- Fitting preprocessing separately on all data and then passing only the model into the pipeline.
3.5 Use Hyperopt's fmin operation to tune a model's hyperparametersDefining objective, space and algorithm for fmin.
- What it means
- Hyperopt's fmin minimizes an objective function over a defined search space, trying hyperparameter combinations and keeping the best.
- What to understand
- Call fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=N, trials=Trials()). The objective returns a loss to minimize, so return negative accuracy or AUC when maximizing. Spaces use hp.choice, hp.uniform, hp.loguniform and hp.quniform. fmin returns the best parameter values, with hp.choice returned as an index. Databricks now marks Hyperopt deprecated, recommending Optuna or Ray Tune, but the exam still uses it.
- How it shows up
- Expect a question on what the objective must return, or why an accuracy score is negated.
- Common confusion
- Returning accuracy unchanged, so fmin searches for the worst model.
3.6 Perform random or grid search or Bayesian search as a method for tuning hyperparameters.Comparing tuning strategies.
- What it means
- This covers the three main ways to search hyperparameter space and their trade-offs.
- What to understand
- Grid search tries every combination in a fixed grid; it is exhaustive but grows multiplicatively, and Spark's ParamGridBuilder with CrossValidator implements it. Random search samples combinations and often finds good values faster in large spaces. Bayesian search, such as Hyperopt's TPE algorithm, uses earlier results to pick promising settings next, needing fewer evaluations. Hyperopt's rand.suggest gives random search.
- How it shows up
- A large search space and a limited budget point to random or Bayesian search over grid search.
- Common confusion
- Assuming grid search is always the most thorough affordable option.
3.7 Parallelize single node models for hyperparameter tuningDistributing scikit-learn style trials.
- What it means
- Single-node libraries such as scikit-learn train on one machine, but tuning trials can run in parallel across a cluster's workers.
- What to understand
- With Hyperopt, SparkTrials(parallelism=k) distributes single-node trials to workers, while Trials runs them on the driver. Do not use SparkTrials with distributed Spark ML models; use plain Trials and let Spark parallelize each fit. Higher parallelism finishes sooner but gives the adaptive search less feedback. Pandas UDFs or applyInPandas can also train one model per group in parallel.
- How it shows up
- The model library decides between SparkTrials and Trials; scikit-learn means SparkTrials.
- Common confusion
- Using SparkTrials with Spark ML estimators, which are already distributed.
3.8 Describe the benefits and downsides of using cross-validation over a train-validation split.Weighing k-fold validation against a single holdout.
- What it means
- Cross-validation splits training data into k folds, training k times and validating on each fold in turn, while a train-validation split uses a single holdout.
- What to understand
- Cross-validation gives a more reliable estimate with lower variance and uses all data for both training and validation, which matters on small datasets. Its cost is roughly k times more training. A single split, such as Spark's TrainValidationSplit, is faster and adequate for very large data but its estimate depends on one random split. A separate test set is still needed for final evaluation.
- How it shows up
- Small data and a need for a stable estimate favour cross-validation; huge data and tight time favour one split.
- Common confusion
- Thinking cross-validation removes the need for a held-out test set.
3.9 Perform cross-validation as a part of model fitting.Using CrossValidator in Spark ML.
- What it means
- This covers running k-fold cross-validation while fitting, with Spark ML's CrossValidator or scikit-learn utilities.
- What to understand
- CrossValidator takes estimator (often a Pipeline), estimatorParamMaps from ParamGridBuilder, evaluator such as BinaryClassificationEvaluator, numFolds, and optional parallelism. fit returns a CrossValidatorModel whose bestModel was refit on all training data and whose avgMetrics lists the mean score per parameter set. Putting the whole pipeline inside CrossValidator keeps preprocessing fitted per fold.
- How it shows up
- A code fragment needs the evaluator, parameter maps or numFolds argument identified.
- Common confusion
- Placing preprocessing outside CrossValidator, which leaks fold statistics.
3.10 Identify the number of models being trained in conjunction with a grid-search and cross-validation process.Counting models in tuning.
- What it means
- This objective is a calculation: how many models are fitted when every grid combination is evaluated with k-fold cross-validation.
- What to understand
- Multiply the number of values for each hyperparameter to get the combinations, then multiply by the number of folds. Spark's CrossValidator then refits the best combination once on the full training data, adding one more model. For example, 3 values of maxDepth and 4 of numTrees with 5 folds gives 12 x 5 = 60 models, plus 1 refit, so 61.
- How it shows up
- Expect grid sizes and fold counts, with the refit as the detail that separates options.
- Common confusion
- Adding the parameter counts instead of multiplying them, or forgetting the final refit.
3.11 Use common classification metrics: F1, Log Loss, ROC/AUC, etcReading classification evaluation results.
- What it means
- Classification metrics measure how well predicted classes or probabilities match true labels.
- What to understand
- Accuracy is the share of correct predictions. Precision is the share of predicted positives that are real, recall the share of real positives found, and F1 their harmonic mean. ROC AUC measures ranking ability across thresholds; PR AUC suits rare positives. Log loss penalizes confident wrong probabilities and lower is better. Spark uses BinaryClassificationEvaluator and MulticlassClassificationEvaluator.
- How it shows up
- A confusion matrix or metric definition is given, and you compute or interpret a score.
- Common confusion
- Treating high accuracy on imbalanced data as evidence of a good model.
3.12 Use common regression metrics: RMSE, MAE, R-squared, etc.Reading regression evaluation results.
- What it means
- Regression metrics measure the gap between predicted values and the actual numeric target values.
- What to understand
- RMSE squares errors before averaging, so it punishes large misses and shares the target's units. MAE averages absolute errors, is easier to explain and less sensitive to outliers. R-squared is the share of variance explained, up to 1, and can be negative for a model worse than predicting the mean. For RMSE and MAE lower is better. Spark's RegressionEvaluator supports rmse, mae, r2 and mse.
- How it shows up
- A scenario wants large errors penalized or results explained in target units; choose the matching metric.
- Common confusion
- Assuming R-squared is in the target's units or that higher RMSE is better.
3.13 Choose the most appropriate metric for a given scenario objectiveAligning metrics with business cost.
- What it means
- This objective asks you to translate a business goal and the cost of each error type into the right evaluation metric.
- What to understand
- If false negatives are costly, such as missed fraud or disease, favour recall. If false positives are costly, such as blocking good customers, favour precision. When both matter on imbalanced data, use F1 or PR AUC. For ranking across thresholds use ROC AUC, and for calibrated probabilities use log loss. In regression, RMSE suits cases where big misses hurt most; MAE suits a typical-error view.
- How it shows up
- Read which mistake hurts more before choosing the metric.
- Common confusion
- Defaulting to accuracy whatever the class balance or error costs.
3.14 Identify the need to exponentiate log-transformed variables before calculating evaluation metrics or interpreting predictionsReturning log-scale predictions to original units.
- What it means
- When a model is trained on log(y), its predictions are on the log scale and must be converted back before they are compared with actual values or shown to users.
- What to understand
- Apply exp to predictions, or expm1 if log1p was used, then compute RMSE or MAE against the original target. Metrics computed on the log scale measure relative error and are not in currency or count units, so they are not comparable with an untransformed model's metrics. Forgetting this makes errors look tiny and predictions unreadable.
- How it shows up
- A log-target model reports a very small RMSE; the issue is that predictions were not exponentiated.
- Common confusion
- Comparing a log-scale RMSE directly with an RMSE from a model trained on raw values.
3.15 Assess the impact of model complexity and the bias variance tradeoff on model performanceUnderfitting, overfitting and the balance between them.
- What it means
- Bias is error from overly simple assumptions, causing underfitting; variance is error from sensitivity to the training data, causing overfitting.
- What to understand
- As complexity rises, through deeper trees, more features or less regularization, training error falls while validation error first falls and then rises. High training and validation error signals high bias: add complexity or features. Low training error with high validation error signals high variance: regularize, simplify, add data or use ensembles such as bagging. Tuning aims for the lowest validation error.
- How it shows up
- Training and validation scores are given, and you diagnose underfitting or overfitting and the fix.
- Common confusion
- Adding complexity to a model that already overfits the training data.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Estimator
Learns with fit() and returns a model.
- Remember
- Imputer, StringIndexer, LogisticRegression, Pipeline.
- Tested as
- Spark ML structure questions.
- Trap
- Calling VectorAssembler one.
-
Transformer
Maps a DataFrame with transform().
- Remember
- Fitted models are transformers.
- Tested as
- Spark ML structure questions.
- Trap
- Expecting it to need fit.
-
Pipeline
Ordered chain of estimators and transformers.
- Remember
- fit returns a PipelineModel.
- Tested as
- Pipeline questions.
- Trap
- Fitting preprocessing outside it.
-
fmin
Hyperopt function that minimizes a loss.
- Remember
- Negate metrics you want to maximize.
- Tested as
- Tuning code questions.
- Trap
- Returning raw accuracy.
-
TPE
Bayesian algorithm in Hyperopt (tpe.suggest).
- Remember
- Uses past trials to choose the next.
- Tested as
- Search strategy questions.
- Trap
- Calling it random search.
-
SparkTrials
Runs single-node trials across workers.
- Remember
- For scikit-learn style models, not Spark ML.
- Tested as
- Parallel tuning questions.
- Trap
- Using it with Spark ML.
-
CrossValidator
Spark ML k-fold tuning wrapper.
- Remember
- bestModel is refit on all training data.
- Tested as
- Cross-validation questions.
- Trap
- Forgetting the refit.
-
Model count formula
Grid combinations times folds, plus one refit.
- Remember
- Multiply, do not add.
- Tested as
- Calculation questions.
- Trap
- Adding parameter counts.
-
F1 score
Harmonic mean of precision and recall.
- Remember
- Good for imbalanced classes.
- Tested as
- Metric choice questions.
- Trap
- Using accuracy instead.
-
Log loss
Penalizes confident wrong probabilities.
- Remember
- Lower is better.
- Tested as
- Probability metric questions.
- Trap
- Treating higher as better.
-
RMSE vs. MAE
Squared versus absolute error.
- Remember
- RMSE punishes large misses more.
- Tested as
- Regression metric questions.
- Trap
- Swapping their behaviour.
-
Exponentiate predictions
Undo a log transform on outputs.
- Remember
- Use exp or expm1 before metrics.
- Tested as
- Log-target questions.
- Trap
- Reporting log-scale errors.
-
Bias-variance tradeoff
Balance underfitting against overfitting.
- Remember
- Validation error is the guide.
- Tested as
- Complexity questions.
- Trap
- Adding complexity to an overfit model.
-
Class weights
Increase the loss weight of rare classes.
- Remember
- weightCol in Spark ML.
- Tested as
- Imbalance questions.
- Trap
- Resampling before splitting.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Grid vs. random vs. Bayesian search
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Grid search | Every combination in a fixed grid | Exhaustive; cost multiplies | Using it on large spaces |
| Random search | Samples combinations at random | Efficient in large spaces | Expecting full coverage |
| Bayesian (TPE) | Uses past results to guide sampling | Fewest evaluations to good values | Running too many trials in parallel |
Cross-validation vs. train-validation split
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Cross-validation | k folds, each used once for validation | Stable estimate; k times the cost | Skipping the final test set |
| Train-validation split | One holdout set | Fast; depends on one split | Using it on tiny datasets |
Estimator vs. transformer
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Estimator | Has fit(); learns parameters | Returns a fitted model | Treating it as ready to transform |
| Transformer | Has transform(); maps data | No learning step | Expecting VectorAssembler to fit |
Bias vs. variance
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| High bias | Underfitting | High training and validation error | Adding regularization |
| High variance | Overfitting | Low training, high validation error | Adding more complexity |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Forget the refit
- What candidates pick
- Answer 60 for a 12-combination grid with 5-fold cross-validation.
- Why it looks right
- Combinations times folds looks complete.
- Why it's wrong
- CrossValidator refits the best parameter set on the full training data, so 61 models are trained in total.
Clue to watch for The question says the best model is produced or asks for all models trained.
-
Maximize with fmin
- What candidates pick
- Return accuracy directly from the objective function.
- Why it looks right
- Accuracy is the metric being optimized.
- Why it's wrong
- fmin minimizes, so returning accuracy searches for the worst settings; return negative accuracy or a loss.
Clue to watch for The objective returns a score where higher is better.
-
SparkTrials with Spark ML
- What candidates pick
- Use SparkTrials to tune a distributed Spark ML random forest.
- Why it looks right
- Parallel sounds faster.
- Why it's wrong
- Spark ML already distributes each fit; SparkTrials is for single-node libraries, so use Trials with Spark ML.
Clue to watch for The estimator comes from pyspark.ml.
-
Accuracy on rare events
- What candidates pick
- Pick the model with 99% accuracy for 1% positive data.
- Why it looks right
- It is the highest headline number.
- Why it's wrong
- Predicting all negatives gives 99% accuracy; recall, precision, F1 or PR AUC show whether positives are found.
Clue to watch for A heavily imbalanced label.
-
Log-scale RMSE
- What candidates pick
- Report RMSE computed directly on log predictions.
- Why it looks right
- The pipeline output is ready to evaluate.
- Why it's wrong
- Errors on the log scale are relative, not in target units; exponentiate predictions before computing RMSE or MAE.
Clue to watch for The target was log-transformed during preparation.
-
More complexity for overfitting
- What candidates pick
- Increase tree depth when validation error is high but training error is low.
- Why it looks right
- More capacity sounds like better performance.
- Why it's wrong
- That gap signals high variance; reduce complexity, regularize or add data instead.
Clue to watch for Training score is much better than validation score.
Answer framework
Work through these steps for Section 3 scenarios on algorithms, tuning, validation and metrics.
- Name the task from the target: regression, classification or clustering, then note constraints such as interpretability, class balance and data size.
- Check structure: estimators need fit and become transformers, and preprocessing belongs inside a Pipeline that is fitted on training data only.
- For tuning, pick grid search for small spaces and random or Bayesian TPE for large ones; make the fmin objective return a loss, and use SparkTrials only with single-node models.
- For validation, prefer cross-validation on small data and a single split on very large data, and count models as combinations times folds plus one refit.
- Choose the metric from the cost of errors, exponentiate log-target predictions before scoring, and read the training versus validation gap to diagnose bias or variance.
Practice question
Want more like this? The Databricks ML Assoc practice test has questions for every domain.
Checklist
0 of 15
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Split preparation for this section into concepts, Spark ML mechanics and calculations. For concepts, review algorithm families, imbalance handling, metric definitions and the bias-variance tradeoff until you can explain each in a sentence and connect it to a scenario clue.
For mechanics, build one Spark ML Pipeline with an indexer, encoder, assembler and classifier, wrap it in CrossValidator with a ParamGridBuilder grid, and inspect bestModel and avgMetrics. Then tune a scikit-learn model with Hyperopt fmin, first with Trials and then SparkTrials, logging trials to MLflow. Return a negated score once deliberately to see the search go wrong. Recent ML runtimes no longer ship Hyperopt, so install it with pip or use an older LTS ML runtime for practice.
For calculations, drill model counts with different grids and folds, and practise converting log-scale predictions back with exp before computing RMSE. Finish with mixed timed questions, then move to Section 4, Model Deployment.
FAQ
How much of the Databricks ML Associate exam is Model Development?
Model Development is 31% of the exam, about 15 of the 48 scored questions, second only to Databricks Machine Learning.
Is Hyperopt still on the exam even though it is deprecated?
Yes. The current objectives name Hyperopt's fmin. Databricks documentation marks Hyperopt deprecated and recommends Optuna or Ray Tune, but study Hyperopt for the exam.
How do I count models in a grid search with cross-validation?
Multiply the values of each hyperparameter to get the combinations, multiply by the number of folds, then add one for the final refit on all training data.
When should I use SparkTrials?
Use SparkTrials to parallelize trials of single-node models such as scikit-learn across workers. Use the default Trials with Spark ML models, which are already distributed.
Which metric should I use for imbalanced classification?
Use recall, precision, F1 or precision-recall AUC, chosen by which error is more costly. Accuracy is misleading when one class dominates.
Why must I exponentiate predictions from a log-transformed target?
Predictions come out on the log scale. Converting them back with exp or expm1 puts them in the original units so RMSE, MAE and business interpretation make sense.
How does Model Development connect to the other sections?
It uses features prepared in Data Processing, tracks runs and models with MLflow from Databricks Machine Learning, and produces the models served in Model Deployment.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 3 knowledge
Practise metric choice, model counting, Hyperopt tuning and cross-validation questions under timed exam conditions to see where you still lose marks.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Databricks ML Assoc AI tutor Get any concept explained your way