Overview
Advanced Techniques for Machine Learning has four parts. The machine learning review walks the pipeline from raw data to a monitored model and surveys algorithm families. Data mining covers pattern discovery such as association rules and anomaly detection. Measuring model performance separates classification metrics (F1, ROC AUC, precision-recall curves, log loss) from regression metrics (MAE, RMSE, R-squared) and shows how to set hyperparameters without contaminating the test set. Dealing with black boxes introduces interpretation tools for complex models.
The module is the bridge between Module 2's statistics and the specific algorithms in Modules 4, 5 and 8. Whatever model you later choose, it enters this pipeline, is judged with these metrics and, if it is opaque, must be explained with these techniques before a business will act on it.
How it's tested
Expect scenario questions that hinge on choosing the right metric or the right next step in a pipeline, plus some short calculations such as F1, RMSE or rule lift. Many items describe a flawed process and ask what invalidates the reported result, so read the order of operations closely.
-
Pick the metric
A business problem and data balance are described, and you choose the metric that reflects the real cost of errors, such as recall, PR AUC or RMSE.
-
Find the flaw in the pipeline
A workflow is described step by step, and you spot leakage, tuning on test data or a missing validation stage.
-
Explain the model
A stakeholder or regulator needs a reason for an individual prediction or for the model overall, and you choose between global and local interpretation methods.
A team tunes 200 hyperparameter combinations, picks the one with the best test-set score and reports that score. The flaw is that the test set guided model selection, so it no longer estimates performance on unseen data. The fix is to tune with cross-validation or a separate validation set and touch the test set once at the end. Options that change the algorithm miss the problem.
Topics 4
Official topics from the Enterprise Big Data Framework Alliance outline, in outline order.
3.1 Machine Learning ReviewThe machine learning pipeline from raw data to monitored model, and a review of the main algorithm families.
- What it means
- The pipeline orders the work: define the problem, collect and clean data, engineer features, split into training, validation and test sets, train, tune, evaluate, deploy and monitor. The algorithms review groups methods by task: regression, classification, clustering, dimensionality reduction and so on.
- What to understand
- Any transformation that learns from data, such as scaling, imputation or encoding, must be fit on training data only and then applied to validation and test data, or leakage inflates scores. Monitoring after deployment watches for data drift (inputs change) and concept drift (the input-output relationship changes).
- How it shows up
- A pipeline where missing values were imputed with the full-dataset mean before splitting is described, and you identify the leakage.
- Common confusion
- Thinking leakage only means including the target column; preprocessing fitted on all data leaks too.
3.2 Data MiningDiscovering patterns in large datasets, including association rules and anomalies.
- What it means
- Data mining searches large datasets for useful patterns such as frequent itemsets and association rules, clusters, sequences and outliers. Association rules take the form 'if A then B', and are measured by support, confidence and lift.
- What to understand
- Support is the share of transactions containing both A and B; confidence is P(B | A); lift is confidence divided by P(B). Lift above 1 means A and B co-occur more than chance, equal to 1 means independence, and below 1 means they co-occur less. Apriori prunes the search because every subset of a frequent itemset must also be frequent.
- How it shows up
- Transaction counts are given and you compute or interpret the lift of a rule to decide whether a cross-sell is worthwhile.
- Common confusion
- Treating high confidence as proof of a useful rule when the consequent is popular anyway, so lift is near 1.
3.3 Measuring Model PerformanceClassification and regression performance metrics, and setting hyperparameters.
- What it means
- Classification metrics include F1 score, the ROC curve and AUC, the precision-recall curve and log loss; regression metrics include MAE, MSE, RMSE and R-squared. Hyperparameters are settings chosen before training, such as tree depth or regularization strength, unlike parameters learned from data.
- What to understand
- F1 = 2PR / (P + R) balances precision and recall. ROC AUC can look strong on heavily imbalanced data, where the precision-recall curve is more informative. RMSE is in the target's units and penalizes large errors; R-squared = 1 - SS_res / SS_tot and can be negative on new data. Tune with grid, random or Bayesian search on validation folds only.
- How it shows up
- A rare-event classifier reports ROC AUC 0.95 and you must recognize that precision may still be poor, or choose RMSE versus MAE for a cost scenario.
- Common confusion
- Treating hyperparameters and parameters as the same thing, or tuning on the test set.
3.4 Dealing with Black BoxesInterpreting complex models with global and local explanation methods.
- What it means
- Black-box models such as ensembles and neural networks predict well but resist direct inspection. Interpretation methods include permutation feature importance, partial dependence plots, global surrogate models, LIME and SHAP.
- What to understand
- Global methods describe overall behavior; local methods explain one prediction. LIME fits a simple model around one instance; SHAP assigns each feature a contribution based on Shapley values so contributions sum to the difference between the prediction and the baseline. Explanations describe the model, not causation in the world.
- How it shows up
- A customer disputes an individual credit decision and you choose a local method such as SHAP or LIME rather than a global importance chart.
- Common confusion
- Using global feature importance to explain one person's outcome, or reading an important feature as a cause.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Data leakage
Information from outside the training data, such as test-set statistics or future values, influences training and inflates evaluation scores.
- Remember
- Fit every learned preprocessing step on training data only, inside cross-validation folds where used.
- Tested as
- Pipeline-flaw questions usually turn on leakage.
- Trap
- Believing leakage only means the target column slipped into the features.
-
Train, validation and test sets
Training data fits parameters, validation data (or cross-validation) selects hyperparameters and models, and the test set gives one final unbiased estimate.
- Remember
- Once the test set influences a choice, it is no longer a test set.
- Tested as
- Scenarios ask which set should be used for tuning or reporting.
- Trap
- Reporting the best of many test-set scores as expected performance.
-
Support, confidence and lift
Support = P(A and B); confidence = P(B | A); lift = confidence / P(B).
- Remember
- Lift above 1 indicates positive association; lift of 1 indicates independence.
- Tested as
- Market-basket calculations appear as short computation items.
- Trap
- Taking confidence as the measure of rule strength when B is common anyway.
-
F1 score
The harmonic mean of precision and recall, 2PR / (P + R).
- Remember
- It is low if either precision or recall is low, unlike the arithmetic mean.
- Tested as
- Asked when both false positives and false negatives matter on imbalanced data.
- Trap
- Computing the simple average of precision and recall.
-
ROC AUC vs. PR AUC
ROC plots true positive rate against false positive rate across thresholds; PR plots precision against recall.
- Remember
- With very rare positives, PR curves reveal weak precision that ROC AUC hides.
- Tested as
- Rare-event scenarios ask which curve to trust.
- Trap
- Accepting a high ROC AUC as proof a fraud model is usable.
-
RMSE, MAE and R-squared
RMSE is the square root of mean squared error in target units; MAE is mean absolute error; R-squared is the share of variance explained relative to predicting the mean.
- Remember
- RMSE is at least MAE and grows with large errors; R-squared can be negative on test data.
- Tested as
- Regression-metric choice and interpretation items.
- Trap
- Thinking R-squared is bounded between 0 and 1 on any dataset.
-
Hyperparameter search
Grid search tries every combination, random search samples combinations, and Bayesian optimization models the score to choose promising next trials.
- Remember
- Random search often finds good settings faster than grid search when only a few hyperparameters matter.
- Tested as
- Questions ask which search fits a large space or limited compute.
- Trap
- Assuming grid search is always the most thorough practical choice.
-
Apriori principle
If an itemset is frequent, all its subsets are frequent; equivalently, any superset of an infrequent itemset is infrequent.
- Remember
- This lets Apriori prune candidate itemsets level by level.
- Tested as
- Data-mining questions ask why Apriori scales better than brute force.
- Trap
- Inverting it to say supersets of frequent itemsets are frequent.
-
SHAP values
Feature contributions based on cooperative game theory, which add up to the difference between a prediction and the average prediction.
- Remember
- They give consistent local explanations that can be aggregated into global importance.
- Tested as
- Explainability scenarios with an individual decision to justify.
- Trap
- Reading a large SHAP value as evidence that the feature causes the outcome.
-
LIME
Local Interpretable Model-agnostic Explanations perturbs one instance and fits a simple surrogate, such as a sparse linear model, around it.
- Remember
- It explains one prediction locally and can vary with the perturbation settings.
- Tested as
- Contrasted with SHAP and with global methods.
- Trap
- Treating a LIME explanation as describing the whole model.
-
Data drift vs. concept drift
Data drift is a change in input distributions; concept drift is a change in the relationship between inputs and the target.
- Remember
- Concept drift can degrade a model even when input distributions look stable.
- Tested as
- Monitoring questions after deployment.
- Trap
- Monitoring only inputs and missing a changed relationship.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Global vs. local explanation methods
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Permutation feature importance | Drop in performance when one feature's values are shuffled | Global: ranks features for the whole model | Misleading when features are strongly correlated |
| Partial dependence plot | Average predicted outcome as one feature varies | Global: shows the shape of an effect | Averages can hide opposite effects in subgroups |
| LIME | Simple surrogate fitted around one instance | Local: explains one prediction | Results can change between runs with different perturbations |
| SHAP | Shapley-based additive feature contributions | Local, and aggregates to global | Interpreting contributions as causal effects |
Parameters vs. hyperparameters
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Parameters | Values learned from training data, such as regression coefficients or network weights | Set by the training algorithm | Trying to tune them by grid search |
| Hyperparameters | Settings fixed before training, such as k in k-NN, tree depth or learning rate | Chosen with validation data or cross-validation | Selecting them on the test set |
MAE vs. RMSE vs. R-squared
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| MAE | Average absolute error in target units | Treats all errors linearly; robust to outliers | Using it when large errors are disproportionately costly |
| RMSE | Square root of mean squared error in target units | Penalizes large errors more heavily | Comparing RMSE across targets with different scales |
| R-squared | 1 - SS_res / SS_tot, variance explained versus predicting the mean | Unitless, relative measure | Assuming it cannot be negative on test data |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Tuning on the test set
- What candidates pick
- Report the configuration with the best test-set accuracy as the model's expected performance.
- Why it looks right
- It is the highest score and was measured on held-out data.
- Why it's wrong
- Choosing among many configurations by test score overfits the test set, so the reported number is optimistic.
Clue to watch for The stem says the test set was used to compare or select configurations.
-
High confidence, no lift
- What candidates pick
- Promote a cross-sell because 'customers who buy pasta buy milk' has 70% confidence.
- Why it looks right
- Seventy percent sounds like a strong relationship.
- Why it's wrong
- If 70% of all baskets contain milk, lift is 1 and pasta adds nothing; the rule reflects milk's popularity.
Clue to watch for The overall frequency of the consequent is given or is obviously high.
-
ROC AUC on rare events
- What candidates pick
- Deploy a fraud model because its ROC AUC is 0.95.
- Why it looks right
- An AUC of 0.95 is commonly described as excellent.
- Why it's wrong
- With very few positives, a small false-positive rate still yields many false alarms; the precision-recall curve shows the real tradeoff.
Clue to watch for Positives are well under 1% and investigation capacity is limited.
-
Global answer to a local question
- What candidates pick
- Show the model's overall feature-importance chart to a customer disputing a loan denial.
- Why it looks right
- It is the most familiar explanation output.
- Why it's wrong
- Overall importance does not say why this applicant was denied; a local method such as SHAP or LIME does.
Clue to watch for A single individual or a single decision needs justifying.
-
Leakage through preprocessing
- What candidates pick
- Impute missing values with the dataset mean, then split into train and test.
- Why it looks right
- It is simpler and uses all the data.
- Why it's wrong
- Test-set values influence the imputed training values, so evaluation is no longer on truly unseen data.
Clue to watch for A learned transformation happens before the split in the described order.
-
Importance means cause
- What candidates pick
- Conclude that raising a customer's app logins will reduce churn because it is the top SHAP feature.
- Why it looks right
- The feature clearly drives the prediction.
- Why it's wrong
- Explanations describe the model's associations, not interventions; a causal claim needs an experiment or causal analysis.
Clue to watch for The option proposes an intervention based only on a model explanation.
Answer framework
Apply these steps to pipeline, metric and explanation questions in this module.
- Trace the described workflow in order and check that every learned transformation and every tuning decision happens on training or validation data only.
- Identify the class balance or the error costs, then choose the metric that reflects them, such as recall, PR AUC, F1, RMSE or MAE.
- For association rules, compute support, confidence and lift, and judge usefulness by lift rather than confidence alone.
- Decide whether the stakeholder needs a global view of the model or a reason for one specific prediction.
- Reject any answer that reports test performance after test data shaped a decision or that reads an explanation as causation.
Practice question
Want more like this? The EBDS practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Start with topic 3.1 and draw the pipeline yourself, marking where each set is used. Then build one end-to-end notebook with a scikit-learn Pipeline object so that scaling and imputation are fitted inside cross-validation; doing it once makes leakage questions easy to spot.
Next, spend the most time on topic 3.3. Build a metric table with formula, units, what it penalizes and when it misleads. Practice on an imbalanced dataset so you see ROC AUC stay high while precision collapses. Learn the difference between grid, random and Bayesian search in terms of cost and coverage.
Topic 3.2 is commonly underestimated: work three or four market-basket calculations until lift is second nature, because confidence is the planted distractor. Finish with topic 3.4 by running SHAP on a tree model and comparing it with permutation importance. Afterwards, take timed sets that mix pipeline-flaw and metric-choice items, then move to Module 4, where these evaluation tools are applied to specific classifiers.
FAQ
What is the most common way candidates lose marks in this module?
Choosing a metric that ignores class imbalance, and missing leakage in a described workflow. Both are about reading the scenario's process and data, not the algorithm.
How many questions cover advanced machine learning techniques?
The Alliance does not publish module weights. An even split of the 80 questions gives about nine, which is a planning estimate only.
When should I use F1 instead of accuracy?
Use F1 when classes are imbalanced and both false positives and false negatives matter, because accuracy can look high while the minority class is handled badly.
What is the difference between confidence and lift?
Confidence is the probability of B given A. Lift divides that by B's overall probability, so it shows whether A actually raises the chance of B.
Why can R-squared be negative?
On new data, a model can predict worse than simply using the mean of the target, which makes the residual sum of squares exceed the total sum of squares.
Should I explain a model with SHAP or LIME?
Both give local explanations. SHAP has a consistent additive basis and aggregates to global importance; LIME is a fast local surrogate whose results can vary with its sampling settings.
How does this module relate to Supervised and Deep Learning?
Those modules introduce the algorithms; this module supplies the pipeline, metrics, tuning discipline and explanation tools used to judge and justify them.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 3 knowledge
You have covered the leakage-free pipeline, association rules, metric choice, hyperparameter tuning and black-box explanations. Check whether you can spot the flawed step and the right metric under exam timing.