Overview
Supervised Machine Learning covers three topics. Feature selection and validation teaches how to engineer and choose predictors and how to estimate test error honestly with a validation set, k-fold cross-validation and the bootstrap, plus forward and backward stepwise selection. Linear classifiers move from separating hyperplanes and maximal margin classifiers to support vector classifiers and support vector machines, then to logistic regression and linear discriminant analysis. Non-linear classifiers focus on tree-based methods, from single decision trees to bagging, random forests and boosting.
The module builds directly on Modules 2 and 3: the loss functions, bias-variance tradeoff and metrics reappear as concrete choices of C, tree depth or number of folds. It also sets up Module 7, where the same models are trained at scale with Spark MLlib, and Module 8, where neural networks extend the idea of a learned decision boundary.
How it's tested
Questions are mostly model-selection and validation scenarios with BEST or MOST appropriate phrasing: the data size, class structure, interpretability requirement or error pattern points to one classifier or one validation approach. Expect a few numeric items such as the bootstrap's out-of-bag share or interpreting a logistic regression coefficient as an odds ratio.
-
Choose the classifier
The scenario describes boundary shape, sample size, need for probabilities or need for explanation, and you select SVM, logistic regression, LDA or a tree ensemble.
-
Choose the validation approach
Data size, class balance or time ordering is described, and you pick a validation set, k-fold, stratified k-fold, a time-based split or the bootstrap.
-
Tune and diagnose
A model is underfitting or overfitting and you choose the hyperparameter change, such as C, kernel, tree depth, number of features per split or learning rate.
A bank needs a default model whose outputs are calibrated probabilities and whose drivers can be explained as odds to a regulator. Logistic regression fits: it models the log-odds linearly, and each exponentiated coefficient is an odds ratio. An RBF-kernel SVM may separate classes well but gives no native probabilities or coefficient-level explanation, so it is the weaker choice despite possible accuracy gains.
Topics 3
Official topics from the Enterprise Big Data Framework Alliance outline, in outline order.
4.1 Feature Selection and ValidationFeature engineering and selection, the validation set approach, k-fold cross-validation, bootstrapping and stepwise selection.
- What it means
- Feature selection keeps the predictors that carry signal, and validation estimates how a model will perform on new data. The syllabus covers feature selection and engineering, a single validation set, k-fold cross-validation, the bootstrap, and forward and backward stepwise selection.
- What to understand
- A single validation split gives a high-variance error estimate and wastes data; k-fold (usually k = 5 or 10) uses every observation for validation once. A bootstrap sample of n drawn with replacement contains about 63.2% of distinct observations, leaving about 36.8% out-of-bag. Forward stepwise fits about p(p+1)/2 + 1 models instead of 2^p; backward stepwise needs more observations than predictors.
- How it shows up
- A small dataset is evaluated with one 70/30 split and results swing between runs, and you choose k-fold cross-validation as the fix.
- Common confusion
- Assuming stepwise selection finds the best possible subset; it is a greedy search that can miss it.
4.2 Linear ClassifiersHyperplanes and maximal margin hyperplanes, support vector classifiers and machines, logistic regression and linear discriminant analysis.
- What it means
- A maximal margin classifier separates two classes with the hyperplane farthest from the nearest points, the support vectors. A support vector classifier allows some violations (soft margin), and a support vector machine applies kernels to create non-linear boundaries. Logistic regression models log-odds as a linear function; LDA models each class as Gaussian with a shared covariance and applies Bayes' theorem.
- What to understand
- In scikit-learn, a larger C penalizes margin violations more, giving a narrower margin with lower bias and higher variance. Kernels such as polynomial and radial basis function (RBF) compute similarities in an expanded feature space without building it. In logistic regression, e to the power of a coefficient is the odds ratio for a one-unit increase. LDA is stable when classes are well separated or n is small, where logistic regression estimates can become unstable.
- How it shows up
- Classes form concentric rings and you choose an SVM with an RBF kernel, or a coefficient of 0.69 is given and you interpret it as roughly doubling the odds.
- Common confusion
- Confusing the maximal margin classifier, which needs perfectly separable classes, with the support vector classifier, which tolerates violations through a soft margin.
4.3 Non-Linear ClassifiersTree-based methods: decision trees, pruning, bagging, random forests and boosting.
- What it means
- Decision trees split the feature space with recursive binary splits chosen to reduce impurity, measured by Gini index or entropy. Ensembles combine many trees: bagging averages trees fit on bootstrap samples, random forests also restrict each split to a random subset of predictors, and boosting fits trees sequentially to the errors of previous ones.
- What to understand
- Single deep trees have high variance; cost-complexity pruning trades fit for simplicity. Random forests decorrelate trees, typically trying about the square root of p predictors per split for classification. Boosting reduces bias with many shallow trees and a small learning rate but can overfit if run too long. Trees need no feature scaling and handle interactions naturally.
- How it shows up
- Bagged trees all split first on the same dominant predictor and barely beat a single tree, and you choose a random forest to decorrelate them.
- Common confusion
- Treating bagging and boosting as the same: bagging builds independent trees in parallel to cut variance, boosting builds dependent trees in sequence to cut bias.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
k-fold cross-validation
Split the data into k folds, train on k - 1 and validate on the remaining fold, rotating so each fold validates once, then average the errors.
- Remember
- Use stratified folds for imbalanced classes and forward-chaining splits for time-ordered data.
- Tested as
- The default answer for honest error estimation on limited data.
- Trap
- Shuffling time-series data into random folds, which lets the model see the future.
-
Bootstrap and out-of-bag error
Draw n observations with replacement; about 63.2% of distinct points appear and about 36.8% are left out.
- Remember
- Out-of-bag observations give a built-in validation estimate for bagging and random forests.
- Tested as
- Numeric items and questions on how random forests self-validate.
- Trap
- Thinking a bootstrap sample is smaller than the original dataset.
-
Forward and backward stepwise selection
Forward starts with no predictors and adds the most helpful one at each step; backward starts with all and removes the least useful.
- Remember
- Both are greedy; backward requires n > p, forward does not.
- Tested as
- Asked when p is large relative to n or compute is limited.
- Trap
- Choosing backward selection when there are more predictors than observations.
-
Filter, wrapper and embedded selection
Filters score features independently (correlation, mutual information), wrappers search subsets using a model, embedded methods select during training (lasso, tree importance).
- Remember
- Lasso (L1) can shrink coefficients to exactly zero; ridge (L2) cannot.
- Tested as
- Feature-selection scenarios balancing cost and accuracy.
- Trap
- Expecting ridge regression to drop features.
-
Support vectors and margin
Support vectors are the observations on or inside the margin; they alone define the SVM boundary.
- Remember
- Moving a point far from the margin does not change the boundary.
- Tested as
- Conceptual items on why SVMs are robust to distant points.
- Trap
- Believing every training point shapes the boundary equally.
-
SVM C and kernels
C sets the penalty for margin violations; kernels (linear, polynomial, RBF) define the similarity function and boundary shape.
- Remember
- In scikit-learn, larger C means fewer tolerated violations and higher variance; RBF gamma controls how local each point's influence is.
- Tested as
- Tuning and boundary-shape scenarios.
- Trap
- Increasing C to fix overfitting.
-
Logistic regression odds ratio
The model is log(p / (1 - p)) = b0 + b1x; exp(b1) is the multiplicative change in odds for a one-unit increase in x.
- Remember
- exp(0.693) is about 2, so a coefficient near 0.69 doubles the odds, not the probability.
- Tested as
- Interpretation items and regulator-facing scenarios.
- Trap
- Reading the coefficient as a change in probability.
-
Linear discriminant analysis
Models each class as a Gaussian with a common covariance matrix and classifies with Bayes' theorem, producing linear boundaries.
- Remember
- Quadratic discriminant analysis allows class-specific covariances and curved boundaries but needs more data.
- Tested as
- Compared with logistic regression on small or well-separated data.
- Trap
- Using LDA when classes clearly have very different spreads.
-
Gini index and entropy
Impurity measures for a tree node; both are zero for a pure node and highest when classes are evenly mixed.
- Remember
- Splits are chosen to maximize the reduction in impurity.
- Tested as
- Tree-building questions.
- Trap
- Using classification error rate to grow trees, which is less sensitive to node purity.
-
Bagging vs. random forest vs. boosting
Bagging averages trees on bootstrap samples; random forests add random predictor subsets per split; boosting adds shallow trees sequentially on residual errors.
- Remember
- Random forests mainly cut variance by decorrelating trees; boosting mainly cuts bias.
- Tested as
- Ensemble-choice scenarios.
- Trap
- Adding more bagged trees to fix correlation between trees.
-
Cost-complexity pruning
Grows a large tree and then removes branches that do not justify their complexity, tuned by a penalty parameter alpha via cross-validation.
- Remember
- Pruning reduces variance at a small cost in bias.
- Tested as
- Single-tree overfitting scenarios.
- Trap
- Stopping early with a high minimum split threshold, which can miss useful later splits.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Validation set vs. k-fold cross-validation vs. bootstrap
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Validation set approach | One train/validation split | Fast, but the estimate varies with the split and uses less training data | Relying on it for small datasets |
| k-fold cross-validation | k rotating splits, errors averaged | Every observation validates once; more stable estimate | Random folds on time-ordered data |
| Bootstrap | Resample n with replacement many times | Estimates variability of a statistic; out-of-bag points validate ensembles | Treating in-bag performance as test performance |
Logistic regression vs. LDA vs. SVM
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Logistic regression | Linear model of log-odds fit by maximum likelihood | Gives probabilities and odds-ratio interpretation | Unstable estimates when classes are perfectly separated |
| LDA | Gaussian classes with shared covariance plus Bayes' theorem | Stable with small n and well-separated classes; assumes normality | Applying it when class covariances differ greatly |
| SVM | Maximum soft margin, with kernels for non-linear boundaries | Boundary set by support vectors; no native probabilities | Forgetting to scale features before training |
Bagging vs. random forest vs. boosting
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Bagging | Average of deep trees on bootstrap samples | Parallel and independent; reduces variance | Trees stay correlated when one predictor dominates |
| Random forest | Bagging plus a random predictor subset at each split | Decorrelates trees for a larger variance reduction | Setting the subset to all predictors, which is just bagging |
| Boosting | Sequential shallow trees fit to previous errors | Reduces bias; needs a learning rate and stopping point | Running too many rounds without validation, causing overfitting |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
More trees for correlated trees
- What candidates pick
- Increase the number of bagged trees from 100 to 1,000.
- Why it looks right
- More trees usually stabilize an ensemble.
- Why it's wrong
- If every tree splits first on the same dominant predictor, the trees stay correlated and averaging more of them gives little gain; random predictor subsets per split fix that.
Clue to watch for One strong predictor dominates the top split of every tree.
-
Random folds on time series
- What candidates pick
- Shuffled 10-fold cross-validation for a daily demand forecast.
- Why it looks right
- k-fold is the standard recommended validation.
- Why it's wrong
- Shuffling lets the model train on future days and predict the past, inflating accuracy; time-ordered splits keep validation strictly after training.
Clue to watch for The data is ordered in time and the goal is forecasting.
-
Higher C for overfitting
- What candidates pick
- Raise C when an SVM fits training data perfectly but validates poorly.
- Why it looks right
- C sounds like a quality or confidence setting.
- Why it's wrong
- In the scikit-learn convention, larger C tolerates fewer violations and fits training data more tightly, increasing variance; lower C widens the margin.
Clue to watch for Training accuracy is near perfect while validation accuracy is much lower.
-
Coefficient as probability
- What candidates pick
- A logistic coefficient of 0.69 raises default probability by 69 percentage points.
- Why it looks right
- Linear regression coefficients are read that way.
- Why it's wrong
- The coefficient is on the log-odds scale; exp(0.69) is about 2, so the odds roughly double, and the probability change depends on the starting level.
Clue to watch for The model is logistic and the question asks how to interpret a coefficient.
-
Backward selection with wide data
- What candidates pick
- Backward stepwise selection for 500 predictors and 200 observations.
- Why it looks right
- Starting from all features feels thorough.
- Why it's wrong
- The full model cannot be fit when predictors outnumber observations; forward selection or a penalized method such as lasso can.
Clue to watch for p is larger than n.
-
Scaling not needed for SVM
- What candidates pick
- Train an RBF SVM on raw income in dollars and age in years.
- Why it looks right
- Scaling is often said to be optional for many models.
- Why it's wrong
- SVMs rely on distances, so the large-scale feature dominates the kernel; standardize first. It is trees that do not need scaling.
Clue to watch for Features are on very different scales and the model is distance or kernel based.
Answer framework
Use these steps to choose and validate a supervised model in an EBDS scenario.
- Note the data size, the ratio of predictors to observations, class balance and whether the observations are ordered in time.
- Pick the validation approach that matches those facts: k-fold, stratified folds, time-ordered splits or the bootstrap with out-of-bag error.
- Read the requirement that constrains the model, such as calibrated probabilities, explainability to a regulator, a non-linear boundary or a very small sample.
- Select the classifier that meets that requirement, then confirm its preprocessing needs, such as scaling for SVMs and none for trees.
- Diagnose bias or variance from the reported errors and adjust the matching knob: C, kernel, tree depth, predictors per split or boosting rounds.
Practice question
Want more like this? The EBDS practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Start with topic 4.1, because every later choice in the module is judged through validation. Implement a validation split, 5-fold and 10-fold cross-validation and a bootstrap on the same small dataset and compare how much the error estimates move. Add a time-ordered split on a dated dataset to feel why shuffling misleads.
Then take topic 4.2 one model at a time. Plot SVM boundaries for several C and gamma values, fit logistic regression and convert coefficients to odds ratios, and compare LDA with logistic regression on a small, well-separated sample. High-priority concepts are support vectors, the direction of C, and odds-ratio interpretation.
Topic 4.3 is commonly underestimated because trees feel intuitive. Fit a single tree, a pruned tree, bagging, a random forest and gradient boosting on the same data and note how each changes variance and bias; this makes ensemble questions quick. Afterwards, drill timed model-selection scenarios, then move to Module 5 to contrast these labeled methods with clustering and PCA.
FAQ
Which supervised topics matter most for the EBDS exam?
Validation approaches, SVM tuning, logistic regression interpretation and the differences between bagging, random forests and boosting carry the most scenario value, because they decide which model or process is correct.
How many exam questions cover supervised machine learning?
No module weights are published. An even split of the 80 questions suggests about nine, as a planning estimate only.
Why are support vector machines listed under linear classifiers?
The syllabus builds from separating hyperplanes to SVMs. A linear-kernel SVM is a linear classifier, and kernels such as RBF extend the same margin idea to non-linear boundaries.
When is LDA better than logistic regression?
When classes are well separated, when the sample is small, or when the Gaussian, shared-covariance assumption roughly holds, LDA tends to give more stable estimates.
What is the difference between bagging and boosting?
Bagging builds independent trees on bootstrap samples and averages them to reduce variance. Boosting builds trees sequentially, each correcting earlier errors, mainly to reduce bias.
Do I need to scale features for tree-based models?
No. Tree splits depend only on the order of values, so monotonic rescaling leaves them unchanged, unlike SVMs and other distance-based methods.
How does this module connect to Distributed Systems?
Module 7 trains the same model families on large data with Spark MLlib, where validation and tuning follow the same rules but run across a cluster.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 4 knowledge
You have covered validation and feature selection, margin-based and probabilistic classifiers, and tree ensembles. Check whether you can pick the right model and validation method for a scenario under time pressure.