Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Module 1.7 — Feature Engineering & The Data Reality

Mode: curated breadth Gate: concept check Est. effort: 6–10 focused hours

Why this matters

Tutorials hand you clean data. Reality does not. The data you actually face has missing values, inconsistent categories, skewed distributions, mixed types, outliers, and a class you care about that makes up two percent of the rows. On tabular problems, the work of turning that mess into informative features routinely matters more than the choice of model. Two practitioners with the same gradient-boosting library and different feature work will get very different results, and the gap is usually feature work, not hyperparameters.

This is the least glamorous module in the phase and one of the most valuable. It is curated breadth with a concept-check gate, because the skill is built through repeated practice on real datasets rather than a single from-scratch build. The goal is to internalize the menu of transformations and, more importantly, the judgment for when each applies and how to apply it without leaking information, which connects directly back to the previous module.

What you will be able to do

By the end of this module you will be able to:

Prerequisites

Curated path

  1. Kaggle Learn, the Feature Engineering and Data Cleaning courses. Practical, hands-on, and focused on real tabular problems: missing values, categorical encodings, scaling, and creating features that help tree and linear models. Free, with exercises. https://www.kaggle.com/learn/feature-engineering and https://www.kaggle.com/learn/data-cleaning

  2. scikit-learn user guide, preprocessing and pipelines. The reference for encoders, scalers, imputers, and column transformers, and crucially how to compose them into a Pipeline so every step is fit on training data only. This is the mechanism that prevents the leakage from the previous module. https://scikit-learn.org/stable/modules/preprocessing.html and https://scikit-learn.org/stable/modules/compose.html

  3. A grounded treatment of class imbalance. Read the imbalanced-learn documentation’s introduction and the guidance on resampling (SMOTE and friends) versus class weights, along with the warning that resampling must happen inside cross-validation, never before the split. https://imbalanced-learn.org/stable/

  4. Feature engineering for categorical data: target and other encodings. Read the scikit-learn TargetEncoder documentation, which explains the technique for high-cardinality categories and the cross-fitting that makes it safe from leakage. https://scikit-learn.org/stable/modules/preprocessing.html#target-encoder

Deliberately skip for now: automated feature engineering tools and heavy domain-specific feature libraries. Build the manual judgment first; automation is only useful once you know what good features look like.

Background: the ideas in words

Missing data is information, not just absence. Sometimes a value is missing at random; sometimes the fact that it is missing is itself predictive (an unanswered income field may correlate with the target). Reflexively dropping rows discards data and can bias the sample; filling with the mean distorts the distribution. Better approaches include model-based imputation and adding an explicit “was missing” indicator so the model can use the missingness signal. Tree-based models can often handle missing values natively.

Categorical encoding turns categories into numbers a model can use. One-hot encoding is the safe default for low-cardinality categories. For high-cardinality categories (zip codes, product IDs), one-hot explodes the feature space, and target encoding, replacing a category with a smoothed average of the target for that category, is powerful but dangerous: done naively it leaks the target, so it must be cross-fitted.

Numerical transformations matter for some models and not others. Linear models and SVMs need features on comparable scales and often benefit from transforming skewed features (a log transform, for instance); tree-based models are invariant to monotonic rescaling and need neither. Knowing which model cares saves wasted effort.

Class imbalance distorts both training and evaluation. The fixes include class weights (telling the model to care more about the rare class), resampling the data (oversampling the minority with techniques like SMOTE, or undersampling the majority), and, always, evaluating with metrics suited to imbalance from Module 1.6. The cardinal rule: any resampling happens inside the cross-validation loop on training folds only, or you leak and fool yourself.

The pipeline ties it together. Every transformation, imputation, encoding, scaling, resampling, must be fit on training data and merely applied to validation and test data. A Pipeline enforces this automatically, which is why it is the single most important habit in applied tabular machine learning.

Knowledge check

  1. Give two reasons that dropping rows with missing values can be a mistake, and a better alternative for each.

  2. When is one-hot encoding a poor choice, and what would you use instead? What makes that alternative risky?

  3. Which kinds of models require feature scaling and which do not, and why?

  4. Why must target encoding be cross-fitted, and what leaks if it is not?

  5. Name two ways to handle class imbalance and the one rule you must never break when resampling.

  6. Why is a scikit-learn Pipeline the right tool for preventing preprocessing leakage?

Project

A feature-engineering study on a genuinely messy dataset:

Definition of done: the two pipelines, a fair comparison using leak-free evaluation, and a write-up quantifying the value of the feature work and naming the highest-impact decision.

The workshop: ship it

Build this in its own repository, modelwright-feature-engineering.

  1. Set up the project:

mkdir modelwright-feature-engineering && cd modelwright-feature-engineering
uv init && uv add pandas scikit-learn imbalanced-learn matplotlib && mkdir src notebooks

  1. Build the two pipelines from the project (naive and considered), and compare them with the evaluation harness you built in Module 1.6 (install it or copy it in).

  2. Commit at checkpoints: “Naive pipeline baseline”, then “Considered pipeline”, then “Fair comparison”.

  3. Add a README quantifying how much the feature work moved the honest metric, and which single change mattered most.

  4. Ship it:

gh repo create modelwright-feature-engineering --public --source=. --push

(No gh? Create an empty public repo, then git remote add origin <url> and git push -u origin main.)

Done when: modelwright-feature-engineering is on GitHub, with the two pipelines, a leak-free comparison, and a write-up naming the highest-impact decision.

Going deeper (optional)

Canonical references

Phase 1 — Classical Machine Learning
Module 1.6 — Evaluation, Calibration & Leakage
Phase 2 — Deep Learning
Phase 2 — Deep Learning