Feature Engineering
Feature engineering transforms raw data into inputs that make models perform better. It's often where the real gains are: good features beat complex models, and the leverage comes from domain knowledge — encoding what you know about the problem into representations a model can use. A well-chosen feature can do more than a fancier algorithm ever will.
The work spans encoding categoricals, scaling numerics, extracting signal from dates and text, constructing interactions and aggregations, handling missing values, and selecting the features that actually help. Throughout, guard against leakage — features must use only information available at prediction time.
TL;DR
- Good features > complex models.
- Domain knowledge drives feature creation.
- Scale and encode features appropriately for the model.
- Feature selection reduces overfitting and noise.
Quick Example
One-hot encode a nominal category in one line:
What Makes a Good Feature?
- Informative — correlates with the target.
- Independent — low correlation with other features.
- Simple — easy to compute and understand.
- Robust — generalizes to unseen data.
Feature types drive the technique: numerical, categorical, ordinal, datetime, text, boolean.
Encoding Categoricals
⚠️ Target encoding leaks if computed over the whole dataset — compute it within cross-validation folds or on the training split only.
Scaling Numericals
Datetime & Cyclical Features
Interactions & Aggregations
Missing Values & Selection
Text Features
See NLP for richer text representations.
Best Practices
- Encode by type — one-hot for nominal, ordinal mapping for ordered, target encoding carefully.
- Fit transforms on training data only (inside a pipeline) to prevent leakage.
- Add missingness indicators — the absence is often signal.
- Use cyclical encoding for periodic features (hour, month, day-of-week).
- Select features to cut noise and overfitting; let importance and RFE guide you.
Common Mistakes
Leakage from fitting on the whole dataset
Label-encoding nominal categories
FAQ
When should I use one-hot vs label vs target encoding?
Use one-hot for nominal categories with no inherent order (color, country) — it avoids implying a false ranking. Use label/ordinal encoding only when categories are genuinely ordered (small < medium < large). Use target encoding (mean target per category) for high-cardinality categoricals where one-hot would explode dimensionality — but compute it within cross-validation folds to avoid leakage. Match the encoding to the variable's nature.
Why encode cyclical features with sine and cosine?
Because raw values misrepresent periodicity: encoding month as 1–12 tells the model December (12) and January (1) are far apart, when they're adjacent. Sine/cosine encoding maps the cycle onto a circle so the distance between consecutive periods is correct everywhere. This helps any feature with wraparound — hour of day, day of week, month, angle.
How is feature engineering different from feature selection?
Feature engineering creates new, more informative inputs (encodings, interactions, aggregations, extracted signals). Feature selection picks the subset of features that actually help, discarding noise and redundancy to reduce overfitting and speed up training. You typically engineer many candidate features, then select the useful ones via importance scores, recursive elimination, or correlation analysis.
Do deep learning models still need feature engineering?
Less for raw perceptual data — neural nets learn features from images, audio, and text themselves, which is much of their appeal. But for tabular data, hand-crafted features (aggregations, ratios, domain signals) still routinely beat or complement deep models, and gradient-boosted trees on good features remain state-of-the-art for many tabular problems. Feature engineering is far from obsolete; it just matters most outside raw perceptual domains.
Related Topics
- Python for Data Science — The tooling
- Model Evaluation — Measuring feature impact
- MLOps — Serving features in production
- NLP — Text feature extraction
- Data Engineering — Feature pipelines