Computer Vision: Classification, Detection & Segmentation

Q: What's the difference between detection and segmentation?

Detection localizes objects with bounding boxes and class labels — "there's a car here." Segmentation classifies at the pixel level: semantic segmentation labels every pixel by class (all "car" pixels), while instance segmentation produces a separate mask per object (car 1 vs car 2). Choose by how precise the spatial output must be — boxes for counting/locating, masks for exact shape/area.

Computer Vision

Computer vision is about turning pixels into decisions: labels (classification), boxes (detection), masks (segmentation), or embeddings (retrieval). Modern CV is overwhelmingly deep learning — but the part that determines whether a real system works is rarely the model architecture. It's data: label quality, dataset diversity, honest splits, and realistic evaluation.

That's the most important thing to internalize. Teams obsess over picking ResNet vs ViT when the gains almost always come from better labels, more representative data, and catching leakage. Start from a pretrained backbone, then pour your effort into the data.

TL;DR

CV maps pixels → labels, boxes, masks, or embeddings.
Transfer learning (pretrained backbone + fine-tune) is the default start.
Data dominates — labels, diversity, and split design beat model choice.
Plan deployment early — latency, device, and pre/post-processing.

Quick Example

Transfer learning — start from a pretrained model and fine-tune the head:

What You Can Build

Classification — "what is in this image?"
Object detection — "what objects, and where?" (boxes)
Segmentation — "which pixels belong to what?" (masks)
Keypoints / pose — skeleton tracking, landmarks
OCR / document vision — text extraction and layout
Similarity search — find similar images/products via embeddings

Core Concepts

CNNs and Vision Transformers

CNNs learn local patterns hierarchically (edges → textures → parts → objects). Many state-of-the-art models are now Vision Transformers (ViTs), but CNN intuition remains useful for understanding how features build up.

Transfer learning

Most teams start from a pretrained backbone and fine-tune — faster results with less data, better generalization, and far less compute than training from scratch.

Data dominates

For real systems, model choice matters less than label quality, dataset diversity, split design (no leakage), realistic augmentations, and consistent evaluation. Invest there first.

A Practical Project Checklist

Define the task and metric (accuracy, mAP, IoU, recall@K).
Decide label format (classes, boxes, masks) and tooling.
Design splits — avoid near-duplicates leaking across train/test.
Baseline with a pretrained model.
Error analysis — confusions, lighting, occlusions, small objects.
Iterate on data — collect, relabel, augment.
Deploy — latency targets, batch vs realtime, device constraints.

Modeling Overview

Classification — start with a ResNet/EfficientNet/ViT backbone; calibrate if you need trustworthy probabilities.
Detection — Faster R-CNN (two-stage, accurate) or YOLO (one-stage, fast); watch small-object performance (resolution, anchors/heads).
Segmentation — semantic (class per pixel, U-Net style) vs instance (object-specific masks, Mask R-CNN style).

Tooling & Augmentations

PyTorch — training flexibility and a large ecosystem.
TensorFlow/Keras — production-friendly pipelines in many stacks.
OpenCV — classical vision and preprocessing (resize, transforms, video IO).
Albumentations / torchvision transforms — augmentations.

Augmentations should reflect real-world variation — color/brightness/contrast, random crop/resize, blur/noise, plausible perspective/rotation.

⚠️ Avoid augmentations that change semantics — e.g. aggressive horizontal flips when text direction or left/right matters.

Evaluation Gotchas

Leakage via duplicates — the same object/photo appearing in both train and test inflates scores.
Imbalanced classes — rare classes need careful sampling and metrics.
mAP / IoU thresholds — know what your reported number actually measures.
Confidence thresholds — tune to your cost of false positives vs false negatives.

Deployment & Ethics

Latency budget — CPU vs GPU vs edge; model format — TorchScript/ONNX/TensorRT.
Pre/post-processing is often the bottleneck (resizing, non-max suppression).
Monitor drift — lighting, camera, and environment changes.
Ethics — document training-data sources and licensing, be cautious with face recognition/biometrics, and evaluate performance across environments and demographics. See MLOps.

Best Practices

Start from a pretrained backbone — rarely train from scratch.
Spend your effort on data — labels, diversity, leak-free splits.
Augment realistically — reflect deployment conditions, preserve semantics.
Do error analysis — find the failure modes (small objects, occlusion, lighting).
Budget for pre/post-processing in latency — it often dominates inference.

Common Mistakes

Duplicate leakage across splits

Optimizing the model while ignoring the data

FAQ

CNN or Vision Transformer — which should I use?

For most applications, start with a strong pretrained backbone of either family — both work well, and transfer learning matters more than the specific architecture. CNNs (ResNet/EfficientNet) are efficient and need less data; ViTs can edge them out at scale with enough data and compute. Pick a well-supported pretrained model in your framework and fine-tune; only revisit architecture after data and training are solid.

Why does "data dominate" model choice in computer vision?

Because modern pretrained backbones are already excellent feature extractors — the marginal gain from swapping architectures is usually small compared to the gain from better labels, more diverse and representative data, leak-free splits, and realistic augmentation. Real-world failures (missed small objects, bad performance in new lighting) trace to data gaps, not model capacity. Effort spent on data has the highest return.

What's the difference between detection and segmentation?

Detection localizes objects with bounding boxes and class labels — "there's a car here." Segmentation classifies at the pixel level: semantic segmentation labels every pixel by class (all "car" pixels), while instance segmentation produces a separate mask per object (car #1 vs car #2). Choose by how precise the spatial output must be — boxes for counting/locating, masks for exact shape/area.

How do I keep latency acceptable in production?

Profile the whole pipeline, not just the model — resizing, format conversion, and non-max suppression often cost more than inference. Pick a model sized to your latency budget and device (CPU/GPU/edge), export to an optimized format (ONNX/TensorRT/TorchScript), batch where possible, and consider lower precision (FP16/INT8). Monitor for drift as cameras and lighting change. See Performance Optimization.