Robot Perception & Sensing: Cameras, Lidar, IMUs

Robot Perception and Sensing

Perception is how a robot turns raw signals — pixels, laser ranges, accelerations, wheel ticks — into information it can act on: where obstacles are, what objects are present, how the ground slopes, where a person is walking. It is usually the most computationally expensive and least predictable part of an autonomy stack, because the physical world is endlessly varied: glare and darkness, rain and dust, glass and mirrors, clutter and crowds.

Good perception engineering starts below the algorithms. Sensors must be chosen for the environment, mounted and calibrated precisely, timestamped consistently, and placed in a correct transform tree. Pipelines must report confidence, not just answers, and every failure seen in the field must be captured in recorded data so it can be reproduced and fixed.

TL;DR

Choose sensors for the environment: cameras for appearance, lidar and depth sensors for geometry, radar for weather, IMUs and encoders for motion, GNSS outdoors.
Every sensor needs intrinsic and extrinsic calibration, accurate timestamps, and a place in the transform tree.
Pipelines detect, segment, and track objects in images and point clouds, then express results in robot frames.
Fuse sensors early (raw data) or late (object lists), and always carry uncertainty and confidence.
Log raw sensor data (rosbag, MCAP), replay it for debugging, and curate datasets from field failures.
Fit perception into compute and latency budgets, and design behavior for when perception is uncertain.

Quick Example

Projecting lidar points into a camera image using intrinsic and extrinsic calibration — the basic operation behind camera–lidar fusion and calibration checks:

Overlaying projected points on the image is a fast visual check: if lidar returns from a pole do not line up with the pole in the image, the extrinsics or timestamps are wrong.

Core Concepts

Sensors and Their Failure Modes

Mount sensors to cover blind spots, protect them from dirt and vibration, and consider the rolling shutter of many cameras, which distorts fast motion.

Calibration, Time, and Transforms

Perception only works when the robot knows where each sensor is (extrinsics), how each sensor maps the world to measurements (intrinsics), and when each measurement happened. A 50 ms timestamp error at 2 m/s places an obstacle 10 cm away from where it is. Sensor data lives in each sensor's frame and is transformed into robot and map frames through a time-aware transform tree. See Sensor Calibration and Time Synchronization.

Perception Pipelines

Neural networks dominate image perception; classical geometric methods remain strong for point clouds and are easier to bound. See Computer Vision.

Fusion and Uncertainty

Early fusion combines raw data (colorized point clouds, depth-augmented images) and can exploit more information; late fusion combines per-sensor results and is easier to debug and degrade gracefully when one sensor fails. Either way, outputs should carry uncertainty — covariances for positions, confidence scores for classes — and downstream planning should treat low-confidence or missing information conservatively.

Data: Logging, Replay, and Datasets

Record raw sensor streams with timestamps and transforms (rosbag2 with the MCAP format is common), triggered on interventions, near-misses, and anomalies.
Replay recordings to reproduce field failures deterministically and to regression-test new perception versions.
Curate datasets that cover lighting, weather, environments, and rare objects, with labeling guidelines and quality checks.
Evaluate with task-relevant metrics — detection precision and recall by distance and class, false obstacle rate, missed obstacle rate — sliced by conditions.

Compute and Latency

Perception runs on embedded CPUs, GPUs, or accelerators with power and thermal limits. Budget end-to-end latency from capture to decision, quantize and optimize models, downsample point clouds, and run expensive models at lower rates than safety-critical checks. Measure latency distributions, not averages, and degrade gracefully under load. See PyTorch Model Deployment.

Best Practices

Start From the Environment

List lighting, weather, surfaces, and objects the robot will face, and choose sensors and placements that cover them.

Treat Calibration and Time as Part of Perception

Verify extrinsics and timestamps continuously, not only at installation.

Report Confidence Downstream

Publish uncertainty with every output and make planners and behaviors use it.

Build a Data Flywheel

Capture field failures automatically, add them to datasets, and gate releases on regression results.

Common Mistakes

Unsynchronized Timestamps

Ignoring Sensor Blind Spots

Lab-Only Datasets

FAQ

Should a robot use cameras or lidar?

It depends on the task and environment. Cameras give rich appearance at low cost; lidar gives accurate geometry independent of lighting. Many robots combine both, plus short-range sensors for safety.

What are intrinsic and extrinsic calibration?

Intrinsic calibration describes how a sensor maps the world to measurements, such as a camera's focal length and lens distortion. Extrinsic calibration describes the sensor's position and orientation relative to the robot or other sensors.

Why do timestamps matter so much?

Because the robot and objects move. Measurements combined with incorrect timestamps are placed in the wrong position, corrupting fusion, localization, and obstacle detection.

What is sensor fusion?

Combining data from multiple sensors to produce more accurate, complete, or robust information than any single sensor provides, either by merging raw data or by combining each sensor's results.

How do I debug perception problems from the field?

Record raw sensor data with timestamps and transforms, replay it offline to reproduce the issue, inspect intermediate outputs, and add the case to a regression dataset once fixed.