Sensor Calibration & Time Synchronization for Robots

Sensor Calibration and Time Synchronization

A robot with a camera, a lidar, and an IMU has three different views of the world, each in its own coordinate frame and on its own clock. Fusion, localization, and obstacle detection all assume those views can be lined up: that the robot knows exactly where each sensor sits, how each sensor's measurements relate to geometry, and when each measurement was taken. Small errors here masquerade as algorithm bugs — maps that smear when the robot turns, obstacles that appear a hand's width from where they are, filters that slowly diverge.

Calibration estimates the parameters that relate sensors to geometry (intrinsics) and to each other (extrinsics). Time synchronization puts all measurements on a common clock with known latency. Both must be done carefully, verified with numbers, and re-checked over time, because mounts loosen, lenses shift, and clocks drift.

TL;DR

Intrinsic calibration models each sensor (camera focal length, principal point, distortion; IMU bias and scale); extrinsic calibration gives poses between sensors and the robot.
Calibrate cameras with checkerboard or ChArUco targets and check reprojection error and coverage of the image.
Calibrate camera–lidar, camera–IMU, and hand–eye extrinsics with dedicated tools, then validate with overlays and consistency checks.
Put sensors on one clock: hardware triggers, PTP with hardware timestamps, or at least NTP/chrony, and timestamp at capture.
Estimate and compensate time offsets and latencies; keep calibration uncertainty with the parameters.
Detect drift with recorded datasets, automated reprojection and alignment checks, and fleet monitoring.

Quick Example

Camera intrinsic calibration with OpenCV from images of a checkerboard:

Estimating a constant time offset between two sensors by cross-correlating motion signals — for example gyroscope angular speed and rotation speed estimated from camera images, resampled on a common time base:

Core Concepts

Intrinsic Calibration

For cameras, cover the whole image — especially corners, where distortion is strongest — with the target at varied distances and angles, and check reprojection error (often well below one pixel for a good calibration) along with per-view errors to find bad images. A low error on poorly distributed views can still give a wrong calibration.

Extrinsic Calibration

Extrinsics are the rigid transforms between sensors and the robot body, used in the transform tree (see Robot Perception and Sensing).

Validate visually (lidar points overlaid on images, aligned point clouds from different sensors) and numerically (residuals, consistency across sessions). Record which calibration version each recording used.

Clocks and Time Synchronization

Each device has its own clock, and messages arrive with transport and processing delays. Strategies, from strongest to weakest:

Timestamp data at capture (exposure midpoint for cameras, per-point time for spinning lidars), not on arrival, and compensate known latencies. Remaining constant offsets can be estimated from motion signals, as in the quick example, or by calibration tools. See Time Synchronization and Certificate Dependencies for clock fundamentals.

Uncertainty

Calibration results are estimates with uncertainty. Keep covariances or at least residual statistics, propagate them into fusion (an extrinsic rotation error of 0.5 degrees displaces points by about 9 cm at 10 m), and inflate measurement noise when calibration is uncertain. See Probability and Linear Algebra for Computing.

Detecting Drift

Calibration changes after impacts, temperature cycles, vibration, and maintenance. Detect it before it corrupts behavior:

Recorded validation datasets re-run after hardware work and software changes, comparing residuals with baselines.
Online checks: reprojection consistency between camera and lidar edges, stereo epipolar error, agreement between odometry sources.
Fleet monitoring of calibration metrics per robot, with thresholds that trigger recalibration.
Version control of calibration files with robot serial numbers and dates.

Localization and mapping quality often degrades first when calibration drifts; see Robot Localization and Mapping.

Best Practices

Calibrate With Coverage and Verify With Numbers

Collect varied views, inspect per-view errors, and validate with independent data.

Use Hardware Time Where It Matters

Trigger or PTP-synchronize sensors that are fused tightly, and timestamp at capture.

Version and Attach Calibration

Store calibration per robot and per sensor with history, and record which version produced each dataset.

Monitor for Drift Continuously

Automate checks on recorded and live data and recalibrate when thresholds are exceeded.

Common Mistakes

Calibrating Only the Image Center

Arrival-Time Timestamps

Calibrations Without History

FAQ

What is a good reprojection error for camera calibration?

Commonly well below one pixel for typical cameras with good target detection, but the distribution of views matters as much as the number. Check per-view errors and validate on independent images.

How do I calibrate a camera to a lidar?

Use targets visible to both sensors, such as boards or spheres, or targetless methods that align edges and planes, then verify by projecting lidar points into images and checking alignment across distances.

What is PTP?

The Precision Time Protocol (IEEE 1588), which synchronizes clocks over a network with hardware timestamping, typically achieving microsecond-level or better accuracy — suitable for synchronizing sensors and computers.

Why do I need to know the time offset between sensors?

Because fusion assumes measurements describe the same moment. When the robot moves, an unknown offset translates into position and orientation errors proportional to the motion.

How often should robots be recalibrated?

When monitoring shows drift beyond thresholds, after impacts or hardware maintenance, and on a schedule appropriate to the environment — supported by automated checks rather than fixed intervals alone.