The robot perception stack

What perception has to deliver

Perception's job is not to recognise objects. Its job is to produce, continuously, an estimate of the world that is accurate enough to act on and honest enough about its own uncertainty that the layers above can decide when not to act. The second half is the part that is routinely skipped, and it is the cause of a large share of confident physical failures.

The sensors, and how each one lies

• RGB cameras. Cheap, high resolution, and entirely dependent on lighting. They give no depth. Exposure changes when a door opens; motion blur appears exactly when the robot is moving fastest.
• Stereo depth. Infers depth from two views, which requires texture. It fails on blank walls, uniform floors and plain surfaces — the things warehouses are full of.
• Structured light. Projects a pattern and reads its deformation. Accurate at close range, degraded by sunlight and by other units projecting into the same space.
• Time-of-flight. Measures return time. Struggles with reflective and transparent surfaces, and suffers multi-path error in corners, where the signal arrives having bounced twice.
• Lidar. Long range and geometrically accurate, sparse vertically, expensive, and unreliable on glass and dark absorbing surfaces.
• Force and torque. The only sensor that reports contact directly, which makes it the only one that can confirm a grasp actually happened.
• Proprioception. Joint positions and velocities, accurate and only describing the robot, not the world.

A useful rule: transparent, reflective, dark and thin objects defeat most depth sensing. Glass, chrome, black plastic and sheet material are not edge cases in real environments, and a system that assumes valid depth everywhere will act on geometry that was never returned.

Calibration, the unglamorous cause of most failures

Every sensor measures in its own frame, and acting requires knowing the relationship between those frames and the gripper. That means hand-eye calibration, camera intrinsics, multi-sensor extrinsics, and the timing offsets between streams.

Calibration drifts. Mounts flex, screws settle, temperature changes dimensions, and something gets knocked. The resulting failures do not look like calibration failures — they look like the policy grasping consistently a few millimetres off, which invites weeks of model debugging. Check calibration before suspecting the model.

Timing is the version people forget. If the depth frame is twenty milliseconds older than the colour frame and the robot is moving, the fused result describes a world that never existed.

The processing chain

• Synchronisation. Aligning streams in time before anything is fused. Getting this wrong corrupts everything downstream and is invisible in static tests.
• Filtering. Removing noise, outliers and invalid returns — and recording that they were invalid rather than treating missing data as empty space. Absent depth is not free space, and systems that conflate them drive into things.
• Segmentation. Separating the scene into surfaces and candidate objects.
• Detection and classification. What the objects are. The part most associated with machine learning and usually not the bottleneck.
• Pose estimation. Where each object is, in full six degrees of freedom, to the precision the task needs. This is frequently the actual bottleneck.
• Tracking. Maintaining identity over time, so the object that moved is understood to be the same object.
• State estimation. Where the robot itself is, usually fusing proprioception with external sensing.
• Scene representation. The structure the planner consumes — occupancy, meshes, object lists, or learned features.

Uncertainty is the part that matters

A perception layer that reports only its best estimate forces every layer above it to treat that estimate as fact. The alternative is to propagate confidence: this pose, with this covariance; this region, unobserved rather than empty; this detection, below threshold.

That matters because the correct response to high uncertainty is usually to stop, look again, or ask — and a system that cannot represent uncertainty cannot choose any of those. It can only act. Most of the physical failures that look baffling afterwards were a confident action taken on a weak estimate.

Classical or learned?

Both, in practice, and the split is reasonably stable.

Learned methods dominate detection, classification, segmentation and increasingly pose estimation, because they handle visual variation that hand-written features never did. Classical methods remain standard for geometry, calibration, filtering, state estimation and sensor fusion, because the mathematics is well understood and the failure modes are analysable.

The deployed pattern is learned perception feeding classical geometry, with the learned part kept inside bounds the classical part enforces. That is an assurance decision as much as a performance one: you can reason about what a Kalman filter will do.

The latency budget

Perception is not free, and on a moving machine its cost is measured in how stale the world model is by the time anything acts on it. The chain accumulates: exposure, transfer, synchronisation, inference, post-processing, then whatever the planner adds. A perception stack that is accurate and two hundred milliseconds behind is describing where things were, which on a machine travelling at a metre per second is twenty centimetres of error nobody accounted for.

This produces real architectural pressure. Heavy models run at low rates and are supplemented by something fast and cheap — a tracker, a geometric filter, or a reactive safety layer that does not wait for the network to finish. The division is deliberate: the slow path understands the scene, the fast path keeps the machine from hitting things while the slow path thinks.

Budget it explicitly, measure it under realistic load rather than on an idle bench, and treat a latency regression as seriously as an accuracy regression. Teams instrument accuracy carefully and discover latency by accident.

Practical notes

• Record raw sensor streams, not just the processed output. You cannot debug a perception failure from a bounding box.
• Validate calibration on a schedule, and log when it was last done.
• Test under the lighting the site actually has, including the window at the wrong time of day.
• Treat invalid depth explicitly. Never let missing data mean free space.
• Check timestamp alignment first when fused results look subtly wrong.
• Measure pose accuracy against ground truth, not detection accuracy. The task needs the pose.

Multi-camera setups and what they buy

Most non-trivial systems end up with more than one camera, and the reasons are specific rather than general.

• Wrist cameras see the workspace from the gripper's point of view, which is the view that matters at the moment of grasping. They also move, which means motion blur and constantly changing extrinsics.
• Overhead or scene cameras give stable global context and are occluded by the robot itself at exactly the wrong moment.
• Multiple scene cameras reduce occlusion by viewing from different angles, at the cost of calibration between them and the question of which to believe when they disagree.

Disagreement is the interesting problem. Two cameras producing different poses for the same object is common, and resolving it requires either a principled fusion with uncertainty or an explicit priority rule. Systems that silently take the first answer, or average two estimates that should not be averaged, fail in ways that are hard to trace back to the camera.

Testing perception properly

Perception is tested badly more often than it is built badly. A few practices that separate systems that work from systems that worked in the lab:

• Test against ground-truth pose, not against detection accuracy. The task consumes pose; a correct detection with a wrong pose still fails.
• Build an adversarial object set deliberately: transparent, reflective, matte black, thin, deformable. These are not edge cases in a real building.
• Test the lighting you will actually have, at the times of day you will have it, including the one window.
• Record and replay real failures as regression tests, so a fix for one does not quietly break another.
• Measure latency end to end, not per component, because the sum is what the robot experiences.