Teleoperation and robot data collection

Why this is the bottleneck

Language models had a corpus lying on the internet. Robotics has nothing equivalent, and the difference shapes the entire field. Every unit of robot training data has to be produced by a robot, acting in the world, at wall-clock speed, supervised by a person, on hardware that wears out.

That single constraint explains most of the research agenda: cross-embodiment pooling exists because no one robot produces enough; simulation exists because real collection is slow; learning from human video is pursued because video is abundant. None of these has removed the bottleneck.

How teleoperation rigs work

• Leader-follower arms. The operator moves a smaller replica and the robot mirrors it. Gives direct, intuitive joint-space control and produces data in the robot's own action space, which is what makes it immediately usable. Requires purpose-built hardware per platform.
• VR and motion capture. The operator's hand pose is tracked and retargeted to the robot's end effector. Natural for the operator, and introduces a retargeting problem because human and robot kinematics differ.
• Game controllers and 3D mice. Cheap, available, and awkward for anything needing dexterity or feel.
• Kinesthetic teaching. Physically guiding the arm. Most direct for the operator; limited to backdrivable robots and reachable workspaces, and the operator's body occludes the cameras.
• Wearables and exoskeletons. Better mapping for multi-finger hands, more setup, more calibration.

The trade-offs that actually matter

Force feedback. Without it the operator cannot feel contact, so they demonstrate contact-rich tasks badly — pressing too hard, missing the moment of engagement, failing to exploit compliance. Demonstrations of insertion or deformable handling collected without feedback frequently teach the wrong thing. With it, the rig costs more and the engineering is harder.

Latency. Delay between command and observed motion degrades operator performance and, worse, teaches the policy timing that does not reflect the robot's actual dynamics.

Observation parity. The policy will see camera images. If the operator is working from a different viewpoint, or from direct line of sight, they are solving a different problem than the one the policy must solve, using information it will not have.

Action space. Data collected in end-effector space does not transfer cleanly to a policy acting in joint space, and vice versa. Decide this before collecting.

Data quality, which decides everything

• Consistency. Five hundred demonstrations done the same way beat two thousand done five different ways. Multiple operators with different strategies produce a policy that averages between them, which can be a strategy that works for nobody.
• Coverage of exceptions. Demonstrations usually capture the happy path, because the collection session is set up to go well. Reality is mostly exceptions, and the policy will meet them for the first time in production.
• Recovery behaviour. If failures and corrections were never demonstrated, the policy has no model of getting out of trouble — and its own small errors will put it there.
• Calibration drift. Data collected before and after a calibration change are different datasets wearing the same name.
• Provenance. Operator, date, hardware revision, calibration state, configuration. When a subset turns out to be bad, this is how you find it, and nobody ever regrets having recorded it.

Scaling attempts, and what they cost

• Parallel rigs. More stations, more operators. Linear scaling, linear cost, and a consistency problem that grows with the number of people.
• Cross-embodiment pooling. Combining data from many robot types, following the finding that such data transfers rather than conflicts. The dominant structural answer so far.
• Simulation. Cheap volume, with a reality gap that falls hardest on contact — which is most of manipulation.
• Human video. Abundant, unlabelled, and from the wrong body. Real research progress, not a pipeline.
• Autonomous collection. Letting robots gather their own data under supervision. Attractive and slow, and the data is only as good as the policy generating it.
• Play data. Unstructured interaction without a specific task, used for pre-training representations rather than policies.

The operator is part of the system

Collection throughput and data quality are both bounded by the person doing it, and this is systematically under-designed.

• Fatigue degrades data. Demonstrations late in a long session are measurably worse, and they go into the dataset carrying the same weight as the good ones.
• The interface sets the ceiling. An awkward rig produces awkward motion, and the policy learns the awkwardness faithfully.
• Training the operator matters. An operator who understands what the policy will have to reproduce makes different choices than one who is simply completing the task.
• Multiple operators need one agreed strategy. Otherwise the dataset contains several strategies and the policy learns a blend of them.
• Reset time dominates. In most collection setups, more wall-clock time goes to resetting the scene between attempts than to the demonstrations themselves. Designing the reset is as valuable as designing the rig.

Who owns the data

This is a commercial question that gets deferred until it is contentious, and it should not be. If most deployed performance comes from post-training on data collected at a customer's site, then that data is a significant asset.

The questions worth settling in writing before collection starts: who owns demonstrations recorded on a customer's premises; whether that data may be used to improve a product sold to their competitors; whether it leaves the site at all; what happens to it when the relationship ends; and what is recorded incidentally — faces, documents, processes — that nobody intended to collect.

An operator who asks these questions is not being difficult. They are asking the thing that will matter most in two years.

Practical advice

• Decide the action space and observation set before collecting anything.
• Give operators the same view the policy will have.
• Write down the strategy for the task and have every operator follow it.
• Deliberately collect perturbations and recoveries, not just clean runs.
• Record full observation streams, not a reduced trace — you will want to retrain differently.
• Budget for re-collection. The first pass teaches you what you should have collected.

What a collection session actually looks like

Descriptions of data collection skip the logistics, and the logistics are most of it. A realistic session involves setting up the scene, verifying calibration, running a handful of warm-up attempts that get discarded, then alternating between demonstration and reset for several hours.

The ratios are worth knowing before planning a programme. Reset frequently takes longer than the demonstration itself — placing objects back, clearing a dropped item, re-stacking a bin. A task that takes twelve seconds to demonstrate can take forty to reset, which means most of your collection budget buys reset time, not data. Teams that design the reset — fixtures that return items to position, a second person resetting while the first demonstrates, tasks chosen so the end state is a valid start state — collect several times more data from the same hours.

The other overlooked cost is verification. Demonstrations that were mis-recorded, dropped frames, or captured with a stale calibration are worse than no data, because they are in the dataset wearing the same label as the good ones. Checking as you go beats discovering it during training.

Recording practices that pay off later

• Record more streams than you think you need. Retraining with a different observation set is common, and you cannot add a camera retroactively.
• Timestamp everything from one clock. Reconciling two clocks after the fact is tedious and sometimes impossible.
• Store calibration with the episode, not alongside it. Calibration that lives in a separate file becomes detached from the data it describes.
• Mark attempts as you go — success, failure, aborted, discarded. Nobody reliably remembers which run was which a week later.
• Keep the failures. They are the only data you have about what going wrong looks like, which is exactly what failure detection needs.