Reinforcement learning in robotics

The idea

Reinforcement learning trains a policy by trial and error against a reward signal. The system acts, observes the result, receives a scalar reward, and adjusts to get more of it. No demonstrations required — which is the appeal, and the source of nearly every difficulty.

In robotics it is used far more narrowly than popular coverage suggests, and where it is used it is usually decisive.

Why it mostly happens in simulation

Reinforcement learning is sample-hungry. Learning a locomotion policy can take the equivalent of years of robot experience, and a real robot collects experience at wall-clock speed, needs supervision, and damages itself while exploring.

Simulation runs faster than real time, in thousands of parallel instances, and resets instantly. That is the only arrangement in which the sample requirement is affordable, which makes essentially all serious robot reinforcement learning simulation-first, with a transfer problem attached.

The consequence is that the quality of a reinforcement learning result is usually bounded by the quality of the simulation, and by how deliberately the reality gap was managed.

Where it genuinely wins

• Locomotion. The clearest success. Walking, running and recovering over varied terrain is hard to demonstrate, easy to reward, and well-served by massive parallel simulation. Modern legged robot control is largely a reinforcement learning achievement.
• Whole-body control and balance. Same reasons: high-dimensional, continuous, hard to specify by hand.
• Dynamic tasks. Behaviours where timing and momentum matter and a human demonstration is difficult to produce.
• Refining an imitation-trained policy. Starting from demonstrations and improving against a reward combines a good initialisation with the ability to exceed the demonstrator.

Where it is a poor fit

• Tasks whose success is hard to score. If you cannot write a reward, you cannot use the method. "Tidy the room" has no scalar.
• Contact-rich manipulation. Exactly where simulation is least faithful, so the transfer penalty is worst.
• Anything where exploration is unsafe. On real hardware, exploration means trying things that might break something.
• Long-horizon, sparse-reward tasks. If reward only arrives at the end of a hundred-step sequence, random exploration will essentially never find it.

Reward design is the real work

The method optimises what you wrote, not what you meant, and the gap between the two is where most practitioner time goes.

• Reward hacking. Policies reliably find behaviours that score well and are useless — exploiting a simulator's contact model, vibrating to accumulate a shaped term, or achieving the measured proxy while missing the goal entirely.
• Sparse versus shaped. Sparse rewards are honest and often unlearnable. Shaped rewards are learnable and encode your assumptions about how the task should be done, which may be wrong.
• Term balancing. Real reward functions combine progress, effort, smoothness, safety margins and constraint penalties. The weights interact, and tuning them is empirical.
• Specification by reward is a design decision, not a neutral description. It is worth writing down what you believe the reward captures, so the eventual surprise is diagnosable.

Sim-to-real, in brief

A policy trained in simulation usually underperforms on hardware, with contact, friction, actuator dynamics and latency the main culprits. The standard response is domain randomisation — train across wide ranges of physical parameters so the policy must be robust to all of them, making reality one more sample. It works, at the cost of conservatism, since robustness across a range trades away performance at any point in it.

This is covered properly in the sim-to-real guide, and it is the single largest determinant of whether a reinforcement learning result survives contact with hardware.

Reinforcement learning or imitation?

A reasonable default: if a human can demonstrate it, imitate; if a human cannot demonstrate it but you can score it, use reward; if both, combine them.

• Imitation gets to a working version faster, with far less data, and is bounded by the demonstrator.
• Reinforcement learning can exceed human performance and needs a reward plus a simulator plus patience.
• The hybrid — pre-train on demonstrations, refine with reward — is increasingly standard, because it inherits the sample efficiency of one and the ceiling of the other.

What the algorithms are, briefly

The algorithm choice matters less than the reward and the simulation, but the vocabulary is worth having.

• On-policy methods learn from data generated by the current policy. They are stable and comparatively easy to tune, which is why they dominate locomotion work, and they discard experience after use — acceptable when simulation is cheap.
• Off-policy methods reuse past experience from a buffer, making them far more sample-efficient and more delicate to tune. These are what you reach for when samples are expensive, which means on real hardware.
• Model-based methods learn a model of the dynamics and plan against it, improving sample efficiency at the cost of compounding any error in the learned model.
• Offline reinforcement learning trains from a fixed dataset with no further interaction. Attractive for robotics because collection is the bottleneck, and constrained by not being able to try the thing it is uncertain about.

Exploration on real hardware

Exploration means doing things that might be wrong, and on a physical machine wrong can mean damaged. This is the reason real-robot reinforcement learning is rare outside carefully bounded setups.

Where it is done, it is made survivable rather than safe by construction: tight joint and velocity limits, soft or compliant tooling, automatic reset mechanisms so a human does not have to intervene thousands of times, and a supervisory layer that terminates an episode before a limit is reached rather than after. Those constraints also bound what can be learned, which is the trade being made.

What to ask about a reinforcement learning result

• Was it evaluated on hardware, and on how many trials?
• What was the reward function, and what did it turn out to incentivise?
• How much domain randomisation was applied, and what does that cost in performance?
• Was the real evaluation in the environment the simulation modelled, or a different one?
• How much real fine-tuning was involved, and is the result still fairly described as sim-to-real?
• What does the policy do when it encounters something outside the training distribution?

Reward hacking, with examples worth recognising

Every practitioner has a collection of these, and they are instructive because the policy is never malfunctioning — it is optimising exactly what was written.

• A locomotion policy rewarded for forward velocity that learns to fall forward repeatedly, because falling is fast and the reward said nothing about remaining upright.
• A policy rewarded for reducing distance to a target that learns to hover just short of it, because arriving ends the episode and the shaped reward was worth more than completion.
• A manipulation policy that exploits the simulator's contact solver to hold an object through a surface, a behaviour with no physical analogue that scores perfectly.
• A policy rewarded for a sensor reading rather than the underlying state, which learns to satisfy the sensor — pressing a contact switch without having actually grasped anything.
• Vibration and jitter that accumulate a small per-timestep reward, producing behaviour that is useless and well-scored.

The lesson is not that reward design is hard, though it is. It is that the policy is a faithful reading of your specification, and the specification was wrong. The practical defence is to watch the resulting behaviour rather than the reward curve, because a rising curve is consistent with both learning the task and learning to game it.

Where to start if you are new to it

A reasonable sequence, in order of ambition: reproduce a known locomotion result in simulation before attempting anything original, because the tooling and the parallelism are most of the difficulty; instrument behaviour visually from the beginning rather than relying on reward curves; keep the first reward function sparse and honest to see what the task actually requires before shaping it; and do not attempt hardware transfer until the simulated result is robust across a wide randomisation range, because transferring a brittle policy only teaches you that it was brittle.