The humanoid robot software stack

Why the stack framing matters

A humanoid robot is not short of capability in the mechanical sense. Modern platforms have the degrees of freedom, the actuation and the sensing to do a great deal. What they lack is the software that decides what to do and the assurance that it will be done safely. Describing that software as a stack is useful because it makes the responsibilities explicit, and because most integration failures are boundary failures — two layers that each work, disagreeing about who owns something.

Layer 1: hardware abstraction

At the bottom sits the interface to the machine: joint commands, state feedback, sensor streams, safety signals, and the mode logic that says what the robot will currently accept.

The central design judgement in the entire stack is made here. An abstraction that hides differences between robots makes capability portable and makes robots unsafe, because the differences it hides include joint limits, thermal constraints, control rates and failure semantics. An abstraction that hides nothing is not an abstraction. Deciding what must stay visible is the architectural decision everything else inherits.

Layer 2: perception and state estimation

Turning raw sensor data into an estimate of the world and of the robot's own configuration. In practice this layer includes calibration, synchronisation, object detection and pose estimation, scene segmentation, and localisation.

Two things are routinely underweighted. The first is calibration drift: a stack that assumes a fixed camera-to-base transform will degrade as the machine is used, and the degradation looks like a policy problem. The second is uncertainty. A perception layer that reports only its best guess forces every layer above it to behave as if that guess were correct, which is the mechanism behind a large fraction of confident physical failures.

Layer 3: representation

Some account of the scene that decision-making can act on: objects and their poses, surfaces, free space, affordances, constraints, and task state. This layer is where approaches diverge most sharply. Classical stacks maintain an explicit world model. End-to-end learned systems often maintain an implicit one inside the network, which is more flexible and considerably harder to inspect when something goes wrong.

Layer 4: policy and skills

The part most people mean when they say robot intelligence: the component that decides what action to take. The landscape here, as of 2026, spans three broad approaches.

• Engineered pipelines. Hand-written perception, grasp planning and motion generation. Predictable, inspectable, and brittle at the edges of their assumptions.
• Learned policies, including vision-language-action models. Trained from demonstration and interaction, these generalise across variation better and are far harder to characterise, verify or debug.
• Hybrid. Learned perception and grasp proposal feeding classical planning and control, with learned components kept inside bounds a classical layer enforces. This is where most deployed systems actually sit, for reasons of assurance rather than fashion.

Layer 5: task planning and sequencing

Turning a goal into an ordered set of skills, monitoring whether each one succeeded, and deciding what to do when one did not. The hardest part is not sequencing; it is success detection. Knowing that the previous skill actually achieved its effect is a perception problem that tends to be less developed than the skills themselves, and it is what makes long-horizon tasks compound error.

Layer 6: motion planning and control

Generating trajectories that respect kinematic limits, dynamics, balance and collision constraints, then executing them stably in real time. For a humanoid this layer carries work a wheeled platform does not: whole-body coordination, balance under disturbance, and the fact that a planning failure can mean falling over rather than stopping.

This is also the layer with the firmest real-time requirements, and the one most likely to be owned by the robot manufacturer rather than by a third party.

Layer 7: safety supervision

A layer whose only job is to stop the machine, and whose correctness must not depend on the layers it is supervising. Its responsibilities include enforcing operating envelopes, monitoring for conditions that invalidate the current plan, and executing a safe stop.

Two principles are non-negotiable. Safety authority stays with the machine and the party responsible for operating it — a third-party capability layer does not get to overrule an interlock. And a capability without a well-specified refusal behaviour is not deployable regardless of how well it performs on average.

Layer 8: operations

The layer that is usually missing entirely from stack diagrams and is decisive in practice: deployment and versioning, staged rollout, immediate and tested rollback, observability, and fleet management.

Shipping software to a machine that moves is not shipping to a server. The machine needs to know precisely what version it is running, to refuse an update that would put it outside its validated envelope, and to produce enough telemetry to reconstruct what it perceived, what it decided, what it did, and where those diverged. Instrumentation of that kind has to be designed in; it cannot be added after the first confusing failure.

Where the boundaries actually cause trouble

• Abstraction and safety. When the capability layer believes it may do something the platform considers out of bounds.
• Perception and policy. When uncertainty is dropped at the interface and the policy acts on a point estimate.
• Policy and control. When a learned policy emits commands that are dynamically infeasible and the control layer has to reconcile them.
• Capability and operations. When a capability is validated in one configuration and deployed in another, because nothing enforced the match.
• Vendor and integrator. When a task fails and nobody can say whether it was hardware, capability or environment, because no layer records enough to tell.

Who owns which layer in a real arrangement

In any actual integration these layers are owned by different parties, and writing that down early prevents most of the disputes later.

• The manufacturer almost always owns hardware abstraction, low-level control, balance where relevant, and the safety supervision that cannot be delegated. These are warranty and liability surfaces, and no serious platform hands them over.
• The capability provider typically owns perception, representation, policy and task sequencing, operating inside limits the platform enforces.
• The operator owns the environment, the task definition, site safety, and the decision about what the machine is permitted to do.
• Operations is the contested one. Someone has to own versioning, rollout, rollback and telemetry, and it is frequently assumed to be somebody else's job until an incident proves otherwise.

The questions worth settling before any code is written: who can authorise a deployment, who can halt one, who holds the telemetry, who triages a failure, and who the customer calls when a task stops working. A stack diagram that does not answer those is an architecture sketch rather than a plan.

What portability would require

For a capability to move between robots, it has to carry more than weights. It needs its preconditions, the hardware class it assumes, the sensing it requires, the envelope it is safe inside, its failure behaviour, and evidence of how it performed and where. Almost none of that has an agreed representation today, which is why capability currently stays with the team that built it.

That gap — not model quality — is the reason the same problems are solved repeatedly across the industry.