Robot pilot acceptance criteria and scorecard

Published 14 November 2026 · scorecard version 0.1 · published and edited by Randy Levine for Synthari AI LLC. This page has not been reviewed by a qualified robotics professional and is not a safety assessment. Every figure on it is hypothetical. Corrections and disagreements are welcome at support@embodiedsoftwarelabs.com.

The number everyone gets wrong

Take a hypothetical pilot that reports two figures: 190 completed tasks out of 200 attempts — 95% completion, and 20 human interventions across those same 200 attempts — 10 interventions per 100 attempts.

Those look contradictory. If only 10 attempts failed, how were twenty interventions needed? They are not contradictory, and the reason is the whole reason this page exists.

Fourteen interventions landed on tasks that then completed. A person stepped in, the attempt recovered, and it counts inside the 190. So an intervention is not the same thing as a failed task.

Four failures involved nobody at all. The capability refused before moving, because the object was outside its stated envelope, so the task did not complete and no human was required. So a failed task is not the same thing as an intervention.

Both rates are correct. They are answering different questions, and a pilot that reports one of them alone has told you half of what happened. There is also a third figure sitting free in those four cells that almost nobody calculates: 14 of the 20 interventions rescued the attempt, so interventions succeeded 70% of the time. That says something quite different about the capability than either headline number does.

And none of these thresholds is universally acceptable. 95% completion is excellent where a failure means re-presenting a tote; it is unacceptable where a failure means damaging what is being handled or stopping a line. The number does not carry its own interpretation — whoever operates the site supplies that.

1. Define the task, site and baseline

Write the task as an action on an object in a place, name the robot, the end effector and the versions, and state who is nearby while it runs. Then the field that is most often skipped and most often fatal: who owns the risk assessment, and has it been done. That belongs before a pilot, not among its outcomes.

The baseline is the half of this section that gets dropped, and a pilot without one produces a number with nothing to compare it against. How is the task done today, what is the current throughput measured the same way the pilot will be measured, what is the current error rate, defined how, who measured it and when.

One trap worth naming: a baseline measured on a good week against a pilot run on a bad one is the commonest way a pilot produces a misleading answer. Record the period each came from, and check that the shift pattern and product mix match. If no baseline exists, write that down — it is a legitimate state and a much better one than an invented figure.

2. Specify what counts as a completed task

This section decides what every later number means, so it comes before any of them.

Write completion as one observable sentence, then say who observes it, whether the judgement is made by a person or by the system, and — if they disagree — which wins. Say what counts as started, because an attempt that never began is not a failure. Say whether a partial result is a completion, a failure or an exclusion.

Then the one that gets argued about afterwards: is a clean refusal before moving a failed task? A robot that declines to act because the object is outside its stated envelope has not malfunctioned — it did exactly what it said it would. Either answer is defensible. Neither is defensible if it is decided after the numbers are in.

The scorecard asks for a table of every outcome a single attempt can have, with the definition used and whether it counts toward completion and toward interventions. An outcome missing from that table is an outcome nobody will know how to record when it happens at eleven at night.

3. Measure interventions and recovery time

Define an intervention in one sentence, then settle the boundaries that make it countable: does touching the controls count, does touching the object count, does watching closely count, and is a single incident that takes three touches one intervention or three.

Recovery time needs two endpoints. Measured from what, to what — and is the operator's travel time to the cell included? It usually is not measured and it is usually most of the number.

Count intervention types separately rather than reporting one total: re-presented the object, cleared an obstruction, restarted the capability, restarted the controller, completed it by hand, called maintenance, stopped the line. Those have different causes and different fixes, and a single total hides which one you actually have.

One more field, and it is the one that quietly decides whether the data is any good: are interventions recorded at the time, or reconstructed at the end of a shift? Reconstructed counts are low, consistently, and always in the same direction.

4. Set the test window and sampling rules

How many attempts, over how many days, across how many shifts, how many physical robots and how many operators. Consecutive or sampled — and if sampled, by what rule.

Then the questions that decide whether the window represents the site: does it include a Monday and a Friday, does it include a night shift, and does it include the real product mix or only the easy variant. A window that runs day shift only, on the primary product, on the robot commissioned most recently, will produce a number. It will not produce an answer about the site.

And one governance field: who may stop the pilot early, and on what grounds. Agreeing that while things are going well is much easier than agreeing it in week three.

5. Record exceptions and excluded conditions

Exclusions are legitimate. An upstream line stopping is not the capability's fault, and counting it as one would understate the result. The problem is not that exclusions exist — it is that they are usually written down after somebody has seen which way the numbers went.

Agree the exclusion list before the pilot starts, with a named person and a date against each one. The scorecard lists the eight that come up most, with columns for whether it is excluded, why, who agreed and when.

Then three rules that protect the result: who decides an attempt is excluded and whether that decision is made at the time or in review; whether excluded attempts are still logged, which they should be; and — the field that does the real work — the maximum share of attempts that may be excluded before the result is void. Without a cap, enough exclusions can turn any pilot into a success.

6. Agree signoff and unresolved issues

One decision, from four: proceed, proceed with conditions, repeat, or stop. Two signatures, one each side, and a date. Plus a check that the risk assessment covers whatever the next step is, because a pilot's successor is usually larger than the pilot.

Then the honest part. Every unresolved issue, carried forward with an owner and a must be resolved before condition — because an issue closed on the grounds that the pilot ended is not closed, it is forgotten. And three closing fields that are worth more than the scorecard above them: what was not tested at all, what we would do differently, and what would have to be true to run this at scale.

What a scorecard row needs

The scorecard is one row per metric, and the columns are the point. A metric with a result and a threshold but no definition cannot be checked by anybody who was not standing there.

Not measurable is a real outcome and belongs in the list. A pilot that could not measure something it intended to measure has learned a fact about its own instrumentation, and recording that is more useful than quietly dropping the row.

References

Every figure on this page is invented and the arithmetic is self-contained — it follows from the numbers shown and from nothing else. There is no published study behind it, and it is illustrative arithmetic rather than field evidence. The related resources on this site:

• Robot capability specification template
  Section 5 of that template covers test evidence and the definitions a metric has to carry. The scorecard here is what you fill in while running the test; the specification is what you fill in afterwards to describe what you found.
• Robot software integration requirements checklist
  Section 6 of that questionnaire collects acceptance criteria before a project is scoped. This page is the same question asked again at a finer resolution, once a pilot is actually being planned.
• CC BY 4.0
  The licence this scorecard is published under.