# Robot pilot acceptance scorecard **Version 0.1 — 14 November 2026** Published by Embodied Software Labs (Synthari AI LLC). Licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Use it, change it, publish your filled-in copy. No attribution is required in the filled-in document itself. --- ## Read this first **Fill this in before the pilot starts, not after.** A scorecard written once the numbers are in is not a scorecard; it is a negotiation. Every threshold in it should be agreed by a named person while nobody yet knows whether it will be met. **This is not a safety assessment.** Nothing here determines whether an installation is safe. Safety requirements are set separately, by a competent assessor, for your specific application — before the pilot runs, not as one of its outcomes. A pilot that meets every metric below and has no risk assessment behind it has not succeeded at anything. **No threshold in this document is universally acceptable.** The examples use 95% and 10 interventions per 100 attempts because they are round numbers, not because they are good ones. Whether a number is good depends entirely on what a failure costs at your site, and that is a judgement for whoever operates it. **Three rules while filling it in:** 1. **Every metric carries its own definition.** A number without the definition used is not evidence. Write the numerator, the denominator and who decides each one. 2. **Agree the threshold and the owner at the same time.** A threshold with no named owner gets renegotiated by whoever is most tired at the end. 3. **Write the exclusions down in advance.** Conditions excluded after the fact are how a pilot produces a number nobody believes. --- ## 1. The task, the site and the baseline | Field | Answer | | --- | --- | | 1.1 The task, as an action on an object in a place | | | 1.2 Where exactly — line, cell, shift | | | 1.3 Which robot, which end effector, which versions | | | 1.4 Who is nearby while it runs | | | 1.5 Who owns the risk assessment, and is it done? | | | 1.6 Pilot start date | | | 1.7 Pilot end date | | **1.8 — The baseline.** What happens today, measured the same way the pilot will be measured. Without this the pilot has nothing to be compared against. | Field | Answer | | --- | --- | | How the task is done today | | | Current throughput, measured how, over what period | | | Current error or rework rate, defined how | | | Who measured it, and when | | | Is the baseline from the same shift pattern and product mix as the pilot? | | | If no baseline exists, say so here | | > A baseline measured on a good week and compared against a pilot run on a bad one is the > commonest way a pilot produces a misleading answer. Record the period both came from. --- ## 2. What counts as a completed task This is the section that decides what every later number means. | Field | Answer | | --- | --- | | 2.1 A task is complete when… *(one sentence, observable)* | | | 2.2 Who observes it | | | 2.3 Is completion judged by a person, by the system, or both? | | | 2.4 If both disagree, which wins? | | | 2.5 What counts as started? *(an attempt that never began is not a failure)* | | | 2.6 Is a partial result a completion, a failure, or excluded? | | | 2.7 Is a clean refusal before moving a failure, or a separate category? | | **2.8 — The result categories.** Every outcome a single attempt can have. If an outcome is not in this list, nobody will know what to do with it when it happens. | Category | Definition used | Counts as a completion? | Counts as an intervention? | | --- | --- | --- | --- | | Completed, no human involvement | | | | | Completed after a human stepped in | | | | | Failed, a human stepped in | | | | | Failed, nobody involved — clean refusal | | | | | Refused before starting — precondition not met | | | | | Timed out | | | | | Excluded — see section 5 | | | | > 2.7 is the field people argue about afterwards. A robot that declines to act because > the tote is outside its stated envelope has not malfunctioned. Decide now whether that > is a failed task, and either answer is defensible as long as it is written down first. --- ## 3. Interventions and recovery | Field | Answer | | --- | --- | | 3.1 An intervention is… *(one sentence)* | | | 3.2 Does touching the controls count? Touching the object? Watching closely? | | | 3.3 Who records it, and in what? | | | 3.4 Is it recorded at the time, or reconstructed later? | | | 3.5 Recovery time — measured from what, to what? | | | 3.6 Is operator travel time to the cell included? | | | 3.7 Is a single incident that needs three touches one intervention or three? | | **3.8 — Intervention types.** Count these separately. They have different causes and different fixes, and a single total hides which one you have. | Type | Count | Median recovery | Worst recovery | | --- | --- | --- | --- | | Re-presented the object | | | | | Cleared an obstruction | | | | | Restarted the capability | | | | | Restarted the robot or controller | | | | | Completed the task by hand | | | | | Called for maintenance | | | | | Stopped the line | | | | > 3.4 matters more than it looks. Interventions reconstructed from memory at the end of a > shift are undercounted, consistently and in the same direction. --- ## 4. The test window and sampling | Field | Answer | | --- | --- | | 4.1 How many attempts, minimum | | | 4.2 Over how many days | | | 4.3 Across how many shifts, and which ones | | | 4.4 Across how many physical robots | | | 4.5 Across how many operators | | | 4.6 Consecutive attempts, or sampled? If sampled, how? | | | 4.7 Does the window include a Monday and a Friday? A night shift? | | | 4.8 Does it include the product mix, or only the easy variant? | | | 4.9 Who may stop the pilot early, and on what grounds? | | > A window that runs only on day shift, only on the primary product, only on the robot > that was commissioned most recently, will produce a number. It will not produce an > answer about the site. --- ## 5. Exceptions and excluded conditions **Agree these before the pilot starts.** An exclusion written afterwards is an excuse. | # | Condition | Excluded? | Why | Agreed by | Agreed on | | --- | --- | --- | --- | --- | --- | | 5.1 | Upstream line stopped | | | | | | 5.2 | Power or network outage | | | | | | 5.3 | Object outside the agreed envelope | | | | | | 5.4 | Wrong object presented | | | | | | 5.5 | Robot fault unrelated to the capability | | | | | | 5.6 | Calibration drift | | | | | | 5.7 | Operator error | | | | | | 5.8 | Conditions not covered by the risk assessment | | | | | | Field | Answer | | --- | --- | | 5.9 Who decides an attempt is excluded? | | | 5.10 Is the decision made at the time or in review? | | | 5.11 Are excluded attempts still logged? *(they should be)* | | | 5.12 Maximum share of attempts that may be excluded before the result is void | | > 5.12 is the field that protects the number. Without a cap, enough exclusions can turn > any pilot into a success. --- ## 6. The scorecard One row per metric. **Every row needs its numerator, its denominator and its definition**, or the number cannot be checked by anyone who was not there. | Metric | Numerator | Denominator | Units | Definition used | Data source | Threshold | Threshold owner | Result | Decision | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Task completion | | | | | | | | | | | Interventions | | | | | | | | | | | Clean refusals | | | | | | | | | | | Worst single cycle | | | | | | | | | | | Median cycle | | | | | | | | | | | Recovery time | | | | | | | | | | | Unplanned stoppages | | | | | | | | | | | Excluded attempts | | | | | | | | | | **Decision** is one of: `met`, `not met`, `met with conditions`, or `not measurable` — and `not measurable` is a real outcome worth recording rather than hiding. --- ## 7. Signoff and unresolved issues | Field | Answer | | --- | --- | | 7.1 Overall decision — proceed, proceed with conditions, repeat, stop | | | 7.2 Who signs for the operator | | | 7.3 Who signs for the supplier | | | 7.4 Date | | | 7.5 Does the risk assessment cover the proposed next step? | | **7.6 — Unresolved issues.** The honest part of the document. Carry every one forward; an issue closed because the pilot ended is not closed. | # | Issue | Raised by | Owner | Must be resolved before… | Status | | --- | --- | --- | --- | --- | --- | | 1 | | | | | | | 2 | | | | | | | 3 | | | | | | | Field | Answer | | --- | --- | | 7.7 What was not tested at all | | | 7.8 What we would do differently next time | | | 7.9 What would have to be true to run this at scale | | --- ## Appendix A — The worked example (HYPOTHETICAL) **This example is entirely fictional.** No robot has run this pilot, the site does not exist, and every number below was invented to show the arithmetic. **None of it is field evidence and none of it should be cited.** It is illustrative arithmetic, nothing more. ### The headline numbers > **190 completed tasks out of 200 attempts is 95% completion.** > **20 human interventions across 200 attempts is 10 interventions per 100 attempts.** Those two look contradictory. They are not, and seeing why is the whole point of the example. ### How the 200 attempts actually break down | | No human involved | A human stepped in | Row total | | --- | --- | --- | --- | | **Completed** | 176 | 14 | **190** | | **Failed** | 4 | 6 | **10** | | **Column total** | 180 | **20** | **200** | Read it twice, because both of the useful facts are in the off-diagonal cells: - **14 interventions landed on tasks that then completed.** A person stepped in, the attempt recovered, and it counts in the 190. So an intervention is *not* the same thing as a failed task. - **4 failures involved nobody at all.** The capability refused before moving — the tote was outside its stated envelope — so the task did not complete and no human was required. So a failed task is *not* the same thing as an intervention. Both rates are correct. They measure different things: | Metric | Numerator | Denominator | Result | | --- | --- | --- | --- | | Task completion | 190 completed | 200 attempts | 95.0% | | Interventions | 20 interventions | 200 attempts | 10 per 100 attempts | | Clean refusals | 4 refusals | 200 attempts | 2 per 100 attempts | | Intervention success | 14 recovered | 20 interventions | 70% | > That last row is the one nobody calculates and everybody wants. **70% of the times a > person stepped in, the task was rescued.** That says something quite different about the > capability than either headline number does, and it comes free from the same four cells. ### Why this changes what you do next A naive reading — "95%, good enough" — misses that a person was pulled to the cell twenty times in two hundred attempts. Whether that is acceptable depends on what else that person was doing and how far away they were, which is an operations question and not a robotics one. The opposite naive reading — "ten interventions per hundred, that's a failure" — misses that fourteen of those twenty ended in a completed task, and that four of the ten failures cost nobody anything because the capability declined cleanly. **And none of these thresholds is universally acceptable.** 95% completion is excellent for a task where failure means re-presenting a tote. It is unacceptable for a task where failure means damaging what is being handled or stopping a line. The number does not carry its own interpretation; the person who operates the site supplies that. ### What the example does not tell you | Question | Why the numbers cannot answer it | | --- | --- | | Is this safe? | A completion rate is not a risk assessment. Separate process, separate competent person, before the pilot. | | Will it hold at scale? | 200 attempts on one robot says nothing about fifty robots across three shifts. | | Is the baseline better or worse? | There is no baseline in this example. Without one, 95% is a number without a comparison. | | Will it hold next quarter? | Nothing here measures drift, wear, seasonal product mix or a firmware change. | --- ## Change log | Version | Date | Change | | --- | --- | --- | | 0.1 | 14 November 2026 | First publication. | ## Review status This document has **not been reviewed by a qualified robotics professional** and is not a safety assessment. Corrections and disagreements are welcome at `support@embodiedsoftwarelabs.com`. ## References - CC BY 4.0 — All arithmetic in Appendix A is illustrative and self-contained; it is derived from the invented figures shown, not from any published study or field data.