How to distinguish a demo from a deployable product

A demonstration establishes what a system did under particular conditions. A deployment claim requires evidence relevant to the intended environment, users and operating requirements. Assess the gap by checking the configuration, test conditions, failures, integration and support needs. Do not treat a polished video, a maturity label or a single successful run as a complete deployment assessment.

What a demonstration can establish

Start with the narrowest description supported by the material. Name the task, the system version, the environment and the result. A demonstration may be useful evidence without answering every question about repeated operation or customer use.

GAO's Technology Readiness Assessment Guide emphasizes evidence of maturity and the readiness of technology for integration in acquisition programs. The Academy worksheet adapts that distinction to project research; it is not a substitute for a formal assessment under that guide. GAO-20-48G

Replace labels such as "production ready" with a question: production of what, for whom, under which requirements? Without those boundaries, different readers may be evaluating different claims.

Read a readiness level in its original framework

NASA describes nine technology-readiness levels. Its higher-level examples are expressed in a space and flight context. That makes the framework useful for understanding staged maturity, but not a universal certification ladder to copy unchanged into software, medical devices or robots. NASA, Technology Readiness Levels

When a company cites a readiness level, record the framework, assessor, assessment date, system boundary and evidence. If those details are absent, record the label as a claim. Do not turn it into a verified Orbitrum score.

Maturity also addresses a different question from valuation. A mature product may still face commercial or financial questions. An immature technology may be an intentional research-stage project. The review should explain the distinction instead of treating either stage as a complete investment verdict.

Build an evidence ladder, not an automatic score

The following is an Academy working structure. It is not a new standardized readiness scale, and its rows do not map automatically to NASA levels.

Use the table to locate missing evidence rather than to award points. A supplier may have strong evidence for one component and little for the integrated system. State which system boundary the conclusion covers.

Ask for the denominator and the omitted runs

NIST's 2024 mobile-manipulator report varies experimental conditions rather than treating one observation as a complete performance claim. Its specific industrial experiment is a reminder to examine how a result was produced, not a benchmark for unrelated products. NIST AMS 100-57

In your notes, distinguish selected examples from the full set of trials. Identify what counts as success, which failures were excluded, whether a human intervened and whether conditions changed. A percentage without its definition and denominator leaves the conclusion underspecified.

For a qualitative result, the same discipline applies. "Users completed the task" needs the task, user group, assistance allowed and setting before it can support a broader claim.

Check integration and operating assumptions

A useful technical review does not stop at the component's best result. Ask which external systems, data inputs, facilities and human roles are assumed. Then identify evidence about the proposed integration, not only the isolated component.

Suggested questions for the memo are: what must the customer supply; which interfaces were tested; what happens when an expected input is missing; how are updates handled; and who diagnoses failures? These are educational prompts, not a claim that one particular test suite is required for every product.

Do not describe missing public information as a discovered defect. Write "integration evidence was not available in the material reviewed" when that is the actual limitation.

Keep software security and AI risk visible

NIST's Secure Software Development Framework provides practices and vocabulary for discussing software development and vulnerability response with suppliers. It is not a certificate that a product is secure. NIST SP 800-218, version 1.1

Where relevant, ask which development and response practices are documented, which version they concern and who is responsible. Do not infer secure operation from feature performance alone.

For AI-enabled systems, NIST's AI RMF organizes risk work around Govern, Map, Measure and Manage. Its voluntary framework does not prove a particular model is reliable in every setting. NIST AI RMF 1.0

Keep the demonstrated task result and the remaining risk questions in the same assessment. Neither should silently erase the other.

Practice: narrow an overbroad claim

Hypothetical exercise - no real project is being described. A team shows a system completing a workflow on five prepared examples. It says that the product can handle any customer's production workload.

A defensible initial note is: "The material demonstrates the five presented examples in the shown configuration. It does not establish coverage of other workloads, unsupported inputs or operational constraints. A relevant-environment evaluation against stated requirements would address part of that gap."

Do not conclude that the system cannot work. The issue is that the evidence supports a narrower statement than the claim. This difference is the central skill the exercise teaches.

A technical assessment memo

Record the claim and intended environment; configuration and system boundary; material reviewed; test method and outcome; important limitations; integration and support assumptions; and the next evidence request.

For an expert, this structure makes the reasoning inspectable. For a reader, it helps distinguish an observed result from a prediction. Continue to commercial traction evidence before treating deployment evidence as customer demand.

Questions about technical readiness

Does a high readiness level prove product-market fit?

No. Technology maturity and commercial adoption are different questions. Examine customer and commercial evidence separately.

Can a video be useful evidence?

Yes, for what it actually shows. Preserve its conditions and limits; do not assume unseen trials, unattended operation or independent verification.

Should missing public technical evidence automatically produce a low score?

This guide provides no such scoring rule. Record the uncertainty, the scope of available evidence and the limits of the conclusion. A missing public document is not proof of a failed test.