Technology

How we think about Physical AI evaluation.

This page separates what exists from what is direction. Status labels apply to each item.
Available nowPrototypeResearch directionPlanned

Problem architecture

Research direction

Physical AI failures are multimodal and temporal. We treat each interaction as an episode that can be stored, searched, compared and replayed, rather than as raw video.

Data inputs

Prototype

Video is the first input. Camera streams, model outputs, robot logs, telemetry, other sensor data and simulation traces are targets. Only video via in-browser pose analysis works today.

Physical episode representation

Research direction

A structured schema of agents, objects, actions, contacts, timeline, predictions, outcome and uncertainty. The schema is being designed with developer feedback.

Perception layer

Available now

Human pose estimation (MediaPipe) runs in the browser in the Labs demo. Object and multi-agent perception are not built. Pose is one method among several, not the core of the product.

Temporal / event layer

Prototype

Frame-level tracking over time exists in the demo. Event extraction such as contacts, transitions and anomalies is a research direction.

Evaluation layer

Research direction

Compare expected and observed behavior, detect failures, and compare model versions. Not implemented.

Simulation interface

Planned

Export episodes in a form that simulators can approximate and vary. Interchange formats such as OpenUSD are being explored. Not implemented.

Uncertainty

Research direction

Episodes should carry confidence so engineers can separate model errors from perception errors. Design stage only.

Privacy and data lifecycle

Prototype

The Labs demo analyzes video in the browser. We intend to work on structured representations rather than retaining raw footage where possible. Formal data-protection review is still to be done; see the Privacy Policy for current handling.

API direction

Planned

A programmatic interface to submit episodes and query evaluations is a long-term goal. No public API exists.

Compute architecture

Planned

GPU-accelerated batch inference and simulation-based evaluation are expected needs. Triton, OpenUSD and Isaac Sim/Lab are being explored; none is integrated today.

Illustrative

Example physical episode record

A sketch of the kind of structured record we are designing. Values are examples, not real data.
{
  "episode_id": "ep_0142",            // illustrative
  "sources": ["video", "model_output", "telemetry"],
  "agents":  [{ "id": "arm_01", "type": "robot_arm" }],
  "objects": [{ "id": "container_A" }],
  "events": [
    { "t": 4.0, "type": "contact", "agent": "arm_01", "object": "container_A" },
    { "t": 6.0, "type": "slip",    "object": "container_A" }
  ],
  "prediction": { "model": "v13", "label": "grasp_success", "confidence": 0.91 },
  "outcome": "failure",
  "failure": { "kind": "grip_lost", "confidence": 0.87 }
}

Layers

Architecture status

PerceptionAvailable now
Temporal understandingPrototype
InteractionResearch direction
Physical eventsResearch direction
EvaluationResearch direction
Simulation bridgePlanned
Structured outputPlanned

NVIDIA ecosystem

Where accelerated computing could help

GPU inference / Triton

Research direction

Batch evaluation of video and multiple model versions.

Isaac Sim / Isaac Lab

Planned

Target simulators for approximate episode replay.

OpenUSD

Research direction

Candidate interchange format for episodes and scenes.

Cosmos

Research direction

Possible world-model tooling for scenario variation, if technically relevant.

None of these is integrated today. Tenslam Vision has no partnership with, and is not endorsed by, NVIDIA. Product names are trademarks of their owners.

Talk to us

Working on a related problem?

Discuss a Use Case