GaP — Graph-as-Policy

01 · task class

What is Variational Automation?

We identify Variational Automation (VA) — a class of tasks where a robot persistently performs varying instances of a task, with non-trivial variation in the geometry and pose of objects (e.g., sort packages, make coffee in a café, or build sandwiches in a commercial kitchen). VA sits between fixed automation and full generalist robotics: the workcell, robot, and sensors are known and fixed, and the range of objects and poses is bounded — so a system can reuse object models, calibrated sensors, and modular skills while still adapting to every new instance.

FA

Fixed Automation

A robot persistently performs identical instances — blindly repeating the same motions (spot welding, spray painting). Tuned by hand for high reliability and throughput, but no adaptivity.

VA

Variational Automation

A robot persistently performs varying instances: objects vary in geometry (different SKUs) and arrive in a distribution of poses — within a known, bounded workcell. This is GaP’s target.

GR

Generalist Robotics

A robot must perform a broad variety of open-ended tasks, mostly via model-free end-to-end VLA policies. Highly flexible, but not yet at commercial / industrial reliability.

02 · the artifact

A prompt becomes a computation graph

Give GaP a task description. An orchestration agent partitions it into skill-aware segments; skill agents synthesize localized subgraphs from MORSL; the orchestrator wires them into one executable, type-checked graph. Below is the actual grocery-packing graph — rendered live from its workflow.json. Click any node to inspect its code, inputs, and routing.

TASK

▸ orchestrator: 3 skill agents synthesizing subgraphs … wiring control flow

examples/grocery_packing/packing_graph/workflow.json

tool call script router success route failure route click a node to inspect →

One interpretable graph — perception, grasp, transport, recovery routes — with a backward edge that loops until every item is packed. Failure routes converge on abort; success exits at done.

SIMULATION

REAL ROBOT

The optimized graph runs unchanged in simulation and on the physical cell — VA-II · pack grocery items, playback accelerated.

03 · self-learning

Rehearse, localize failures, refine the graph

On the long-horizon Make Popcorn task, GaP’s initial graph turns the stove knob on and off but mis-grasps the pan, succeeding only 33% of the time. Through 10 iterations of internal simulation rehearsal, GaP autonomously diagnoses failures from contact feedback and edits the graph — reaching 94% in simulation and 90% (18/20) on the real robot.

Pan-pose variations and the on-burner success-rate curve climbing from 30% to 94% across ten rehearsal iterations — Pan-pose variations (left) drive a 10-iteration sequence of graph updates; on-burner success rate (right) climbs 30% → 94% through three phases — grasp improvement, transport adjustment, and placement refinement.

SIMULATION · rehearsal

REAL ROBOT · 16×

The refined graph placing the pan on the burner — in the rehearsal simulator, and on the physical robot.

04 · benchmark

8 open Variational Automation tasks

Four in simulation and four in the real world — grocery fulfillment, grocery packing, popcorn making, USB-C cable insertion, and industrial crate washing. Six run on a Franka arm (LIBERO-derived scenes), the cable benchmark uses a UR5 with force feedback, and crate washing is bimanual. Every instance introduces non-trivial variation in object pose and geometry.

VA-I

Grocery Fulfillment

SimReal

Each instance randomizes target objects, poses, and basket placement — the policy adapts every episode.

VA-II

Grocery Packing

SimReal

Loops perceive → grasp → transport until every item is packed; item set and poses vary per episode.

VA-III

Make Popcorn

SimReal

Long-horizon stove manipulation — knob, pan, burner; refined via self-learning (16× playback).

VA-IV

USB-C Cable Insertion

Real

UR5 with force feedback; one graph handles ascending, descending, odd- and even-port orders (50×).

VA-V

Crate Washing

Simbimanual

An industrial crate-washing cell: crate poses and arrival order randomized; two Franka arms coordinate to pick, wash, and stack each instance — modeled from a real washing line.

What positional variation looks like

Every simulated instance re-samples object poses and arrangements — the same policy graph must absorb the disturbance. Sampled initial states from the benchmark:

“Pick up the alphabet soup and place it in the basket.” — pose-randomized episodes

“Pick up the chocolate pudding and place it in the basket.” — pose-randomized episodes

Make Popcorn — sampled initial pan / knob / stove states

05 · method

The GaP harness

Given a natural-language VA task and geometric object models, the multi-agent harness decomposes the task into semantic segments, composes a computation graph from MORSL skills, refines it through internal simulation rehearsal, and ships the optimized graph G* to an edge interpreter for persistent execution.

GaP pipeline: task spec and geometry enter the agentic harness, which generates a computation graph, self-learns in simulation, and hands the optimized graph to an edge executor — The harness converts a VA task specification into a computation graph, optimizes it via self-learning in simulation, and hands G* to an edge-based interpreter that drives the robot — sim and real feedback close the loop.

🧩 Multi-agent harness

An orchestration agent partitions the task and dispatches skill agents that synthesize localized subgraphs; every edge is statically type-checked. Separating generation from testing limits context and reduces incentives to “cheat”.

📚 MORSL skill library

51 initial skills — perception, grasp planning, motion planning, 2D/3D vision, ROS translation, verification — each declaring graph inputs, outputs, parameters, and pre-conditions in agent-readable conventions.

🕸️ Graph-as-Policy

A policy is a directed computation graph of atomic nodes joined by data and control edges — interpretable, modular, reusable. Inspired by ROS computation graphs and TAMP hierarchies.

🔁 Self-learning rehearsal

Parallel rehearsals in a parameterized simulation register pre/post node states; contact and state feedback localize failures to specific nodes; agents edit topology and parameters until performance plateaus.

GaP system architecture with the eight VA benchmark tasks — System architecture: the agentic harness (over coding tools such as Claude and Gemini) draws skills from MORSL — combining model-based procedures (e.g., ROS) and model-free policies (e.g., GraspGen) — self-learns in NVIDIA Isaac, and executes on the edge. Bottom: the 8 VA benchmarks.

06 · results

Results

Across simulation and real-world VA benchmarks, GaP maintains robust success rates under pose and geometry variation where VLA, TAMP, and single-agent code-as-policy baselines degrade — and it can also stage VLA policies into distribution for >2× gains. Best per column in bold; GaP rows highlighted.

Success rate over 5,500 trials, 100 task instances per cell. LIBERO / LIBERO-Pro have small pose variation; VA columns add larger variation. Rows 5–6: GaP stages the wrist camera, then hands off to a VLA policy.

Method	LIBERO	LIBERO-Pro	X-Y 20×20	basket_swap	permutation	mixed_all	Pack fixed	Pack varied
CaP-X	—	0.22	0.07	0.05	0.11	0.10	0.01	0.01
π₀.₅	0.96	0.24	0.78	0.15	0.20	0.20	0.17	0.18
MolmoAct2	0.97	0.43	0.90	0.26	0.10	0.20	0.18	0.18
TipTop	0.22	0.22	0.29	0.24	0.31	0.24	0.34	0.46
π₀.₅ w/ GaP	0.85	0.60	0.79	0.32	0.50	0.39	0.67	0.66
MolmoAct2 w/ GaP	0.70	0.62	0.84	0.58	0.39	0.66	0.59	0.59
GaP	0.95	0.95	0.95	0.97	0.93	0.97	0.99	0.98

Real-robot success rate. Avg. single pick-and-place completion time: TipTop 95 s vs. GaP 67 s.

Task	TipTop	GaP
Fulfill Grocery Orders	8/25	25/25
Pack Grocery Items	10/30	28/30
Make Popcorn	0/20	18/20

130 total instances. SR = success rate, ET = execution time (s). Per-trial vs. full-sequence success across goal conditions.

Goal condition	SR (per trial)	SR (all)	ET (s)
Ascending order	32/35	2/5	247.0
Descending order	32/35	4/5	223.2
Odd-index port	20/20	5/5	136.8
Even-index port	14/15	4/5	104.6

150 trials plus a 3-hour continuous throughput run. GaP approaches an expert hand-engineered graph.

Method	Success rate	Avg. cycle (s)	Throughput (succ/hr)
Hand-Engineered (expert)	0.987 (148/150)	176.47	19.33
GaP	0.953 (143/150)	179.13	18.33

✔

Graph beats graphless. Replacing the graph with a single LLM emitting raw Python collapses success to zero — interface and syntax mismatches terminate runs even when the high-level logic is right.

✔

Multi-agent decomposition matters. Condensing the specialized authoring agents into one LLM also drops success to zero; every trial fails static structural verification.

✔

Positional robustness. GaP holds 0.93–0.99 across pose variations where VLA baselines fall to 0.20; staging the wrist camera with GaP more than doubles VLA success.

✔

Self-learning works. Rehearsal raises Make Popcorn from 33% to 94% in sim and 90% real through autonomous, feedback-driven graph edits.

✔

Approaches expert engineering. Matches a hand-engineered bimanual crate-washing graph (0.953 vs 0.987) and reaches 25/25 on real grocery fulfillment where TipTop manages 8/25.

Industrial reliability & throughput

Cycle times remain below the ~7 s/instance industrial standard; fewer VLM calls and faster IK/motion planning — plus more self-learning — are needed.

Beyond quasi-static

The benchmarks focus on quasi-static pick-and-place; only cable insertion needs force sensing. Deformables, dynamics, and moving targets are future work.

Scaling the harness

As applications grow and MORSL expands, more sophisticated agent harnessing is required to compose, verify, and maintain reliable graphs at scale.

07 · cite

Citation

bibtex

@article{chen2026gap,
  title   = {GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness
             for Variational Automation Tasks},
  author  = {Chen, Kaiyuan and Xie, Shuangyu and Fu, Letian and Yu, Justin and
             Pacini, William and Bajamahal, Sandeep and Kim, Hudson and
             Drake, Jaimyn and Kim, Daehwa and Xue, Haoru and Francis, Jonathan and
             Juette, Christian and Schaldenbrand, Peter and Seker, Muhammet Yunus and
             Wickramarachchi, Ruwan and Yoo, Uksang and Wang, Guanzhi and
             Murali, Adithyavairavan and Sundaralingam, Balakumar and
             Sastry, S. Shankar and Huang, Spencer and Zhu, Yuke and
             Fan, Linxi and Goldberg, Ken},
  journal = {arXiv preprint arXiv:2607.05369},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.05369}
}