Uncovering Why the Robot Misses the Towel: Mechanism Analysis of Real-World Policies and an Automated Investigation Using 4 AI Roles

In a linen supply face towel folding task, our physical robot policy exhibited a phenomenon where the gripper closed prematurely before reaching the cloth—resulting in an air grasp—and proceeded to transport nothing. In this post, we share technical findings revealed by over 10,000 offline control inferences and internal feature analyses, specifically the "tug-of-war between vision and action history" and a "double lock mechanism." We also present our practical process of combining three AI models across four distinct roles with human review to autonomously complete this complex investigation in roughly half a day.

1. Introduction: Air Grasps and Empty Transport on the Factory Floor

In linen supply industrial facilities handling textiles, there is high demand for automating the process of picking face towels one by one from a stack, folding them accurately, and stacking them. However, when deploying a real-world policy based on a state-of-the-art vision-language-action (VLA) model^[1] with real-time chunking (RTC) control^[2], we observed a distinctive failure mode.

Specifically, while the robot arm approaches the towel pile, it closes its gripper before the fingertips reach the cloth (air grasp). Furthermore, despite failing to grasp the cloth, instead of reopening immediately over the pile to retry, the arm frequently exhibited a behavior where it continues transporting to the folding table with the gripper closed and empty (empty transport).

This failure pattern raises two fundamental questions:

• Question 1 (Premature Closure): Why does the policy close the gripper before reaching the cloth, causing an air grasp?
• Question 2 (Delayed Recovery): Why does the robot fail to recover by reopening immediately over the pile after an air grasp, instead continuing to carry nothing to the table?

This article is organized into two parts. In Part 1 (§2–§7), we explain the underlying mechanisms uncovered through over 10,500 offline control inferences, layer-wise feature probing, and cross-referencing against 200 expert demonstrations. In Part 2 (§8–§10), we share our practical workflow where a lean team—with a lead engineer also handling implementation—combined three AI models across four roles with human quality gates to autonomously complete this complex investigation in just half a day (approx. 4 hours and 40 minutes).

2. Verification and Rejection of Initial Hypotheses

Before diving into the underlying mechanisms, we examined several intuitive hypotheses initially raised by the team. Based on robot logs and control inference data, all four initial hypotheses were ruled out as primary causes.

To supplement Hypothesis 2 in particular, the forward advancement during closure in successful grasps was 4.76 mm, 1.36 mm, and −0.76 mm, showing that successful trials did not advance deeper. When positioned properly to envelop the cloth, a secure grasp can be achieved with minimal movement, whereas air grasp trials failed to capture the cloth despite continued forward motion. Therefore, insufficient hardware tracking or arm retraction cannot be considered the primary cause of air grasps.

3. Why Does the Gripper Close Prematurely? — Mechanism of Early Closure

3.1 Visual Intervention Experiments: Disentangling via Frame Replacement and Masking

To determine whether the model is actually utilizing visual information to govern its grasping behavior, we conducted input intervention experiments on images from the robot's left wrist camera.

While holding the robot state, joint postures, and random seeds fixed, we substituted only the left-wrist camera frame with either a frame from "25 steps prior (pre-approach)" or "25 steps later (post-approach)" within the same episode. This intervention produced marked behavioral shifts.

When presented with an earlier frame where approach had not yet progressed, the model substantially increased its approach distance (27.73 mm → 38.80 mm) and delayed gripper closure. Conversely, when given a later frame where approach had progressed further, the model curtailed its approach and predicted earlier closure.

Furthermore, to determine whether the model is attending to the cloth or to its own gripper configuration, we conducted end-effector masking experiments. When masking and freezing the gripper region while substituting only the background cloth region with frames from 25 steps prior or later, the predicted gripper value in the latter half shifted dramatically from 0.454 to 0.161 (prior frame) / 0.703 (later frame). In contrast, when freezing the background and substituting only the gripper region, the value barely budged from 0.454 to 0.451 / 0.454. These results confirm that the model is far from blind: it visually perceives the relative approach state between the cloth and its surroundings to regulate approach and closure.

3.2 Strong Closure Constraints Imposed by Action History (Prefix)

If vision actively tracks cloth approach, why does the gripper close prematurely? The core reason lies in the powerful constraint exerted by the immediate action history (action prefix).

In addition to the current observation image and state, the target policy generates the subsequent 50-step action chunk conditioned on the sequence of actions executed over the preceding several steps (the prefix). For 12 episodes where the first grasp attempt missed, we evaluated the differences in action generation just prior to closure (10 steps before gripper command 0.3) with and without the prefix.

In successful trials, the initial arm pose relative to the cloth is favorable, allowing the gripper to envelop the cloth even with a brief approach; thus, the grasp succeeds even when action history encourages closure. In failure trials, however, when the situation demands "pushing forward just a few more millimeters," feeding in the immediate action history of "approaching and starting to close" pulls the generated trajectory aggressively toward closure. As a result, the required approach is truncated by approximately 5.8 mm, causing the gripper to close prematurely before the cloth.

4. Why Can't the Model Self-Correct When It "Sees" the Miss? — The Disconnection Between Representation and Action

4.1 Retention of Grasp Outcome Signals Across Model Layers

Preliminary investigations suggested that grasp success or failure could be predicted with high accuracy from left-wrist camera frames. In this investigation, we extracted internal representations at each layer of the policy and rigorously evaluated how far downstream the discriminative signals propagate using episode-level cross-validation.

During the closure phase, from the image encoder through multimodal fusion to the KV cache directly queried by the Action Expert, signals distinguishing success from failure were consistently preserved at an exceptionally high level of AUROC 0.88–0.95. Compared to joint state (qpos) alone, which was essentially indistinguishable at AUROC 0.35–0.44, it is evident that outcome information is never lost from visual features.

4.2 Overwhelming Action History Bias in Flow Matching

If outcome information reaches the doorstep of the Action Expert, why does it fail to generate corrective actions to avert air grasps? For seven air-grasp episodes where an external probe assigned a high "failure" score, we examined differences in generated actions with and without the prefix.

• Without prefix (no action history): In 6 out of 7 cases, the model aborted closure midway and generated actions toward reopening (mean gripper value of 0.622).
• With 6-step prefix (immediate action history present): In all 7 out of 7 cases, the gripper command in the latter half of the prediction saturated at 0.998, outputting complete sustained closure.

This controlled experiment vividly demonstrates a profound divide between "being able to probe outcome information from features" and "the model itself utilizing that information for behavioral correction." During flow-matching trajectory generation, the continuity bias imposed by recent action history overwhelms the corrective signals from visual features, suppressing any behavioral correction (reopening).

5. Why Does the Robot Carry Nothing Instead of Recovering? — The Double Lock Mechanism

5.1 Decoupling Gripper Closure and Arm Transport

Next, we investigated our second question: "Why does the robot carry empty to the table instead of reopening immediately over the pile after an air grasp?" We compared three scenarios: immediate recovery over the pile after an air grasp (①), empty transport to the table after an air grasp (②), and normal transport to the table with a held towel (③).

This comparison reveals that the primary driver of sustained closure during empty transport (②) is not the current sensor state, but rather the action history in the prefix indicating "it was closed just prior." Even when the state input is spoofed to "open," closure is sustained (0.722) as long as the prefix remains closed; conversely, flipping the prefix gripper actions to "open" immediately induces reopening to 0.103.

On the other hand, in pile recovery (①), open predictions of 0.179–0.192 are output even when the state or prefix is "closed." When strong visual and postural contexts indicate that the arm remains directly over the pile, they can override the action history constraint to trigger reopening.

5.2 Decoupling "Opening" and "Returning": The Double Lock

The analysis showed that "reopening the gripper" and "returning the arm to the pile" are constrained by entirely distinct mechanisms.

During empty transport observations, altering the gripper actions in the prefix to open commands succeeded in opening the gripper itself. However, the predicted terminal angle of Joint 1 in the arm trajectory shifted only slightly from −99.9° to −98.2°, still persisting in transport toward the table (compared to ~−87.6° when predicted without prefix). Conversely, resetting only the arm joint angles to a posture near the pile directed the arm back toward the pile, but left the gripper open/close predictions virtually unaffected.

5.3 Behavior of the RTC Controller: Sustained Signals vs. Micro-Pulses

To address concerns that "queuing in the real-time chunking (RTC) controller might be suppressing open commands," we conducted simulation experiments using the control codebase.

When the model outputs a sustained open prediction (an open command lasting several steps or more), it is adopted into physical robot commands within just 4 steps (approx. 0.08 s) after receiving the new chunk, even with existing chunk consumption (3–25 steps) and latency. Therefore, the multi-second empty transport cannot be attributed to RTC queue delays.

However, when the model outputs micro-pulse predictions where the gripper opens for merely 1–3 steps, this signal can be washed out by blending with preceding chunks (a 5-step blending process) or adoption timing. The real issue is not the RTC buffering itself, but that the model fails to generate a sustained recovery sequence.

6. Structural Causes in Training Loss and Demonstration Data

Why does the model rely excessively on action history and fail to generate recovery trajectories from failures? Comparing the training setup against 200 expert demonstrations brought three structural causes to light.

First is a blind spot in the RTC training loss design. During training, the first 0–9 steps of ground-truth action sequences from successful demonstrations are provided as clean prefixes to minimize subsequent prediction errors. Consequently, the training objective never provided direct opportunities to learn recovery from negative examples—such as conditioning on an "erroneously closed action history" or an "air-grasp history" and detecting the error visually to abort and reverse course.

Second is the absence of physical constraint terms in the flow-matching loss. The current objective merely minimizes imitation error over smooth motion trajectories, lacking explicit penalties for cloth contact verification or residual geometric distance to the target. For the model, predicting "close if previously closing, transport if previously transporting" represents the easiest path to loss minimization, entrenching an excessive history-following bias.

Third is the kinematic characteristics of the demonstrations themselves. An analysis of grasping motions across 200 successful demonstrations revealed that closure duration (median 0.32–0.48 s) and forward travel during closure (median 2.41 mm) differed little from physical failure cases. Human teleoperators did not push deeply into the cloth before closing slowly; rather, they executed quick sweeping motions with shallow strokes. In attempting to imitate this razor-thin timing, the model frequently fell short, triggering air grasps.

7. Mechanism-Grounded Improvement Approaches

Based on the uncovered mechanisms, we outline concrete improvement approaches and corresponding verification metrics.

8. Part 2: Automating the Investigation with 3 AI Models and Human Review

8.1 Motivation: Autonomous Investigation in a Lean Team

The deep mechanistic analysis described above requires immense effort, including over 10,000 control inferences, layer-wise feature analyses, and cross-referencing demonstration data. However, our team operates within a small startup where the lead engineer oversees architectural design, implementation, and team progress simultaneously.

To resolve physical robot challenges rapidly in such an environment, rather than having engineers get bogged down writing code and running inferences manually, it was essential to establish a workflow where human attention focuses strictly on "problem formulation and quality gates," delegating experimental scripting, running 10,000+ inferences on GPUs, log aggregation, and report generation autonomously to AI agents.

8.2 Selection Rationale for the 3 AI Models and the 4 Roles

Rather than relying entirely on a single model, this investigation leveraged the distinctive strengths of frontier models available in September 2026, dividing four roles across three models^[5].

The pipeline consists of four stages. Stage 1 launches the preliminary investigation; Stage 2 takes human review and defines the detailed investigation requirements; Stage 3 completes the autonomous GPU experiments and analysis; and Stage 4 restructures the findings into a human-facing explainer and shares it with the internal knowledge base. The roles played at each stage and the rationale behind them are as follows.

8.3 Collaboration Timeline: 4 Hours and 40 Minutes

From launching the preliminary investigation to detailed mechanistic analysis, explanatory drafting, and internal knowledge sharing, the entire process was completed on September 9, 2026, in just 4 hours and 40 minutes (UTC 02:49–07:28).

9. The Only Role Played by Humans: The "This Doesn't Answer the Question" Quality Gate

9.1 Rejections as the Greatest Leverage

Throughout this entire pipeline, not a single line of code or report prose was directly written or edited by a human. The actual time spent by the human typing prompts and review feedback totaled only about 30 minutes.

The decisive leverage governing overall investigation quality was the rejection issued against the preliminary report: "This does not answer the question."

In the preliminary report, the agent delivered measurements such as "delayed reopening originates prior to command emission" and "altering state inputs alters predictions." To this, the human responded with two incisive critiques:

• Critique 1: "Regarding the delayed reopening, you only point out that it is delayed, barely touching on why. In the end, what is your conclusion as to the root cause?"
• Critique 2: "We have separate evidence that success or failure is predictable from visual information. If the information exists, why does it close prematurely? Dig deeper into the specific mechanism. In that sense, your analysis is undeniably insufficient. Draft a prompt for a deeper analysis."

Refusing to settle for superficial factual reporting and persistently pressing "why" forced the agent to recognize its own shortcomings—namely, failing to disentangle the impact of action history (prefix) from state inputs. Consequently, the agent itself formulated a detailed specification imposing the strict completion condition that "mere paraphrasing of the phenomenon or listing candidate hypotheses shall not constitute completion."

9.2 Separation of Specification Design and Autonomous Execution

Another critical design choice was cleanly separating the session for specification design (dialogue) from the session for experimental execution (unattended execution).

Instructed by the human that "this will be executed in a separate session; do not execute it yourself and delegate operational details to the executing agent," Codex did not have to worry about the burden of executing the work itself. As a result, rather than compromising to shrink the investigation scope, it was able to author specifications holding an uncompromisingly high standard of rigor as a pure specification architect.

In turn, the agent in charge of unattended execution carried out the experiments with scientific discipline that exceeded human instructions—pre-registering hypothesis falsification protocols and autonomously catching and correcting prompt mismatches with training.

10. Conclusion: Grounded Reflections from Practice

The overarching lesson from this investigation is that while blindly offloading tasks to AI tools cannot reach deep engineering truths, having humans formulate precise questions and maintain uncompromising quality gates empowers even a lean team to execute research and development of extraordinary depth at remarkable speed.

AI agents excel at executing prescribed tasks, but critically assessing whether the results truly answer the questions that matter remains the essential role of humans. Only when backed by human review that challenges logical leaps and pushes back with "this does not answer the question" do AI agents unlock their full potential.

Moving forward, we will continue exploring grounded applications of AI agents on gritty real-world robotics challenges, pursuing the fastest path to practical robot intelligence.