Retries After Empty Grasps on the Round Table: Two Real-Robot Fixes

After missing the towel on the round table, the immediate retry also comes up empty. In real-robot towel folding, lifting the towel from the round table onto the folding work surface is often the bottleneck. Once the first grasp misses, the following retry fails more often than a typical first grasp (on both path baselines, conditional retry success is only about 19%–29%). Our previous post ^[1] traced empty grasps to prefix inertia rather than perception collapse and proposed two fixes; this sequel validates both on the robot for retries after a round-table miss. Conditional retry success rises from about 19.2% to 58.8% on the training path and from about 29.4% to 66.7% on the deployment path; mean successful-episode completion time shortens by about 4 s.

1. Introduction: hard recovery on round-table retries, and two fixes to validate

Author and working style. I am Yunhao Chi, an early machine-learning engineer at Omotenashi Robotics and joined when the company started. This work takes the team’s mechanistic conclusions on empty grasps onto the real robot: diagnosing, designing, and validating fixes for retries after a round-table failure. I first form hypotheses myself from failure videos and action/state curves, then have an AI independently read the same material so the two views can collide and converge on exploration directions. For each direction I draft the idea and experiment plan, then ask the AI to stress-test and fill gaps. Log mining, scripting, review, and result aggregation lean heavily on AI, while diagnosis and the experimental mainline stay human-led. What follows is that loop brought onto the robot.

Task and system. Our station is a bimanual robot with an overhead camera and left/right wrist cameras. In the towel-folding workflow, the left arm picks the towel up from a circular supply table and places it on the folding work surface; later folding can finish only after that. The policy is a π₀-style flow-matching vision–language–action (VLA) model ^[2]: vision and language are encoded into a shared representation, then an action expert generates a multi-step future action chunk. Real-robot control uses Real-Time Chunking (RTC) ^[3]: while executing the current chunk, the robot writes the most recently executed steps as a clean prefix (deterministic steps without noise) that conditions the next generation. In deployment this prefix length is clamped to at most 9 steps, so when we later compare length 9 vs length 0 (cleared), the former is the online default and the latter is clean regeneration after an interrupt.

In our previous post ^[1] we showed that during empty grasps, visual and fused representations still encode grasp success/failure (AUROC 0.88–0.95), but the RTC action-history prefix locks generation into closing—and we hypothesized two potential fixes: (a) adding recovery demonstrations on the training side, (b) resetting the prefix after empty-grasp detection. Those fixes had not yet been validated on the robot.

A recurring field observation: if the first closure is empty, opening and grasping again still often fails. Quantitatively, both fix paths have weak baselines: without recovery data, the training baseline’s conditional retry success is only 19.2% (5/26); with no runtime intervention, the deployment baseline is 29.4% (5/17). Once the first grasp has already failed, retries do not easily self-correct. This article tests whether the same mechanism also governs failed retries on the round table, and implements and validates both paths on the robot: a training-side data recipe and a deployment-side runtime detector.

2. Retry-specific verification

Before testing the two fixes below on the robot, we first check whether a failed retry on the round table is still governed by the same prefix-inertia mechanism as a first-grasp failure, or whether something else breaks specifically between the first and second grasp. We check two retry-specific questions.

First, is hard recovery because “between the failed first closure and the retry closure, the left arm already walked success-style vs failure-style modes”? On 64 first-failure trajectories, we cut left-arm 7-D joint trajectories from the first closure onset peak to the retry onset peak and compare by retry outcome. Under a label-permutation test, the between-group distance is not extreme (p≈0.53, not significant); leave-one-out classification accuracy is nearly identical to a majority-class baseline that always guesses failure (~53%). The forked left-arm execution hypothesis is not supported.

Second, on the retry window: vision / fused probe AUROC remain about 0.96 / 0.94, which does not support “perception collapsed at the retry instant.” Shortening the clean prefix from length 9 to 0 raises the offline action-chunk critic (a shallow readout whose input is features of an offline-generated action chunk and whose output is a 0–1 “will succeed” score) window-aggregated predicted score from about 0.64 to about 0.75 (mean difference +0.106, paired test p=0.00014). Success/failure decodability of generated actions with vs without prefix is similar (0.858 vs 0.860); the prefix does not erase decodability, but it can still pin wrong closing. Critic scores are offline ranking scores, not calibrated success probabilities, and are not the same as the gripper already pinching the towel on the robot.

In one line: evidence on the retry window matches the previous post—the bottleneck remains prefix inertia, not perception collapse or arm-trajectory divergence. The two paths below attack training data and runtime intervention respectively.

How to read the metrics and experimental scope

• Conditional retry success: among episodes whose first round-table grasp already failed, the fraction whose second grasp then succeeds. It is not whole-task success.
• Completion time and efficiency: successful-episode start/end times are judged by an automatic evaluator from camera views (for example, whether the towel has been placed on the work surface); completion time, throughput, and human intervention rate are computed from those boundaries. Human intervention rate is how often an operator actively intervenes per unit time (for example, catching a towel or manually resetting). It is not the same as detector-triggered clean regeneration (discarding the current chunk and regenerating with an empty prefix).
• The two paths are comparable only within themselves: §3 (training) and §4 (deployment) are evaluated on two different fine-tuned policies. Compare each only to its own baseline; do not treat absolute numbers as cross-path comparisons. The primary metrics are conditional retry success and first-grasp failure rate. Conditional retry measures whether a failed first grasp can be recovered; first-grasp failure is a concurrent improvement on the training path, and on the deployment path it confirms that the intervention does not degrade the first grasp. Completion time and other task-efficiency numbers are secondary within-path evidence.

3. Training-side path: mix in short “fail-then-succeed” demos

§7 of the previous post already proposed “adding recovery demonstrations under a wrong prefix.” This section gives the real-robot implementation and results.

The recovery clips come from autonomous rollout logs of the towel-folding task on the real robot, not from human teleoperation. On the left-gripper closure (joint-command dimension 6, about 50 Hz, near 1 meaning closed) we mark the two closure peaks and keep only trajectories whose first round-table grasp fails and whose immediate retry then succeeds. The ordinary fine-tuning data is the same as the baseline that contains no recovery clips: only two batches of success demonstrations of this same task (about 650k frames), with no other tasks. Relative to the baseline, what this fine-tune adds is these short clips.

Clips are cut between the two closure peaks, not as full episodes. The first closure peak t_peak is the first local maximum of left-gripper closure at or above 0.95 (successive peaks at least 30 frames apart). The retry peak t_peak2 requires that the gripper then open below 0.5, after which we take the first frame of the next rise past 0.95—the onset of closure, not an arbitrary high point on a long hold plateau. We keep only fast recoveries whose two peaks are less than 6 seconds apart. The closed cut is exactly [t_peak, t_peak2]: from the failed closure to the successful retry closure, and no further. These clips are about 1.7–6 seconds, with a median of about 3.2 seconds. A short clip contains only the failed closure and the correction that follows, so when ordinary training samples a window at random inside it, what it sees is that recovery, not a full fold that never failed.

253 fast-recovery clips enter training. The loader has no separate mixture weight, so this set of clips is listed three times and concatenated with the ordinary success demonstrations, which are not repeated. By frame count, after that repetition and after dropping padded start frames at the tail, the recovery clips are about 12% of the mix. Fine-tuning continues from the baseline weights.

On 100 real-robot episodes, conditional retry success is 58.8% (10/17), above the no-recovery-data baseline 19.2%, consistent with learning to correct after failure. In the same evaluation, first-grasp failure falls from baseline 26.0% to 17.0% (17/100); better first grasp and better conditional retry are two concurrent improvements.

On task efficiency, mean successful-episode completion time is 70.3 s (baseline 74.7 s), median 62.2 s (baseline 71.3 s); throughput 32.5 units/hour (baseline 32.0); intervention rate 12.0/hour (baseline 11.5, slightly up). This is consistent with less wasted time in the early round-table stage.

4. Deployment-side path: clean regeneration after empty-grasp detection

§7 of the previous post also proposed “resetting the prefix and triggering recovery after empty-grasp detection.” This section gives a real-robot intervention that needs no retraining: once an empty grasp is detected, discard the current action chunk and clean-regenerate with an empty prefix.

4.1 Labeling window and detector

When the gripper goes from open toward closed to grasp the towel, we look at the closure curve: 0 is fully open, 1 is fully closed (normalized joint command). Rising from 0 toward 1 is closing, not opening. For offline labeling, we want labels in the interval that is about to close but still interruptible, not over the whole grasp. Design goals: cover annotated failure times as much as possible; at a working point of “about 80% of true failures reported” (recall ≈ 0.8), reduce false triggers while keeping some lead time.

In practice, find the first time closure exceeds about 0.1 — closure onset, denoted t_rise — and the time when closure finishes and the curve enters a high plateau — t_peak (closure peak). For cutting training and evaluation labels offline, the labeling window is [t_rise − 20, t_peak − 8] (frames), i.e. from ~0.4 s before closure onset to ~0.16 s before the closure peak (~50 Hz): the window end stays near imminent closure completion; the start covers a little before closure onset so the labeled interval includes lead time. Relative to a wider control window [t_rise − 50, t_peak − 8], this window has fewer false triggers at the ≈0.8-recall working point. Online, we do not slide this window; the detector scores each policy inference from wrist vision features; when the score crosses a threshold and the commanded chunk is about to close the gripper, we discard the current chunk and clean-regenerate with an empty prefix — without needing a future t_peak.

The detector is trained on independently collected fail/success windows (about 164 train windows, 41 same-distribution validation windows, and about 100 windows from another collection for generalization). At deployment we use a shallow head on wrist vision features; online recall and false-trigger rate were not separately measured (see Limitations).

4.2 Immediate clean regeneration

Same-policy baseline with no runtime intervention: conditional retry success 29.4% (5/17), first-grasp failure 17.0% (17/100). With the empty-grasp detector on and immediate discard of the current chunk plus empty-prefix clean regeneration: conditional retry success 66.7% (12/18); first-grasp failure 18.0% (18/100), same order as baseline, consistent with “the intervention acts on generation after failure is detected.”

On task efficiency, mean successful-episode completion time is 69.9 s (baseline 73.6 s), median 64.4 s (baseline 70.2 s); throughput 35.7 units/hour (baseline 34.5); intervention rate 10.0/hour in both. Shorter completion times are consistent with interrupt-and-retry when the detector fires, without raising human intervention rate.

5. Results summary and limitations

Limitations

• Conditional-retry denominators are often only on the order of a dozen; point estimates fluctuate.
• On the training path, first-grasp failure also falls (26.0% → 17.0%), so the conditional-retry denominator population changes; the two rates are not computed on fully homogeneous failure samples.
• Probe success/failure labels depend on human correction; label definitions can still drift.
• The empty-grasp detector cannot be dropped onto another fine-tuned policy as-is; a new model needs a matching detector retrained with the same protocol.
• Online detector recall and false-trigger rate were not separately measured; §4 real-robot numbers are post-intervention task results.

6. Closing

As a sequel to the previous post, this article re-checks the prefix-inertia bottleneck on retries after a round-table failure, and validates two real-robot fixes: mixing in fail-then-succeed short demos on the training side (conditional retry about 19.2% → 58.8%), and clean regeneration after empty-grasp detection on the deployment side (about 29.4% → 66.7%). The two paths act at different points and can in principle be stacked; stacked performance has not yet been validated on the robot.

Successful-episode completion time also shortens (training 74.7 s → 70.3 s; deployment 73.6 s → 69.9 s), consistent with less wasted early round-table time; deployment human intervention rate is flat, training slightly up. Next steps include stacked validation and online detector recall / false-trigger statistics.