Plans from Demonstrations, Skills from Reinforcement Learning An agentic teacher and student for long-horizon bimanual tower manipulation

1. The problem

Two seven-joint AgileX Nero arms with parallel grippers, called left and back after where they are mounted, share a table with two tower pads. Each pad carries three fixed foundation blocks; the movable blocks are 150 × 50 × 28 mm. Four RGB cameras watch the scene: two fixed cameras on opposite sides of the table and one on each wrist. There are five tasks, each with a two-layer and a three-layer variant, so ten variants in all.

TransferCross transferVertical stackingPush-downInsertion

Operations are grasp-and-place transfers (Tasks 1–3, 5) or pushes (Task 4). Demo lengths are the range over the organizers' demonstrations: 50–56 per variant, 531 in all, recorded at 30 Hz.

What the policy sees and does

The policy runs as a server. At each call it receives the four camera images (640 × 480), the 14 joint angles, the two gripper widths, the simulated time and the task name, and it returns a chunk of up to 16 rows of joint targets. Each row holds 7 joint angles and a gripper width per arm and is applied for 0.1 s, so the robot runs at 10 Hz and an episode of 210 s has up to 2,100 rows. The simulation pauses while the policy thinks.

The policy is never told block poses, depth, velocities, forces, masses, friction or which robustness case it is in. Each variant is evaluated on ten episodes: one nominal and nine drawn from the robustness cases below, on fixed layouts that participants do not see.

Scoring works at three levels: task success (every block where it belongs, nothing else disturbed), operation success (grasps and places, or pushes), and long-horizon progress, the share of the task's steps completed in order. A disturbance rate counts non-target blocks that were moved. Tasks are weighted 1 : 1 : 1 : 3 : 3, so push-down and insertion carry two thirds of the total.

2. Why long-horizon bimanual manipulation is hard

• Long horizons. Up to nine operations and 2,100 control steps per episode. A block nudged in minute one is scored at the end, and the final arrangement is all that counts.
• Millimetre tolerances. A place must land within 5 mm and 5° of its goal. Insertion leaves about 1 mm of clearance on each side. At the fixed cameras one pixel covers about 1.5 mm.
• Contact. Pushing a block out of a full tower without dragging its neighbours along, with friction anywhere from 0.15 to 1.3 and masses that differ by a factor of five.
• Two arms, one workspace. The arms take turns around the same towers and must stay out of each other's way.
• Soft, position-controlled arms. The joints are driven by stiff-but-not-rigid position servos. At rest, gravity alone pulls the back arm's gripper 2–4 cm below its target, and the task descriptions for Tasks 4–5 limit the finger force to 3 N, so heavy blocks can slip.
• Partial observability. No block poses, only images. Blocks inside a tower are hidden, the arms cover the blocks they work on, and a pushed block leaves the views as it tips over the table edge.
• Open-loop chunks. Up to 16 rows, 1.6 s, can pass between two observations.

3. Why not imitation learning alone

The challenge ships demonstrations, and the obvious approach is to clone them. We decided against making that the core of the system. This is our reasoning, not an experiment we ran:

• The demonstrations show one way to succeed, not how to recover. Every demonstration of a variant follows the same plan (the same block order and the same arm for each step), every one succeeds, and in the episodes we inspected the towers vary by only a few millimetres. The evaluation moves towers by up to 15 mm and changes mass and friction several-fold. A cloned policy that drifts off the demonstrated states has no data on how to get back.
• Errors compound over long horizons. Without interaction, the cost of behaviour cloning can grow with the square of the horizon, because each mistake leads to states the expert never visited (Ross et al., 2011). Here the horizon is up to 2,100 steps.
• The control rate does not match. The demonstrations are recorded at 30 Hz with smooth commands. The evaluator holds each 10 Hz row for 0.1 s. When we replayed the recorded commands that way in our simulator, the soft arms swung enough to drop blocks.
• No time to spare. The demonstrations already take up to 181 s of the 210 s budget, leaving no room for a retry at the demonstrated pace.

Imitation itself is not the problem; imitating a fixed set of demonstrations is. Our student is also trained by imitation, but of a teacher we can query in any state, including states we create by injecting failures. So we split the work: take the plan and the geometry from the demonstrations, learn the contact skills with reinforcement learning, and learn the decisions by interactive imitation of our own teacher.

4. An agentic robot system

By agentic we mean a system that works the way a software agent uses tools: it breaks the goal into steps, calls a specialized skill for each step, checks the result, and decides what to do next. The skills are the tools; some are learned with reinforcement learning and some are planned. In the teacher the decisions come from a task graph and an executor; in the student they come from a learned high-level policy. We think this structure fits long-horizon manipulation for five reasons:

• Short problems for reinforcement learning. Each skill is trained in episodes of 8–15 s (80–150 steps) with a dense reward, instead of 2,100 steps with a final score.
• Progress can be checked. Every skill ends with a success check: is the block lifted 6 cm and held, is it within 5 mm and 5° of its goal, did it leave the tower without moving the others? A failed check triggers a retry instead of silently corrupting the rest of the episode.
• Reuse. One grasp policy per arm serves every transfer variant; one push policy per arm serves both push-down variants.
• Mix and upgrade. Plan where planning is precise (free-space motion, the last millimetre of an insertion) and learn where contact is uncertain (grasping, placing, pushing). Components can be replaced one at a time.
• The structure survives distillation. The student keeps the same skill calls and changes only where the state and the decisions come from.

In the language of hierarchical reinforcement learning, the high-level policy \(\mu\) picks an option (Sutton et al., 1999), a subtask with a goal block and an arm, from the estimated state \(\hat s_t\) and a memory \(m_t\) of earlier subtasks and their outcomes, and the option's skill produces the actions until its success check ends it:

4.1 Recreating the evaluator

The evaluator itself is not public, so we rebuilt it in Isaac Lab (Mittal et al., 2023) from the official station and block assets. Every number that matters was matched to the organizers' data:

• Physics. PhysX on the CPU, as in the evaluator, at 120 Hz. Each 10 Hz row is held for 12 physics steps with a one-step command delay. Replaying ten demonstrations on the CPU ends with every block within 0.89 mm of the recording. On the GPU solver the blocks creep 8–19 mm, so we train on the GPU but accept results only on the CPU.
• Robot. The joint stiffness, damping and friction of both arms were read back from the recordings. Our arm controller compensates gravity sag analytically; the model torques match PhysX within 0.1–2 % per joint.
• Cameras. Calibrated against the dataset images; projected block edges fall within 0.25 px.
• Randomization. The ten robustness cases, plus wider ranges for training: mass × 0.35–2.2 per block, static friction 0.13–1.4, and random finger friction.
• Scoring. Our own reconstruction of the grasp, place, push and success checks and of the metrics.

5. The teacher

The teacher is privileged: it reads the exact pose of every block from the simulator, including hidden ones. Its decisions come from a task graph per variant and a deterministic executor; the contact skills are PPO policies.

5.1 Task graphs and the executor

A task graph is a list of skill calls built from the demonstrations' plan. A transfer is four calls per block (move to the pre-grasp pose, grasp and lift, carry to the pre-place pose, place and release), a push-down two per block (move behind the block, push), an insertion six per block (approach, grasp, carry to the slot, insert, move behind the block, push it home). The idle arm is parked whenever the working arm changes. The graphs have 7–27 calls.

{"id": "t0_grasp", "skill": "grasp_lift", "arm": "back", "block": "source_l2_a", "lift_m": 0.06, "timeout_s": 15.0}
{"id": "t0_place", "skill": "place_release", "arm": "back", "tol_pos_m": 0.005, "tol_yaw_deg": 5.0, "settle_s": 1.0}

Two calls from the Task 1-2 graph (some fields left out).

The executor runs at 10 Hz. Each call it builds the current skill from the state, steps it, and judges its outcome when it ends: lifted and held, placed within tolerance, pushed clear with the rest of the tower within 10 mm, inserted to depth. On a failure it retries (up to three attempts per block: re-grasp, re-place, or re-carry if the block is still held), skips the block, or, once per block, hands a place that missed by up to 15 mm to the other arm when its own arm cannot reach the block again. Before every move it parks the other arm if it is within 18 cm of the target.

5.2 Learned contact skills

Grasp-and-lift, place-and-release and push are trained with PPO (Schulman et al., 2017), one policy per skill and arm, in light GPU scenes with one arm and up to 15 blocks, 3,072–4,096 environments in parallel. The setup is an asymmetric actor-critic (Pinto et al., 2018). The actor sees only what a camera-based system could estimate: noisy block poses (a per-episode bias of 2–3 mm and 2–3°, per-step jitter, occasional dropouts and 1 cm outliers) and the arm's own state. The critic additionally sees the true poses, block mass and friction, finger contact forces and joint velocities. Because the actors never relied on privileged inputs, the student reuses them unchanged.

We use the clipped PPO objective with generalized advantage estimation (Schulman et al., 2016), implemented in rsl_rl (Rudin et al., 2022):

The actor \(\pi_\theta\) reads the actor observation \(o_t\) (65–73 inputs); the critic \(V_\phi\) reads the privileged state \(s_t\) (141–161 inputs). \(\epsilon=0.2\), \(\gamma=0.99\), \(\lambda=0.95\), \(c_v=1\) (with a clipped value loss), \(c_e=0.006\); the learning rate adapts to keep the KL divergence between updates near 0.016. Networks: actor 256-256-128, critic 512-256-128, ELU.

From actions to joint targets

A skill's action is a small change of the gripper's target pose in the arm's base frame, plus a gripper command for grasp and place. It is smoothed and scaled with the distance to the goal (fine steps near the block, larger ones far away), then turned into joint targets by damped least-squares inverse kinematics with a posture prior, and finally corrected for gravity sag:

\(s(d)\) grows from 5 to 20 mm (grasp), 4 to 15 mm (place) or 12 to 15 mm (push) per step as the distance \(d\) to the goal grows; \(\lambda=0.02\); four iterations per step, at most 0.15 rad per joint per step; \(\tau_g\) is the gravity torque of the arm (and of a nominal block when holding one), \(K_p\) the joint stiffness.

Rewards

The push skill shows the pattern every skill follows: shaping toward the right contact, progress, penalties for disturbing anything else, and a large bonus for finishing early.

\(x_t\) is the block's travel along the push direction, clipped to [−0.05, 0.16] m, so the progress term is potential-based; \(y_t\) its sideways travel, \(\Delta\psi_t\) its yaw change in degrees, \(d_j\) the displacement of every other block. \(r^{\text{align}}\) rewards the fingertips for reaching the contact pose 86.5 mm behind the block; \(r^{\text{limits}}\) penalizes joints within 0.1 rad of their limits. Success: the block has left the tower (0.16 m from its axis, or 0.09 m and on the table) while no other block moved more than 10 mm, tilted more than 30° or dropped more than 15 mm. Failure: the tower broke, or the block was shoved more than 4 cm sideways. Episodes last 8 s.

Training and results

Each policy trains for a few hundred to about a thousand PPO iterations at about 12,000 environment steps per second; the left grasp policy took about 850 iterations, 84 million steps and 2.1 hours. Measured on 500 episodes per setting on the GPU:

Independent 2 mm biases on the block and the goal limit any place to about 90 % at a 5 mm tolerance, which is why the noisy-pose numbers are lower.

5.3 Planned skills

• Transit moves an arm between skills: rise 2 cm above the higher of start and goal, travel straight, descend, with quintic timing (0.25 m/s empty, 0.08 m/s carrying a block, which stopped heavy blocks from shaking loose).
• Park moves an arm home in joint space.
• Task 5 grasp, insert and push-in are closed-loop controllers. The insert re-aims every step from the observed block and advances 1 mm per step only while it is aligned within 0.8 mm and 0.4°. We also trained a PPO insertion policy: 27 % success on the GPU but 0 of 32 on the CPU, because contact at 1–2 mm clearances differs between the two solvers. The planned controller is reliable on the CPU.

With exact poses, the teacher completes 1,186 of 1,200 episodes on Tasks 1–3 and 446 of 480 across all ten variants (nominal and robustness cases, CPU physics).

6. The student

The student is what the evaluator talks to. It sees only the official observation. It keeps the teacher's structure and executor code but replaces three things: the exact state (by perception), the task-graph pointer (by a learned high-level policy) and the planned insertion controllers (by distilled networks). The task plan, the list of goals and the arm prescribed for each, still comes from the public demonstrations; nothing is taken from the evaluation layouts.

6.1 Perception: keypoints, then geometry

The perception is a class-agnostic detector followed by model-based fitting. A keypoint network finds block corners and edges in each image without knowing which block is which. Identity, pose and confidence come from fitting the known box models to the detections in all four views together, with a tracker that carries each block from call to call.

Network. A U-Net (Ronneberger et al., 2015) on a ResNet-34 encoder (He et al., 2016) outputs four full-resolution maps: corners, edges, block surface, and block surface hidden by an arm or finger. Corners and edges are trained with the quality focal loss (Li et al., 2020) against Gaussian targets:

\(\sigma_Y\) is 1.5 px for corners and 1.0 px for edge points; \(N^{+}\) is the number of pixels with target above 0.5; \(M\) the block mask, \(O\) the arm-covered mask (positive weight 5). Training data: 28,227 frames rendered from teacher rollouts in our simulator with randomized lighting, plus 2,800 frames of the organizers' dataset, which we sample three times as often; photometric, geometric and JPEG augmentation. On 400 held-out renders, 98.6 % of corners are found within 3 px (97.8 % precision) with a median error of 0.18 px.

Multi-view fit. Each block is a box with 8 corners and 152 edge samples. Its pose \((R,t)\) minimizes a robust reprojection error over all four cameras (the wrist cameras posed by the arm's forward kinematics), plus a prior from the tracker:

\(\pi_c\) projects into camera \(c\); \(u_{ci}\) is the corner peak matched to corner \(X_i\); \(e_{cj}\) the nearest edge maximum along the projected edge normal \(n_{cj}\); \(s\) the network's peak values; \(w_{\text{c}}=1\), \(w_{\text{e}}=0.3\). \(\rho\) is the Huber loss (1.5 px) for the first two Gauss-Newton iterations and Tukey's loss (2.5 px) for the remaining four; points hidden by other blocks or by an arm are left out. The tracker's prior \((t_0,R_0)\) has base standard deviations of 10 mm and 0.05 rad, tightened for blocks at rest.

Tracker. A set of rules rather than a filter: a block that moved last call is predicted to move as much again; a block in a closed gripper follows the gripper's forward kinematics with a slowly updated in-hand offset; resting blocks are low-pass filtered and anchored; a lost moving block coasts for up to five calls; lost blocks are searched for on the table, on other blocks or leaning. Each block gets one of four confidence levels.

Accuracy and speed. On the organizers' frames, resting blocks are within 0.18 mm (median) and 0.62 mm (90th percentile), held blocks within 0.2 mm (90th percentile). The hard case is an inserted block nearly hidden under the roof of the Task 5 tower (90th percentile 3.5 mm). A call takes about 93 ms on our server, 86 ms of it perception.

6.2 The high-level policy

The high-level policy decides what to do next whenever a subtask ends (and every 2 s while idle). It is a transformer encoder (Vaswani et al., 2017) over an unordered set of tokens: one global token (task, time left, perception health), two arm tokens (gripper pose, holding, parked), up to 16 block tokens (estimated pose relative to the pads, confidence, how far it moved, offset to its goal) and up to 9 goal tokens (goal pose, attempts and outcomes so far, which arm can reach it, whether it is done). It has 3 layers, 4 heads and width 128, under one million parameters, and decides in 1–5 ms on a CPU. The decision is factorized: first the subtask type (acquire, deliver, push off, insert, push home, park, idle, re-push), then the goal by pointing at a goal token (Vinyals et al., 2015), then the arm:

Invalid choices are masked (for example, delivering a block the arm does not hold). A fourth head predicts, for every goal, whether it is satisfied right now.

The expert. The labels come from a privileged expert: the task graph's decision logic, but reading the simulator's truth about which goals are satisfied. It closes goals that are done even after a failed check, reopens a finished goal that was disturbed later, never hands a block to the other arm (that recovery failed in 206 of 217 tries for the teacher), gives up on tilted blocks, and leaves failed goals that are now buried under later layers alone.

Behaviour cloning, then DAgger. Rollouts run in the recreated evaluator with block poses passed through a perception-error model: recorded errors of the real estimator, replayed by block class (resting, held, moving, being pushed, foundation, fallen) with their confidence levels and the chance of losing a pushed block as it tips. With probability 0.15 per subtask a failure is injected (a shifted pre-grasp, a dropped block, a shifted goal, an off-centre grip, a short push), so that the data contain recoveries. Round 0 is driven by the expert with 10 % random exploration; three DAgger rounds (Ross et al., 2011) follow, in which the learner's decision is executed and the expert only labels (with probability β the expert's decision is executed instead):

\(\beta\) = 0.5, 0, 0.5 in the three DAgger rounds. Off-nominal rows (after a failure, an injection or exploration) weigh five times more. \(y_{bj}\) is the true "goal \(j\) satisfied" flag. Each round retrains from scratch on all data so far: 3,600 episodes and 65,242 decisions in the end. Validation decision accuracy 96.8 % (95.5 % on off-nominal rows); the satisfied-goal head reaches an AUC of 0.999.

6.3 Skills in the student

• Grasp and place: the teacher's PPO policies, unchanged.
• Push: the teacher's PPO policies fine-tuned under the perception-error model, with two new inputs (whether the target block is currently seen, and for how long it has not been). The teacher's push froze for up to 1.5 s whenever the target was lost, which happens as a block tips over the edge. Success under the error model, robustness cases: 98.8 % (left), 95.4 % (back).
• Task 5 grasp, insert and push-in: distilled from the planned controllers with DART (Laskey et al., 2017), so that the networks see the states where the controller has to correct itself.
• Transit and park: planned, as in the teacher.

For DART, a slowly varying perturbation is added to the planned controller's target before every recorded step, and the label is the controller's correction from the perturbed state:

\(\sigma\) per step is 1 mm and 0.15° (grasp), 0.2 mm and 0.04° (insert), 0.5 mm and 0.1° (push-in), scaled by a factor drawn from U(0, 1.5) per segment. The network is an MLP 68 → 256 → 256 → 256 → 7 with tanh-bounded motion outputs and a gripper logit, trained on 328,113 steps from 1,200 episodes. Distilled, the three skills succeed in 83 of 96 Task 5 test episodes against 78 for the planned controllers they imitate. One lesson: when the previous action was an input, the networks learned to copy it and drifted in closed loop; the previous-action input is now masked out.

6.4 From the first student to the submitted one

Testing against the official evaluator changed four things between our first student and the submitted one, student v1.3:

• Camera orientation. On Tasks 1–3 the evaluator delivered the two fixed cameras rotated by 180° relative to the public dataset, which our first student did not expect. The submitted student picks the orientation of each episode by correlating its first images with dataset references.
• Knowing when to stop. Official episodes ran to the full 210 s even after success. A high-level policy never trained on finished tasks sometimes reworked goals that were already done: under our error model, Tasks 1–3 succeeded in 1 of 120 episodes without a guard against this and in 120 of 120 with it. The submitted high-level policy was retrained with richer goal tokens (done, moved since, how long ago) and an expert that leaves buried failures alone, and it learned to idle: the guard did not intervene once in 240 test episodes.
• Learned recoveries. The new policy also learned to push a Task 5 block home again when the push-in stopped short, and to redo a failed place at once.
• Two hand-written rules. After the push-down plan, a block seen leaning on the base is pushed again. And a block gripped 4 mm or more off-centre is placed by a slower planned place instead of the learned one: with every grip forced 4–8 mm off-centre on three transfer variants, success fell from 97 % to 81 % with the learned place and returned to 97 % with the rule.

The submitted student is one frozen bundle of networks and configuration files served over a WebSocket, with independent state per evaluation session.

7. Training pipeline

Rebuilding the submission from the public downloads takes about 40 hours on one RTX A5000 (24 GB) and a 24-thread CPU: PPO on the GPU, rollouts and every acceptance test on CPU physics.

Lessons

• Check the real observations of every task. The camera rotation on Tasks 1–3 cost us a whole official run's worth of perception on those tasks.
• Reproduce the official scoring locally. Run every episode to the time limit and judge the final arrangement; behaviour after success matters.
• The CPU solver is the evaluator. Skills trained on the GPU move differently on the CPU; accept only on the CPU.
• Never feed a cloned network its own previous action. It learns to copy it.
• Train under the deployment's information pattern. Chunked observations (below) and target losses (pushes) had to be part of training, not patched in afterwards.
• After an environment or reward redesign, retrain from scratch. Resumed runs inherited bad habits.

8. Results (local evaluation)

We test the student two ways. On CPU physics with the perception-error model, many episodes are cheap. In the closed loop on rendered camera images, our simulator renders the four cameras and the student's real perception runs on them; this is the closest to the official setting we have, but slow and not deterministic. Both run every episode to 210 s and judge success on the final arrangement, as the official evaluation did. Every test covers the ten robustness cases.

Student v1.3. Error model: 24 episodes per variant over two seeds. Rendered cameras: 10 episodes per variant per seed; two seeds where two numbers are given. Progress is our reconstruction of the operation-level progress rate.

• Transfer and stacking are nearly solved: 59 of 60 episodes on rendered cameras. The one failure (2-2, high friction) was a place that opened 14 mm above its goal; the high-friction fingers flung the block out of every camera's view.
• Push-down is where most points are lost. A success needs every block of the tower pushed clear without disturbing the rest. On 4-3 (nine pushes), 55 of the 90 pushed blocks ended flat and clear of the base, but no episode got all nine. The usual failures: a push drags a neighbour along, an early top-layer push shifts the rows below, lower-layer blocks land on the debris of the upper ones, and a lost block's coasted estimate ends a push too early, leaving the block leaning on the base.
• Insertion fails mostly at the very end: the inserted block is nearly hidden under the roof, its estimated depth can be off by 5–10 mm, and the push-in stops short. The rendered-camera results on 5-3 swing from 2 to 7 of 10 between seeds, a reminder of how noisy ten-episode tests are.

Did distillation work?

The fair comparison is teacher and student on the same rendered images, so that both depend on the same perception. For our first student (same skills and perception as the submission, an earlier high-level policy), on all ten variants with 8 nominal and 8 robustness-case episodes each, the student succeeded in 135 of 160 episodes and the teacher in 131. The student matches its teacher; what limits both is perception and contact on Tasks 4 and 5, not the decisions.

Successes out of 16 per variant (8 nominal, 8 robustness cases), both policies on block poses estimated from our rendered images. Episodes in this test ended at success.

9. Videos

Student v1.3 in the closed loop on our rendered images: the front camera (top left), the camera behind the left arm (top right) and the two wrist cameras, exactly the images the student receives, with its own estimates of the block poses drawn on top. One frame is shown per policy call, so arm motions that send 16 rows per call appear sped up.

Box colours: green blocks seen with full confidence, magenta the block of the current subtask (labelled with its name: source or destination tower, layer, position), cyan a block in the gripper, blue foundation blocks, yellow, orange and red lower confidence, not found in this call, or lost. A white box is the simulator's true pose, drawn only where the estimate is more than 2 mm off; the student never sees it.

All ten variants

4:47, silent. One nominal episode of every variant (the first successful one of two rendered), then two robustness cases, heavy blocks and changed lighting. Each starts with a card that gives its outcome and the simulated time at which it succeeded. Of the twenty nominal episodes we rendered, eighteen succeeded; the two that did not were one three-layer push-down (shown in the results above) and one three-layer insertion. Download the video (MP4, 17 MB). Chapters:

• 0:00Task 1-2: Tower transfer, 2 layers
• 0:25Task 1-3: Tower transfer, 3 layers
• 1:03Task 2-2: Cross transfer, 2 layers
• 1:17Task 2-3: Cross transfer, 3 layers
• 1:37Task 3-2: Vertical stacking, 2 layers
• 1:51Task 3-3: Vertical stacking, 3 layers
• 2:10Task 4-2: Push-down, 2 layers
• 2:29Task 4-3: Push-down, 3 layers
• 3:00Task 5-2: Insertion, 2 layers
• 3:24Task 5-3: Insertion, 3 layers
• 4:14Task 2-3: Cross transfer, 3 layers, heavy blocks
• 4:34Task 3-2: Vertical stacking, 2 layers, changed lighting

10. Next steps: student v1.5 and more learned components

The submitted student still plans every free-space arm motion: the transits between skills follow an inverse kinematics path with fixed timing. After the deadline we replaced them with learned reaching, which did not get an official evaluation because submissions had closed. We include it because it shows the point of the agentic structure: one planned component was swapped for a learned one without retraining the high-level policy or any other skill.

Learned and planned components across versions. Student v1.4 and v1.5 are the same system with learned reaching; v1.5 uses a later checkpoint of the back arm's reacher.

Learned reaching

The reacher follows our earlier work on joint-space reaching with reinforcement learning, JAiLeR (Kumar et al., 2021): a goal-conditioned PPO policy per arm that outputs joint velocities directly, without inverse kinematics. The goal is a pose for the gripper. During a transit it slides from waypoint to waypoint at a speed limit (0.05 m/s for the left arm and 0.15 m/s for the back arm when carrying a block), so the reacher always tracks a nearby moving goal. Training includes carrying blocks of random mass and friction with random grip offsets.

\(d\) compares three points 10 cm along each axis of the goal frame \((g,R_g)\) and of the gripper frame \((p,R_p)\). The first term is JAiLeR's; the two wider Gaussians give gradient from far away, and \(r^{\text{hold}}\) rewards staying still on the goal. The remaining penalties keep the joints off their limits and the gripper off the table, and, when carrying, punish slip and drops. The action costs act on the raw network output, not the clipped command; otherwise PPO gets no gradient from them.

Training under the deployment's information pattern. The evaluator executes up to 16 rows per call, so the reacher must act for up to 1.6 s without a new measurement. A reacher trained with a fresh measurement every step failed under 16-row chunks: it carried blocks at about twice the goal's speed, dropped held blocks and fell into limit cycles. We retrained it with chunked observations: inside a chunk, the policy sees the joint prediction \(q_{\text{des}}-(q_{\text{des}}-q)_{\text{first row}}\) and the number of rows since the last measurement. That alone fixed it.

Arrival errors, drops, episode lengths and teacher results: CPU physics. Student rows: closed loop on rendered cameras, the same seeds for both. Learned reaching drives every transit except the short moves before a push or a push-in: there, contact skills were sensitive to the exact arm posture the transit left behind, and the learned arrivals cost Task 4 and 5 episodes. Student v1.4 and v1.5 differ only in the back arm's reacher checkpoint; in a paired comparison over 512 episodes they succeeded in 493 and 491 (no detectable difference).

The reacher makes the plans faster and the arm motions closed-loop while keeping success where it was. It does not yet buy points, because the points are lost in pushing and insertion. The directions we see from here:

• Push-down (where most points are lost): pushes that stop on what the camera sees rather than on a coasted estimate, a learned decision of when a push is done, and pushes that do not drag neighbours.
• Use the high-level policy's "goal satisfied" head as the judge of when a subtask is done, in place of hand-written success checks.
• A place policy trained on off-centre grips, to retire the hand-written slow place.
• Learned reaching everywhere: arrival postures matched to the next contact skill, so that the moves before a push and a push-in can be learned too; recovery pushes.

11. Reproducibility

The code is at github.com/VisakK/tower_rl. Its reproduce/ folder rebuilds student v1.3 from the public downloads in seven stages, one script each: data, teacher, perception, student, freezing the bundle, checks with the numbers to compare against, and serving it on the challenge protocol. It needs Isaac Sim 5.1 and Isaac Lab 2.3 and about 40 hours on one GPU. Trained weights are not in the repository. GPU training is not bit-exact, so a rebuild gives an equivalent policy rather than identical bytes.

References

1. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017.
2. J. Schulman, P. Moritz, S. Levine, M. Jordan, P. Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR, 2016.
3. L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, P. Abbeel. Asymmetric Actor Critic for Image-Based Robot Learning. RSS, 2018.
4. S. Ross, G. Gordon, J. A. Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS, 2011.
5. M. Laskey, J. Lee, R. Fox, A. Dragan, K. Goldberg. DART: Noise Injection for Robust Imitation Learning. CoRL, 2017.
6. R. S. Sutton, D. Precup, S. Singh. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence, 1999.
7. V. Kumar, D. Hoeller, B. Sundaralingam, J. Tremblay, S. Birchfield. Joint Space Control via Deep Reinforcement Learning. IROS, 2021. arXiv:2011.06332.
8. X. Li, W. Wang, L. Wu, S. Chen, X. Hu, J. Li, J. Tang, J. Yang. Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection. NeurIPS, 2020.
9. O. Ronneberger, P. Fischer, T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI, 2015.
10. K. He, X. Zhang, S. Ren, J. Sun. Deep Residual Learning for Image Recognition. CVPR, 2016.
11. A. Vaswani et al. Attention Is All You Need. NeurIPS, 2017.
12. O. Vinyals, M. Fortunato, N. Jaitly. Pointer Networks. NeurIPS, 2015.
13. N. Rudin, D. Hoeller, P. Reist, M. Hutter. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. CoRL, 2022.
14. M. Mittal et al. Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments. IEEE RA-L, 2023 (now Isaac Lab).
15. TOWER Benchmark contributors. TOWER-SimData and TOWER-Assets. Hugging Face, 2026. CC BY 4.0.

BibTeX

@misc{kumar2026agentic,
  title        = {Plans from Demonstrations, Skills from Reinforcement
                  Learning: An Agentic Teacher and Student for
                  Long-Horizon Bimanual Tower Manipulation},
  author       = {Kumar, Visak},
  year         = {2026},
  month        = nov,
  howpublished = {Blog post},
  url          = {https://visakk.github.io/agentic-tower/}
}