Research Overview

1. Hardware

Our Trossen AI Stationary was purchased in May 2025. We are still running Trossen Arm Driver v1.7.8. Our local computer is a System76 Thelio Mira desktop running Ubuntu 22.04 with an RTX 5090 GPU. The RTX 5090 is used for LoRA fine-tuning of pi0, but for full fine-tuning we have been using H100 GPUs remotely on runpod.io.

2. Software

We started by augmenting and tweaking the gym-aloha environment, as well as the (deprecated) Trossen lerobot framework, with the goal of providing seamless sim to sim, sim to real, and real to sim support for the Trossen AI Stationary robot. We have also been tweaking the lerobot software for smoother Trossen AI Stationary real robot dataset acquisition. In addition, we have added real and simulated Trossen AI Stationary support to the openpi framework. Very recently, we started experimenting with Isaac GR00T N1.7 on our robot too! Our forks are at github.com/anredlich. Highlights:

• gym-aloha: we added *.xml mujoco files to the assets folder and augmented the sim.py and sim_end_effector.py simulator code to give gym-aloha the ability to simulate the Trossen AI Stationary robot (mujoco files and code adapted from trossen_arm_mujoco). This includes both joint controlled and end-effector controlled simulations for the transfer-cube task. For this task, we added environmental options such as box size, box position, box orientation, and box color, as well as some control over lighting, robot joint reference angles, and robot base positions.
• lerobot: we added control_sim_robot.py which uses the augmented gym-aloha environment to create and replay simulated datasets for the Trossen AI Stationary robot. We also added scripted_policy.py, a heuristic waypoint policy adapted from trossen_arm_mujoco, for the simulated robot rollouts. In addition, we modified train.py and eval.py so that they can train and evaluate policies for the simulated Trossen AI Stationary robot. Together these additions allow full sim to sim, sim to real, and real to sim evaluations. Combining simulated and real robot replay can also be used to calibrate/match the simulated to the real robot. We also added 4 new evaluate_*.py and train_*.py example files for both the old aloha and the new Trossen AI simulated robots.

  To improve dataset acquisition and workflow for the real robot, we added better text to voice and additional voice prompts to control_robot.py . More recently, we added a task splitting tool, dataset_splitter.py, which takes a long high-level task, and splits it into sub-task with their own sub-task prompts. We also added a dataset_merge.py tool to pool lerobot datasets into a single dataset. (Note that all our mods are built on the older Trossen lerobot fork. We have not yet ported to the newer plugin version of lerobot.)

• openpi: we have added hardware driver support to run pi0/pi0.5 policies on the Trossen AI Stationary Robot within the openpi framework. This was done by adapting the trossen_ai example, in particular the TrossenOpenPIBridge class in main.py, from the Trossen openpi fork. We have more recently added record.py which allows human intervention and teleoperation during a pi0/pi0.5 policy rollout. It records this improved episode in lerobot format, and can also record pure rollouts and pure teleoperation, see Policy Improvement using Human Interventions. We have also added an aloha_sim_trossen_ai example with full simulated Trossen AI Stationary robot support using our gym-aloha fork in place of the gym-aloha that downloads with openpi.

  (Note that we still use the older v1.7.8 robot driver and deprecated lerobot code in our main.py and record.py. To use the newer versions, both main.py and record.py need to be brought back closer to the original Trossen code. There are, however, some other differences between our fork and the Trossen fork that we believe might be important, including in aloha_policy.py, which we discuss here.)

• Isaac GR00T N1.7: we forked — see our develop branch — the Isaac GR00T repository and added the example folder trossen_ai with everything needed to run GR00T models on our Trossen Stationary AI (1.7.8 Trossen driver for the moment). We then trained on our transfer-cube dataset using an H100 GPU on runpod.io. The resulting policy was crazy shaky and almost destroyed the robot, so we added EMA low-pass filtering; re-trained with the action chunk increased from 16 to 32; and finally used the Trossen robot clipping variable max_relative_target to remove any remaining large action deltas. This smoothed out the robot! We look forward to testing GR00T on more tasks.

3. Datasets

We have been acquiring and uploading — to huggingface — both real robot and simulated robot datasets. The real robot datasets were acquired using the lerobot control_robot.py with the record option, or using record.py in our openpi — for human interventions. The simulated datasets were acquired using our control_sim_robot.py with the record option. These datasets can be visualized using lerobot's visualize_dataset.py tool or online at lerobot/visualize_dataset. See the anredlich/lerobot readme for more details. Datasets have 50-500 episodes. Here are the dataset repo_ids:

Real robot:

• ANRedlich/trossen_ai_stationary_transfer_20mm_cube_01
  small red cube.
• ANRedlich/trossen_ai_stationary_transfer_40mm_cube_02
  large red cube.
• ANRedlich/trossen_ai_stationary_transfer_multi_cube_03
multiple cube sizes, colors, and orientations.
• ANRedlich/trossen_ai_stationary_place_lids_04
pick up lid and place on pan. 6 different lids and 8 pans.
• ANRedlich/trossen_ai_stationary_pour_box_05
  pour small cube from one cup to another.
• ANRedlich/trossen_ai_stationary_pop_lid_06
  pop lid on small plastic takeout container.
• ANRedlich/trossen_ai_stationary_pick_and_place_07
  multi-task: e.g. 'pick up red cube and place in silver pan'
• ANRedlich/trossen_ai_stationary_pick_and_place_08
  multiple sub-tasks that flow into each other without going home first (250 episodes, ~30min total).
• ANRedlich/trossen_ai_stationary_pick_and_place_09
  larger version of 08 (500 episodes, ~60min total).
• ANRedlich/trossen_ai_stationary_place_bead_on_string_10
  high dexterity, small objects (50 episodes).
• ANRedlich/trossen_ai_stationary_place_lids_13
  rollouts from place_lids_04 policy with human intervention at failure.
• ANRedlich/trossen_ai_stationary_place_bead_on_string_14
  one iteration of human interventions at failure added to _10 (100 episodes).
• ANRedlich/trossen_ai_stationary_place_bead_on_string_15
  second iteration of human interventions at failure added to _14 (150 episodes).
• ANRedlich/trossen_ai_stationary_close_tie_wrap_16
  high dexterity (50 episodes).
• ANRedlich/trossen_ai_stationary_close_tie_wrap_17
  one iteration of human interventions at failure added to _16 (100 episodes).

Simulated robot:

• ANRedlich/trossen_ai_stationary_sim_transfer_40mm_cube_07
  cube color=red, size=40mm, tabletop=black, background=none, lighting=bright.
• ANRedlich/trossen_ai_stationary_sim_transfer_40mm_cube_08
  cube_color=dark red, size=40mm, tabletop=mine, background=mine, lighting=medium.
• ANRedlich/trossen_ai_stationary_sim_transfer_40mm_cube_10
  cube_color=r,g,b, size=25,40mm, tabletop=mine, background=none, lighting=bright.
• ANRedlich/trossen_ai_stationary_sim_transfer_40mm_cube_13
  cube_color=red, tabletop=mine, background=mine, lighting=medium.

tabletop=mine is an image of my tabletop; background=mine is a crude image of my office walls.

4. Models

We have been acquiring and uploading — to huggingface — learned models/policies for both the real and simulated robot datasets. The ACT models were trained with chunk_size=100 for 100K steps for the transfer-cube task. Both the real and simulated ACT models can be tested in simulation using our lerobot's eval.py, or for individual episodes, using our evaluate_trossen_ai_stationary_policy.py. See our lerobot readme for more details.

We also uploaed a few pi0 models. Openpi doesn't support huggingface models directly, but they can be downloaded and used as shown here. These pi0 training and evaluation details might be helpful too.

Real robot ACT models:

• ANRedlich/trossen_ai_stationary_real_act2_3
  best real to sim, try in evaluate_pretrained_trossen_ai_policy.py, still only about 20% correct! see Real to Sim section, Fig 5.
• ANRedlich/trossen_ai_stationary_real_act5
  see ACT Fig 7.
• ANRedlich/trossen_ai_stationary_real_act6
  see ACT Fig 5.

Simulated robot ACT models:

• ANRedlich/trossen_ai_stationary_sim_act7
• ANRedlich/trossen_ai_stationary_sim_act8
• ANRedlich/trossen_ai_stationary_sim_act10
  see Fig 1 below.
• ANRedlich/trossen_ai_stationary_sim_act13
  best sim to real policy, but still very sensitive to conditions, see ACT Sim to Real, Fig 4.

Real robot pi0 models:

• ANRedlich/trossen_ai_stationary_real_pi03
  LoRA fine-tuned from pi0_base, see High Dexterity Fig 4.
• ANRedlich/trossen_ai_stationary_real_pi04
  Full fine-tuned from pi0_base, see High Dexterity place-lids discussion.

Simulated robot pi0 models:

• ANRedlich/trossen_ai_stationary_sim_pi013
  LoRA fine-tuned from pi0_base, see Sim to Real Fig 1.

Fig 1. Trossen AI Simulated Robot

ACT model trossen_ai_stationary_sim_act10 was learned from dataset trossen_ai_stationary_sim_transfer_40mm_cube_10.