Isaac GR00T N1.7

Results at a Glance

A first test of the latest NVIDIA Isaac GR00T N1.7 VLA model on the Trossen Stationary AI did not go smoothly! The robot learned the policy but it shook so violently it looked like it might destroy itself. EMA low-pass filtering, a longer action chunk, and some action clipping got it smoothed out.
After training on a transfer-cube dataset with only red cubes, a GR00T model was able to generalize the task to other cube colors.

Resources: Isaac GR00T · Notes & Optimizations · Datasets

Very First Experiment

Fig 1. Isaac GR00T N1.7 model rollout.

Note: dataset only contains red cubes!

We trained an Isaac GR00T N1.7 model on our red cube dataset for 30K training steps. As shown in Fig 1, the model successfully learned to transfer the cube, and it also generalized to other cube colors. This is our best result. Initially, the robot shook violently, but after some trial and error, we were able to smooth out the robot using a combination of EMA low-pass filtering, an increase in the action chunk length to 32, and robot action clipping. See the next section for implementation details. Note that our ACT models also required action clipping to smooth out their trajectories, while our openpi models did not seem to need any smoothing or clipping.

EMA low-pass filtering: At run time, actions a_t are filtered using a'_t = α*a_t + (1-α)*a'_t-1, where a' is the filtered action that is sent to the robot. After some trial and error, the best value was α=0.25.
Action chunk size: The best model was trained with the action chunk size set to 32 in the config file, trossen_ai_config_h32.py. Also, the open_loop_horizon option was set to 32 when running the robot using main_gr00t_trossen.py. The longer action chunks seem to be less noisy and also more reliable.
Action clipping: The Trossen robot parameter, max_relative_target, was set to 0.1 which keeps the maximum change in any action — joint value — below 0.1 radians. At this value, clipping happens only once in a while, but when it does, it prevents sudden large movements.
Real-time chunking: Introduced by Physical Intelligence, Real-Time Action Chunking (RTC) is designed to interpolate successive action chunks to create a smooth, consistent trajectory. GR00T has the RTC machinery, but to use it requires some mods to gr00t/policy/gr00t_policy.py and additions to examples/trossen_ai/main_gr00t_trossen.py. We tried RTC to solve the robot shaking problem but it did not help. We believe this is because intra-chunk noise was a bigger problem than inter-chunk discontinuities.

Implementation Details

The code here uses our fork of Isaac GR00T N1.7. At the moment, the latest code is in the develop branch. The commands below are not an exhaustive set of instructions. You first need to install our develop branch or install the original repository directly from GR00T, and then cut and paste our mods in if you want to use them. We did not add any dependencies not in the original GR00T repository.

Config file: To train a GR00T model on a lerobot dataset for the Trossen Stationary AI robot, it is necessary to add a configuration file which tells GR00T what robot data formats to expect. The one we used in our training example is trossen_ai_config_h32.py:

# SPDX-FileCopyrightText: Copyright (c) 2026 ANRedlich. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # # Modality config for the Trossen Stationary AI bimanual robot. # # Robot details: # - 2 arms x 7 joints each (joint_6 is the gripper on each arm) # - 14-dim concatenated state and action vectors # - 4 cameras: cam_high, cam_low, cam_left_wrist, cam_right_wrist # # Layout of the 14-dim state/action vectors (per meta/info.json column names): # indices 0-5 : left_joint_0 .. left_joint_5 (left_arm) # index 6 : left_joint_6 (left_gripper) # indices 7-12 : right_joint_0 .. right_joint_5 (right_arm) # index 13 : right_joint_6 (right_gripper) from gr00t.configs.data.embodiment_configs import register_modality_config from gr00t.data.embodiment_tags import EmbodimentTag from gr00t.data.types import ( ActionConfig, ActionFormat, ActionRepresentation, ActionType, ModalityConfig, ) trossen_ai_config_h32 = { # 4 cameras: high (third-person), low (third-person), left wrist, right wrist. # delta_indices=[0] = current frame only (no temporal history). "video": ModalityConfig( delta_indices=[0], modality_keys=["cam_high", "cam_low", "cam_left_wrist", "cam_right_wrist"], ), # State: split the 14-dim qpos into 4 named blocks for per-block normalization. # Splitting separates the gripper (0-0.044 m linear stroke) from the arm joints # (radians) so each gets its own normalization range. "state": ModalityConfig( delta_indices=[0], modality_keys=["left_arm", "left_gripper", "right_arm", "right_gripper"], ), # Action: 32-step (was 16) prediction horizon. Same key structure as state. # ABSOLUTE matches the openpi training convention used previously for this robot. "action": ModalityConfig( #delta_indices=list(range(0, 16)), delta_indices=list(range(0, 32)), modality_keys=["left_arm", "left_gripper", "right_arm", "right_gripper"], action_configs=[ # left_arm — 6 joint angles ActionConfig( rep=ActionRepresentation.ABSOLUTE, type=ActionType.NON_EEF, format=ActionFormat.DEFAULT, ), # left_gripper — finger displacement (meters) ActionConfig( rep=ActionRepresentation.ABSOLUTE, type=ActionType.NON_EEF, format=ActionFormat.DEFAULT, ), # right_arm ActionConfig( rep=ActionRepresentation.ABSOLUTE, type=ActionType.NON_EEF, format=ActionFormat.DEFAULT, ), # right_gripper ActionConfig( rep=ActionRepresentation.ABSOLUTE, type=ActionType.NON_EEF, format=ActionFormat.DEFAULT, ), ], ), # Language: task description from tasks.jsonl, indexed by task_index in parquet. "language": ModalityConfig( delta_indices=[0], modality_keys=["annotation.human.task_description"], ), } register_modality_config(trossen_ai_config_h32, embodiment_tag=EmbodimentTag.NEW_EMBODIMENT)

Modality file: It is necessary to add a modality.json file inside the meta directory of each lerobot dataset, as in our transfer-cube dataset. Here is our modality.json file:

{ "state": { "left_arm": { "start": 0, "end": 6 }, "left_gripper": { "start": 6, "end": 7 }, "right_arm": { "start": 7, "end": 13 }, "right_gripper": { "start": 13, "end": 14 } }, "action": { "left_arm": { "start": 0, "end": 6 }, "left_gripper": { "start": 6, "end": 7 }, "right_arm": { "start": 7, "end": 13 }, "right_gripper": { "start": 13, "end": 14 } }, "video": { "cam_high": { "original_key": "observation.images.cam_high" }, "cam_low": { "original_key": "observation.images.cam_low" }, "cam_left_wrist": { "original_key": "observation.images.cam_left_wrist" }, "cam_right_wrist": { "original_key": "observation.images.cam_right_wrist" } }, "annotation": { "human.task_description": { "original_key": "task_index" } } }

Dataset download: The GR00T training script expects a local filesystem path rather than a HuggingFace repo ID for --dataset-path. Therefore, we used huggingface-cli download to move our dataset to demo_data/ before training.

Training: Here is the command line we used for training our transfer-cube dataset with action chunk size 32:

uv run --no-sync python gr00t/experiment/launch_finetune.py \ --base-model-path nvidia/GR00T-N1.7-3B \ --dataset-path demo_data/trossen_transfer_40mm_cube_02 \ --embodiment-tag NEW_EMBODIMENT \ --modality-config-path examples/trossen_ai/trossen_ai_config_h32.py \ --num-gpus 1 \ --output-dir ./outputs/trossen_full_h32 \ --max-steps 30000 \ --save-steps 10000 \ --save-total-limit 10 \ --global-batch-size 32 \ --learning-rate 1e-4 \ --episode-sampling-rate 1.0 \ --dataloader-num-workers 4 \

Running the policy: To serve the policy:

uv run --no-sync python gr00t/eval/run_gr00t_server.py \ --model-path outputs/trossen_full_h32 \ --embodiment-tag NEW_EMBODIMENT \ --device cuda:0 \ --host 127.0.0.1 \ --port 5555

Running the robot: The following script uses the learned model to control the robot. It sends robot and camera states to the policy server which sends back the next action to take:

python examples/trossen_ai/main_gr00t_trossen.py \ --mode autonomous \ --task_prompt "transfer the cube" \ --max_relative_target 0.1 \ --open_loop_horizon 32 \ --action_chunk_size 32 \ --max_steps 1000 \ --action_smooth_alpha 0.25 \ --diagnostics

RTC: Although RTC did not smooth out the robot for the transfer-cube example, it might be useful in the future. Here are the RTC options that we believe should work best. We also reduced open_loop_horizon to 24 because the last 8 steps out of 32 are overlap steps which should not be run. Otherwise, those actions would be either repeated or partly repeated by the first 8 steps of the next chunk:

--open_loop_horizon 24 \ --use_rtc \ --rtc_overlap_steps 8 \ --rtc_frozen_steps 2 \ --rtc_ramp_rate 5.0 \