If you are a ROS developer/user and you blog about it, ROS wants those contributions on this page ! All you need for that to happen is:

• have an RSS/Atom blog (no Tweeter/Facebook/Google+ posts)
• open a pull request on planet.ros tracker indicating your name and your RSS feed/ATOM url. (You can just edit the file and click "Propose File Change" to open a pull request.)
• make your ROS related posts tagged with any of the following categories: "ROS", "R.O.S.", "ros", "r.o.s."

For security reasons, html iframe, embed, object, javascript will be stripped out. Only Youtube videos in object and embed will be kept.

Planet ROS is one of the public faces of ROS and is read by users and potential contributors. The content remains the opinion of the bloggers but Planet ROS reserves the right to remove offensive posts.

Blogs should be related to ROS but that does not mean they should be devoid of personal subjects and opinions : those are encouraged since Planet ROS is a chance to know more about ROS developers.

Posts can be positive and promote ROS, or constructive and describe issues but should not contain useless flaming opinions. We want to keep ROS welcoming :)

ROS covers a wide variety of people and cultures. Profanities, prejudice, lewd comments and content likely to offend are to be avoided. Do not make personal attacks or attacks against other projects on your blog.

If you find any bug or have any suggestion, please file a bug on the planet.ros tracker.

Problem:
Integrating unknown hardware into the companion computer (a custom 1D LiDAR for altitude hold, telemetry radio from decades ago, and a mysterious motor controller) involves lots of frustration trying to find the baud rate manually. The autobaud solutions that I have come across rely on human-readable ASCII and are therefore of no use in determining the baud rate of binary streams.

Solution (autobaud-core):
I’m developing a cross-platform Rust-based autobaud detector as a library and CLI daemon that can be called programmatically to detect baud rates without user interaction.

As opposed to cycling through possible rates blindfolded, my solution involves a two-tier strategy based on:

• Hardware Framing Validation: For Linux systems, I use the nix crate to access termios PARMRK flags. In case the baud rate is incorrect, the software detects it almost immediately at the hardware framing level (the electrical framing is faulty), and it does not waste CPU cycles running the loop.
• Zero-Allocation Matching Heuristics: With the electrical framing established, it proceeds to examine the raw &[u8] byte buffer for protocol-specific “magic” bytes (MAVLink v2 = 0xFD, TFmini LiDAR = 0x59 0x59, NMEA = $GPGGA).

I have used ROS2 myself but I am a beginner but could come up with this idea from my experience with working with drones and sensor fusion

Questions from me:

• Validation: Would such a CLI utility be a real-world thing used when adding new physical devices?
• ROS 2 Integration: I would like to see some integration beyond a command-line utility here. Could we wrap this into a ROS 2 node, e.g., “Dynamic Serial Bridge” that autobauds the connection, determines the protocol, and immediately starts publishing the raw binary data on a certain topic for parsing further?
• Protocols missing in action: Besides the aforementioned MAVLink, NMEA, ASCII protocols, what other raw serial protocols are common when interfacing between hardware and ROS?

2 posts - 2 participants

Read full topic

Hello, robotics developers!

If you’re using cloud robotics at all, we (the Cloud Robotics Working Group) would appreciate you taking the time to fill out a survey.

We’re gathering data on how cloud robotics is used, what it’s doing well, and what still needs improving.

The survey link is here: https://forms.gle/WxXiRcCkRxxJLUop9

Responses will be published anonymously. Thank you!

1 post - 1 participant

Read full topic

Isaac Lab Teleoperation and Data Collection Workflow | Piper Robotic Arm for Embodied AI

0.Preface

This project provides an Isaac Lab-based teleoperation and demonstration collection workflow for the AgileX Piper robotic arm. It is designed for developers working on Embodied AI, robot learning, and imitation learning pipelines using simulation environments.

The workflow supports:

• keyboard-based robot teleoperation
• cube stacking manipulation tasks
• trajectory recording and replay
• imitation learning dataset generation
• ROS2-compatible robotic development
• Isaac Lab manipulation environments

Project Has Already Completed

• employment of the Piper robotic arm inside NVIDIA Isaac Lab
• keyboard teleoperation for robot manipulation
• cube stacking task environment configuration
• URDF-to-USD robot asset conversion
• gripper and actuator parameter configuration
• observation pipeline customization for Piper grippers
• human demonstration data collection workflow
• Robomimic-compatible imitation learning configuration
• replay support for collected robot trajectories

Recommended Environment

• Operating System: Ubuntu 22.04 or compatible Linux distribution
• Simulation Framework: NVIDIA Isaac Lab
• ROS Version: ROS 2 Humble
• Python Version: Python 3.10+

Reference & Resource

• Official IsaacLab Tutorial: Welcome to Isaac Lab! — Isaac Lab Documentation

• IsaacLab: GitHub - isaac-sim/IsaacLab: Unified framework for robot learning built on NVIDIA Isaac Sim · GitHub)

• Project Repository: GitHub - szyzp/IsaacLab_Data_Collection · GitHub

• Piper Robotic Arm Platform

The following section provides how developers can rapidly build a complete robotic manipulation and data collection workflow, steps are shown below:

Pipeline Overview
Keyboard Input
↓
Robot Teleoperation
↓
Isaac Lab Simulation
↓
Trajectory Recording
↓
Dataset Generation
↓
Imitation Learning / RL Training

Step 1.Create an Isaac Lab External Project

Before building the teleoperation and data collection workflow, first create an external Isaac Lab project environment.

1.1 Create a New Isaac Lab External Project

1.1.1 Initialize the Project

Run the following command inside the Isaac Lab root directory.

Make sure Isaac Lab has already been installed and the isaaclab Conda environment is activated.

conda activate isaaclab
./isaaclab.sh --new

1.1.2 Recommended Project Configuration

Configuration are shown above

The initial project structure of agx_teleop is as follows:

├── pyproject.toml
├── README.md
├── scripts
│   ├── list_envs.py
│   ├── random_agent.py
│   ├── sb3
│   │   ├── play.py
│   │   └── train.py
│   └── zero_agent.py
└── source
    └── agx_teleop
        ├── agx_teleop
        │   ├── __init__.py
        │   ├── tasks
        │   │   ├── __init__.py
        │   │   └── manager_based
        │   │       ├── agx_teleop
        │   │       │   ├── agents
        │   │       │   │   ├── __init__.py
        │   │       │   │   └── sb3_ppo_cfg.yaml
        │   │       │   ├── agx_teleop_env_cfg.py
        │   │       │   ├── __init__.py
        │   │       │   └── mdp
        │   │       │       ├── __init__.py
        │   │       │       └── rewards.py
        │   │       └── __init__.py
        │   └── ui_extension_example.py
        ├── agx_teleop.egg-info
        │   ├── dependency_links.txt
        │   ├── not-zip-safe
        │   ├── PKG-INFO
        │   ├── requires.txt
        │   ├── SOURCES.txt
        │   └── top_level.txt
        ├── config
        │   └── extension.toml
        ├── docs
        │   └── CHANGELOG.rst
        ├── pyproject.toml
        └── setup.py

1.1.3 Install External Projects

Running the following commands:

cd ../agx_teleop
python -m pip install -e source/agx_teleop

1.2 Configure the PiPER USD Asset

1.2.1 Prepare the URDF

Clone the official AgileX PiPER URDF repository:

cd agx_teleop
mkdir -p source/agx_teleop/agx_description
cd source/agx_teleop/agx_description
git clone https://github.com/agilexrobotics/agx_arm_urdf.git

Then merge:

• piper_with_gripper_description.xacro
• piper_description.urdf
  into a unified:
• piper_gripper.urdf

Complete PiPER URDF file content:

<?xml version="1.0" encoding="utf-8"?>
<!-- This URDF was automatically created by SolidWorks to URDF Exporter! Originally created by Stephen Brawner (brawner@gmail.com) 
     Commit Version: 1.6.0-4-g7f85cfe  Build Version: 1.6.7995.38578
     For more information, please see http://wiki.ros.org/sw_urdf_exporter -->
<robot name="piper">
    <link name="world"/>
    <joint name="world_to_base_link" type="fixed">
        <parent link="world"/>
        <child link="base_link"/>
        <origin xyz="0 0 0" rpy="0 0 0"/>
    </joint>
    <link name="base_link">
        <inertial>
            <origin xyz="-0.00473641164191482 2.56829134630247E-05 0.041451518036016" rpy="0 0 0"/>
            <mass value="1.02" />
            <inertia ixx="0.00267433" ixy="-0.00000073" ixz="-0.00017389" iyy="0.00282612" iyz="0.0000004" izz="0.00089624"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/base_link.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/base_link.stl"/>
            </geometry>
        </collision>
    </link>
    <link name="link1">
        <inertial>
            <origin xyz="0.00032 -0.00041 -0.00348" rpy="0 0 0"/>
            <mass value="0.71"/>
            <inertia ixx="0.00050027" ixy="0.00000078" ixz="0.00000626" iyy="0.00040879" iyz="0.00000462" izz="0.00045802"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link1.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link1.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint1" type="revolute">
        <origin xyz="0 0 0.123" rpy="0 0 0"/>
        <parent link="base_link"/>
        <child link="link1"/>
        <axis xyz="0 0 1"/>
        <limit lower="-2.6179938" upper="2.6179938" effort="100" velocity="5"/>
    </joint>
    <link name="link2">
        <inertial>
            <origin xyz="0.21852 -0.00861 0.00126" rpy="0 0 0"/>
            <mass value="1.16"/>
            <inertia ixx="0.00111915" ixy="0.00182916" ixz="0.00019233" iyy="0.06763251" iyz="0.00000767" izz="0.067559"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link2.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link2.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint2" type="revolute">
        <origin xyz="0 0 0" rpy="1.5707963 -0.1357866 -3.1415926"/>
        <parent link="link1"/>
        <child link="link2"/>
        <axis xyz="0 0 1"/>
        <limit lower="0" upper="3.1415926" effort="100" velocity="5"/>
    </joint>
    <link name="link3">
        <inertial>
            <origin xyz="-0.01999 -0.14808 -0.00053" rpy="0 0 0"/>
            <mass value="0.5"/>
            <inertia ixx="0.01347608" ixy="0.00161034" ixz="0.00000066" iyy="0.00046377" iyz="0.00000162" izz="0.01363321"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link3.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link3.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint3" type="revolute">
        <origin xyz="0.28503 0 0" rpy="0 0 -1.7938494"/>
        <parent link="link2"/>
        <child link="link3"/>
        <axis xyz="0 0 1"/>
        <limit lower="-2.9670597" upper="0" effort="100" velocity="5"/>
    </joint>
    <link name="link4">
        <inertial>
            <origin xyz="0.00013 -0.00076 -0.00394" rpy="0 0 0"/>
            <mass value="0.38"/>
            <inertia ixx="0.00017844" ixy="0.00000062" ixz="0.00000067" iyy="0.00018914" iyz="0.00000408" izz="0.00014613"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link4.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link4.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint4" type="revolute">
        <origin xyz="-0.02198 -0.25075 0" rpy="1.5707963 0 0"/>
        <parent link="link3"/>
        <child link="link4"/>
        <axis xyz="0 0 1"/>
        <limit lower="-1.7453292" upper="1.7453292" effort="100" velocity="5"/>
    </joint>
    <link name="link5">
        <inertial>
            <origin xyz="0.00018 -0.06104 -0.00227" rpy="0 0 0"/>
            <mass value="0.383"/>
            <inertia ixx="0.00172318" ixy="0.00000259" ixz="0.00000010" iyy="0.00017936" iyz="0.00000990" izz="0.00171172"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link5.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link5.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint5" type="revolute">
        <origin xyz="0 0 0" rpy="-1.5707963 0 0"/>
        <parent link="link4"/>
        <child link="link5"/>
        <axis xyz="0 0 1"/>
        <limit lower="-1.2217304" upper="1.2217304" effort="100" velocity="5"/>
    </joint>
    <link name="link6">
        <inertial>
            <origin xyz="-0.00003 0.00006 -0.002" rpy="0 0 0"/>
            <mass value="0.007"/>
            <inertia ixx="0.00152112" ixy="0.00000202" ixz="0.00000261" iyy="0.00138626" iyz="0.00000002" izz="0.00031152"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link6.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link6.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint6" type="revolute">
        <origin xyz="8.8259E-05 -0.091 0" rpy="1.5707963 0 0"/>
        <parent link="link5"/>
        <child link="link6"/>
        <axis xyz="0 0 1"/>
        <limit lower="-2.0943951" upper="2.0943951" effort="100" velocity="5"/>
    </joint>

    <!-- Gripper components -->
    <link name="gripper_base">
        <inertial>
            <origin xyz="-0.000183807162235591 8.05033155577911E-05 0.0321436689908876" rpy="0 0 0"/>
            <mass value="0.45"/>
            <inertia ixx="0.00092934" ixy="0.00000034" ixz="-0.00000738" iyy="0.00071447" iyz="0.00000005" izz="0.00039442"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/gripper_base.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/gripper_base.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="gripper_base_joint" type="fixed">
        <origin xyz="0 0 0" rpy="0 0 0"/>
        <parent link="link6"/>
        <child link="gripper_base"/>
    </joint>
    <link name="gripper_link1">
        <inertial>
            <origin xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025" rpy="0 0 0"/>
            <mass value="0.025"/>
            <inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/gripper_link1.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/gripper_link1.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="gripper_joint1" type="prismatic">
        <origin xyz="0 0 0.1358" rpy="1.5707963 0 0"/>
        <parent link="gripper_base"/>
        <child link="gripper_link1"/>
        <axis xyz="0 0 1"/>
        <limit lower="0" upper="0.05" effort="10" velocity="3"/>
    </joint>
    <link name="gripper_link2">
        <inertial>
            <origin xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025" rpy="0 0 0"/>
            <mass value="0.025"/>
            <inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/gripper_link2.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/gripper_link2.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="gripper_joint2" type="prismatic">
        <origin xyz="0 0 0.1358" rpy="1.5707963 0 -3.1415926"/>
        <parent link="gripper_base"/>
        <child link="gripper_link2"/>
        <axis xyz="0 0 -1"/>
        <limit lower="-0.05" upper="0" effort="10" velocity="3"/>
    </joint>
</robot>

1.2.2 Convert URDF to USD

Isaac Lab uses USD assets for simulation.

Convert the Piper URDF into a USD robot asset:

python ../IsaacLab/scripts/tools/convert_urdf.py \
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/piper/urdf/piper_gripper.urdf \
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/piper/usd/piper.usd \
--fix-base 

Step 2.Configure the PiPER Robot in Isaac Lab

2.1 Create the Robot Asset Configuration

cd agx_teleop
mkdir -p source/agx_teleop/agx_teleop/assets

2.1.1 Create the Asset Package

Create the assets/__init__.py file:

"""Package containing robot configurations."""
import os
import toml

##
# Configuration for different assets.
##

# Conveniences to other module directories via relative paths
AGX_TELEOP_EXT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../"))
"""Path to the source agx_teleop directory."""

AGX_TELEOP_DESCRIPTION_DIR = os.path.join(AGX_TELEOP_EXT_DIR, "agx_description")
"""Path to the agx_description directory."""

# Robot description assets
AGX_TELEOP_DESCRIPTION_DIR = os.path.join(
    AGX_TELEOP_EXT_DIR,
    "agx_description"
)
"""Path to the agx_description directory."""

2.1.2 Configure the PiPER Robot Asset

Create the assets/piper.py file

import isaaclab.sim as sim_utils
from isaaclab.actuators import ImplicitActuatorCfg
from isaaclab.assets.articulation import ArticulationCfg

from agx_teleop.assets import AGX_TELEOP_DESCRIPTION_DIR


PIPER_GRIPPER_CFG = ArticulationCfg(

    # USD robot asset
    spawn=sim_utils.UsdFileCfg(
        usd_path=f"{AGX_TELEOP_DESCRIPTION_DIR}/agx_arm_urdf/piper/usd/piper.usd",

        # Rigid body physics settings
        rigid_props=sim_utils.RigidBodyPropertiesCfg(
            disable_gravity=False,
            max_depenetration_velocity=5.0,
        ),

        # Articulation solver settings
        articulation_props=sim_utils.ArticulationRootPropertiesCfg(
            enabled_self_collisions=True,
            solver_position_iteration_count=8,
            solver_velocity_iteration_count=0,
        ),
    ),

    # Initial robot pose
    init_state=ArticulationCfg.InitialStateCfg(
        joint_pos={
            "joint1": 0.0,
            "joint2": 1.57,
            "joint3": -1.57,
            "joint4": 0.0,
            "joint5": 1.22,
            "joint6": 0.0,
            "gripper_joint1": 0.0,
            "gripper_joint2": 0.0,
        },
        pos=(0.0, 0.0, 0.0),
    ),

    # Actuator configuration
    actuators={

        # Shoulder joints
        "piper_shoulder": ImplicitActuatorCfg(
            joint_names_expr=["joint[1-3]"],
            effort_limit_sim=100.0,
            velocity_limit_sim=5.0,
            stiffness=80.0,
            damping=20.0,
        ),

        # Forearm joints
        "piper_forearm": ImplicitActuatorCfg(
            joint_names_expr=["joint[4-6]"],
            effort_limit_sim=100.0,
            velocity_limit_sim=5.0,
            stiffness=40.0,
            damping=10.0,
        ),

        # Parallel gripper
        "gripper": ImplicitActuatorCfg(
            joint_names_expr=[
                "gripper_joint1",
                "gripper_joint2"
            ],
            effort_limit_sim=22,
            velocity_limit_sim=1.5,
            stiffness=800.0,
            damping=20.0,
        ),
    },

    # Soft joint limit scaling
    soft_joint_pos_limit_factor=0.9,
)

"""Configuration for the Piper robot with parallel gripper."""

PIPER_GRIPPER_HIGH_PD_CFG = PIPER_GRIPPER_CFG.copy()
PIPER_GRIPPER_HIGH_PD_CFG.spawn.rigid_props.disable_gravity = True
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_shoulder"].stiffness = 400.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_shoulder"].damping = 80.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_forearm"].stiffness = 200.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_forearm"].damping = 40.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["gripper"].velocity_limit_sim = 0.5
"""Configuration of Piper with gripper robot with stiffer PD control.

This configuration is useful for task-space control using differential IK.
"""

Step 3.Build the Manipulation Environment

3.1 Project Structure Setup

The following structure is modified from the official IsaacLab stack task isaaclab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack. We will create a specialized directory for the PiPER robot under the agx_teleop project.

Create Directories

cd agx_teleop
mkdir -p source/agx_teleop/agx_teleop/tasks/manager_based/manipulation/stack/config/piper/agents/robomimic
mkdir -p source/agx_teleop/agx_teleop/tasks/manager_based/manipulation/stack/mdp

Initialize Files

cd source/agx_teleop/agx_teleop/tasks/manager_based/

touch manipulation/__init__.py \
      manipulation/stack/__init__.py \
      manipulation/stack/stack_env_cfg.py \
      manipulation/stack/mdp/__init__.py \
      manipulation/stack/mdp/observations.py \
      manipulation/stack/mdp/piper_stack_events.py \
      manipulation/stack/mdp/terminations.py \
      manipulation/stack/config/__init__.py \
      manipulation/stack/config/piper/__init__.py \
      manipulation/stack/config/piper/agents/__init__.py \
      manipulation/stack/config/piper/agents/robomimic/bc_rnn_low_dim.json \
      manipulation/stack/config/piper/piper_stack_ik_rel_env_cfg.py \
      manipulation/stack/config/piper/piper_stack_joint_pos_env_cfg.py

3.2 Core File Modifications

3.2.1 Standard IsaacLab Porting

The following files are ported directly from the IsaacLab Franka stack example without significant modification. They define the base MDP (Markov Decision Process) settings and the Robomimic imitation learning configuration.

• stack/stack_env_cfg.py: Defines the scene (table, lights, ground) and observation groups.
• stack/mdp/init.py: Handles module exports.
• stack/config/piper/agents/robomimic/bc_rnn_low_dim.json: Standard BC-RNN configuration for low-dimensional observations.

Copy the corresponding code from IsaacLab without modification :

/Isaac/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/stack_env_cfg.py
/Isaac/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/__init__.py
/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/config/franka/agents/robomimic/bc_rnn_low_dim.json

3.2.2 Event Customization (mdp/piper_stack_events.py)

Copy the corresponding code from Issaclab directly, without any modifying :

Isaac/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/mdp/franka_stack_events.py

3.2.3 Adapting Observations & Terminations for Piper

Copy the corresponding code of isaaclab and modify the part related to gripper_open_val. The Franka panda gripper control uses a single joint value, while the piper gripper control uses two joint values, which need to be adjusted to match the piper gripper.

Key Logic Adjustment (Lines 320-343 & 380-407):

stack/mdp/observations.py

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

from __future__ import annotations

from typing import TYPE_CHECKING, Literal

import torch

import isaaclab.utils.math as math_utils
from isaaclab.assets import Articulation, RigidObject, RigidObjectCollection
from isaaclab.managers import SceneEntityCfg
from isaaclab.sensors import FrameTransformer

if TYPE_CHECKING:
    from isaaclab.envs import ManagerBasedRLEnv

def cube_positions_in_world_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
) -> torch.Tensor:
    """The position of the cubes in the world frame."""
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]

    return torch.cat((cube_1.data.root_pos_w, cube_2.data.root_pos_w, cube_3.data.root_pos_w), dim=1)

def instance_randomize_cube_positions_in_world_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
) -> torch.Tensor:
    """The position of the cubes in the world frame."""
    if not hasattr(env, "rigid_objects_in_focus"):
        return torch.full((env.num_envs, 9), fill_value=-1)

    cube_1: RigidObjectCollection = env.scene[cube_1_cfg.name]
    cube_2: RigidObjectCollection = env.scene[cube_2_cfg.name]
    cube_3: RigidObjectCollection = env.scene[cube_3_cfg.name]

    cube_1_pos_w = []
    cube_2_pos_w = []
    cube_3_pos_w = []
    for env_id in range(env.num_envs):
        cube_1_pos_w.append(cube_1.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][0], :3])
        cube_2_pos_w.append(cube_2.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][1], :3])
        cube_3_pos_w.append(cube_3.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][2], :3])
    cube_1_pos_w = torch.stack(cube_1_pos_w)
    cube_2_pos_w = torch.stack(cube_2_pos_w)
    cube_3_pos_w = torch.stack(cube_3_pos_w)

    return torch.cat((cube_1_pos_w, cube_2_pos_w, cube_3_pos_w), dim=1)

def cube_orientations_in_world_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
):
    """The orientation of the cubes in the world frame."""
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]

    return torch.cat((cube_1.data.root_quat_w, cube_2.data.root_quat_w, cube_3.data.root_quat_w), dim=1)

def instance_randomize_cube_orientations_in_world_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
) -> torch.Tensor:
    """The orientation of the cubes in the world frame."""
    if not hasattr(env, "rigid_objects_in_focus"):
        return torch.full((env.num_envs, 9), fill_value=-1)

    cube_1: RigidObjectCollection = env.scene[cube_1_cfg.name]
    cube_2: RigidObjectCollection = env.scene[cube_2_cfg.name]
    cube_3: RigidObjectCollection = env.scene[cube_3_cfg.name]

    cube_1_quat_w = []
    cube_2_quat_w = []
    cube_3_quat_w = []
    for env_id in range(env.num_envs):
        cube_1_quat_w.append(cube_1.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][0], :4])
        cube_2_quat_w.append(cube_2.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][1], :4])
        cube_3_quat_w.append(cube_3.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][2], :4])
    cube_1_quat_w = torch.stack(cube_1_quat_w)
    cube_2_quat_w = torch.stack(cube_2_quat_w)
    cube_3_quat_w = torch.stack(cube_3_quat_w)

    return torch.cat((cube_1_quat_w, cube_2_quat_w, cube_3_quat_w), dim=1)

def object_obs(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
    ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
):
    """
    Object observations (in world frame):
        cube_1 pos,
        cube_1 quat,
        cube_2 pos,
        cube_2 quat,
        cube_3 pos,
        cube_3 quat,
        gripper to cube_1,
        gripper to cube_2,
        gripper to cube_3,
        cube_1 to cube_2,
        cube_2 to cube_3,
        cube_1 to cube_3,
    """
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]

    cube_1_pos_w = cube_1.data.root_pos_w
    cube_1_quat_w = cube_1.data.root_quat_w

    cube_2_pos_w = cube_2.data.root_pos_w
    cube_2_quat_w = cube_2.data.root_quat_w

    cube_3_pos_w = cube_3.data.root_pos_w
    cube_3_quat_w = cube_3.data.root_quat_w

    ee_pos_w = ee_frame.data.target_pos_w[:, 0, :]
    gripper_to_cube_1 = cube_1_pos_w - ee_pos_w
    gripper_to_cube_2 = cube_2_pos_w - ee_pos_w
    gripper_to_cube_3 = cube_3_pos_w - ee_pos_w

    cube_1_to_2 = cube_1_pos_w - cube_2_pos_w
    cube_2_to_3 = cube_2_pos_w - cube_3_pos_w
    cube_1_to_3 = cube_1_pos_w - cube_3_pos_w

    return torch.cat(
        (
            cube_1_pos_w - env.scene.env_origins,
            cube_1_quat_w,
            cube_2_pos_w - env.scene.env_origins,
            cube_2_quat_w,
            cube_3_pos_w - env.scene.env_origins,
            cube_3_quat_w,
            gripper_to_cube_1,
            gripper_to_cube_2,
            gripper_to_cube_3,
            cube_1_to_2,
            cube_2_to_3,
            cube_1_to_3,
        ),
        dim=1,
    )

def instance_randomize_object_obs(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
    ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
):
    """
    Object observations (in world frame):
        cube_1 pos,
        cube_1 quat,
        cube_2 pos,
        cube_2 quat,
        cube_3 pos,
        cube_3 quat,
        gripper to cube_1,
        gripper to cube_2,
        gripper to cube_3,
        cube_1 to cube_2,
        cube_2 to cube_3,
        cube_1 to cube_3,
    """
    if not hasattr(env, "rigid_objects_in_focus"):
        return torch.full((env.num_envs, 9), fill_value=-1)

    cube_1: RigidObjectCollection = env.scene[cube_1_cfg.name]
    cube_2: RigidObjectCollection = env.scene[cube_2_cfg.name]
    cube_3: RigidObjectCollection = env.scene[cube_3_cfg.name]
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]

    cube_1_pos_w = []
    cube_2_pos_w = []
    cube_3_pos_w = []
    cube_1_quat_w = []
    cube_2_quat_w = []
    cube_3_quat_w = []
    for env_id in range(env.num_envs):
        cube_1_pos_w.append(cube_1.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][0], :3])
        cube_2_pos_w.append(cube_2.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][1], :3])
        cube_3_pos_w.append(cube_3.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][2], :3])
        cube_1_quat_w.append(cube_1.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][0], :4])
        cube_2_quat_w.append(cube_2.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][1], :4])
        cube_3_quat_w.append(cube_3.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][2], :4])
    cube_1_pos_w = torch.stack(cube_1_pos_w)
    cube_2_pos_w = torch.stack(cube_2_pos_w)
    cube_3_pos_w = torch.stack(cube_3_pos_w)
    cube_1_quat_w = torch.stack(cube_1_quat_w)
    cube_2_quat_w = torch.stack(cube_2_quat_w)
    cube_3_quat_w = torch.stack(cube_3_quat_w)

    ee_pos_w = ee_frame.data.target_pos_w[:, 0, :]
    gripper_to_cube_1 = cube_1_pos_w - ee_pos_w
    gripper_to_cube_2 = cube_2_pos_w - ee_pos_w
    gripper_to_cube_3 = cube_3_pos_w - ee_pos_w

    cube_1_to_2 = cube_1_pos_w - cube_2_pos_w
    cube_2_to_3 = cube_2_pos_w - cube_3_pos_w
    cube_1_to_3 = cube_1_pos_w - cube_3_pos_w

    return torch.cat(
        (
            cube_1_pos_w - env.scene.env_origins,
            cube_1_quat_w,
            cube_2_pos_w - env.scene.env_origins,
            cube_2_quat_w,
            cube_3_pos_w - env.scene.env_origins,
            cube_3_quat_w,
            gripper_to_cube_1,
            gripper_to_cube_2,
            gripper_to_cube_3,
            cube_1_to_2,
            cube_2_to_3,
            cube_1_to_3,
        ),
        dim=1,
    )

def ee_frame_pos(env: ManagerBasedRLEnv, ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame")) -> torch.Tensor:
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    ee_frame_pos = ee_frame.data.target_pos_w[:, 0, :] - env.scene.env_origins[:, 0:3]

    return ee_frame_pos

def ee_frame_quat(env: ManagerBasedRLEnv, ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame")) -> torch.Tensor:
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    ee_frame_quat = ee_frame.data.target_quat_w[:, 0, :]

    return ee_frame_quat

def gripper_pos(
    env: ManagerBasedRLEnv,
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
) -> torch.Tensor:
    """
    Obtain the versatile gripper position of both Gripper and Suction Cup.
    """
    robot: Articulation = env.scene[robot_cfg.name]

    if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
        # Handle multiple surface grippers by concatenating their states
        gripper_states = []
        for gripper_name, surface_gripper in env.scene.surface_grippers.items():
            gripper_states.append(surface_gripper.state.view(-1, 1))

        if len(gripper_states) == 1:
            return gripper_states[0]
        else:
            return torch.cat(gripper_states, dim=1)

    else:
        if hasattr(env.cfg, "gripper_joint_names"):
            gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
            assert len(gripper_joint_ids) == 2, "Observation gripper_pos only support parallel gripper for now"
            finger_joint_1 = robot.data.joint_pos[:, gripper_joint_ids[0]].clone().unsqueeze(1)
            finger_joint_2 = -1 * robot.data.joint_pos[:, gripper_joint_ids[1]].clone().unsqueeze(1)
            return torch.cat((finger_joint_1, finger_joint_2), dim=1)
        else:
            raise NotImplementedError("[Error] Cannot find gripper_joint_names in the environment config")

def object_grasped(
    env: ManagerBasedRLEnv,
    robot_cfg: SceneEntityCfg,
    ee_frame_cfg: SceneEntityCfg,
    object_cfg: SceneEntityCfg,
    diff_threshold: float = 0.06,
) -> torch.Tensor:
    """Check if an object is grasped by the specified robot."""

    robot: Articulation = env.scene[robot_cfg.name]
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    object: RigidObject = env.scene[object_cfg.name]

    object_pos = object.data.root_pos_w
    end_effector_pos = ee_frame.data.target_pos_w[:, 0, :]
    pose_diff = torch.linalg.vector_norm(object_pos - end_effector_pos, dim=1)

    if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
        surface_gripper = env.scene.surface_grippers["surface_gripper"]
        suction_cup_status = surface_gripper.state.view(-1, 1)  # 1: closed, 0: closing, -1: open
        suction_cup_is_closed = (suction_cup_status == 1).to(torch.float32)
        grasped = torch.logical_and(suction_cup_is_closed, pose_diff < diff_threshold)

    else:
        if hasattr(env.cfg, "gripper_joint_names"):
            gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
            assert len(gripper_joint_ids) == 2, "Observations only support parallel gripper for now"

            # Support per-joint open values (list/tuple) or single scalar (applied to all joints)
            if isinstance(env.cfg.gripper_open_val, (list, tuple)):
                open_val_0 = env.cfg.gripper_open_val[0]
                open_val_1 = env.cfg.gripper_open_val[1]
            else:
                open_val_0 = env.cfg.gripper_open_val
                open_val_1 = env.cfg.gripper_open_val

            grasped = torch.logical_and(
                pose_diff < diff_threshold,
                torch.abs(
                    robot.data.joint_pos[:, gripper_joint_ids[0]]
                    - torch.tensor(open_val_0, dtype=torch.float32).to(env.device)
                )
                > env.cfg.gripper_threshold,
            )
            grasped = torch.logical_and(
                grasped,
                torch.abs(
                    robot.data.joint_pos[:, gripper_joint_ids[1]]
                    - torch.tensor(open_val_1, dtype=torch.float32).to(env.device)
                )
                > env.cfg.gripper_threshold,
            )

    return grasped

def object_stacked(
    env: ManagerBasedRLEnv,
    robot_cfg: SceneEntityCfg,
    upper_object_cfg: SceneEntityCfg,
    lower_object_cfg: SceneEntityCfg,
    xy_threshold: float = 0.05,
    height_threshold: float = 0.005,
    height_diff: float = 0.0468,
) -> torch.Tensor:
    """Check if an object is stacked by the specified robot."""

    robot: Articulation = env.scene[robot_cfg.name]
    upper_object: RigidObject = env.scene[upper_object_cfg.name]
    lower_object: RigidObject = env.scene[lower_object_cfg.name]

    pos_diff = upper_object.data.root_pos_w - lower_object.data.root_pos_w
    height_dist = torch.linalg.vector_norm(pos_diff[:, 2:], dim=1)
    xy_dist = torch.linalg.vector_norm(pos_diff[:, :2], dim=1)

    stacked = torch.logical_and(xy_dist < xy_threshold, (height_dist - height_diff) < height_threshold)

    if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
        surface_gripper = env.scene.surface_grippers["surface_gripper"]
        suction_cup_status = surface_gripper.state.view(-1, 1)  # 1: closed, 0: closing, -1: open
        suction_cup_is_open = (suction_cup_status == -1).to(torch.float32)
        stacked = torch.logical_and(suction_cup_is_open, stacked)

    else:
        if hasattr(env.cfg, "gripper_joint_names"):
            gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
            assert len(gripper_joint_ids) == 2, "Observations only support parallel gripper for now"
            
            # Support per-joint open values (list/tuple) or single scalar (applied to all joints)
            if isinstance(env.cfg.gripper_open_val, (list, tuple)):
                open_val_0 = env.cfg.gripper_open_val[0]
                open_val_1 = env.cfg.gripper_open_val[1]
            else:
                open_val_0 = env.cfg.gripper_open_val
                open_val_1 = env.cfg.gripper_open_val

            stacked = torch.logical_and(
                torch.isclose(
                    robot.data.joint_pos[:, gripper_joint_ids[0]],
                    torch.tensor(open_val_0, dtype=torch.float32).to(env.device),
                    atol=1e-4,
                    rtol=1e-4,
                ),
                stacked,
            )
            stacked = torch.logical_and(
                torch.isclose(
                    robot.data.joint_pos[:, gripper_joint_ids[1]],
                    torch.tensor(open_val_1, dtype=torch.float32).to(env.device),
                    atol=1e-4,
                    rtol=1e-4,
                ),
                stacked,
            )
        else:
            raise ValueError("No gripper_joint_names found in environment config")

    return stacked

def cube_poses_in_base_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
    return_key: Literal["pos", "quat", None] = None,
) -> torch.Tensor:
    """The position and orientation of the cubes in the robot base frame."""

    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]

    pos_cube_1_world = cube_1.data.root_pos_w
    pos_cube_2_world = cube_2.data.root_pos_w
    pos_cube_3_world = cube_3.data.root_pos_w

    quat_cube_1_world = cube_1.data.root_quat_w
    quat_cube_2_world = cube_2.data.root_quat_w
    quat_cube_3_world = cube_3.data.root_quat_w

    robot: Articulation = env.scene[robot_cfg.name]
    root_pos_w = robot.data.root_pos_w
    root_quat_w = robot.data.root_quat_w

    pos_cube_1_base, quat_cube_1_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, pos_cube_1_world, quat_cube_1_world
    )
    pos_cube_2_base, quat_cube_2_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, pos_cube_2_world, quat_cube_2_world
    )
    pos_cube_3_base, quat_cube_3_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, pos_cube_3_world, quat_cube_3_world
    )

    pos_cubes_base = torch.cat((pos_cube_1_base, pos_cube_2_base, pos_cube_3_base), dim=1)
    quat_cubes_base = torch.cat((quat_cube_1_base, quat_cube_2_base, quat_cube_3_base), dim=1)

    if return_key == "pos":
        return pos_cubes_base
    elif return_key == "quat":
        return quat_cubes_base
    else:
        return torch.cat((pos_cubes_base, quat_cubes_base), dim=1)

def object_abs_obs_in_base_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
    ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
):
    """
    Object Abs observations (in base frame): remove the relative observations,
    and add abs gripper pos and quat in robot base frame
        cube_1 pos,
        cube_1 quat,
        cube_2 pos,
        cube_2 quat,
        cube_3 pos,
        cube_3 quat,
        gripper pos,
        gripper quat,
    """
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    robot: Articulation = env.scene[robot_cfg.name]

    root_pos_w = robot.data.root_pos_w
    root_quat_w = robot.data.root_quat_w

    cube_1_pos_w = cube_1.data.root_pos_w
    cube_1_quat_w = cube_1.data.root_quat_w

    cube_2_pos_w = cube_2.data.root_pos_w
    cube_2_quat_w = cube_2.data.root_quat_w

    cube_3_pos_w = cube_3.data.root_pos_w
    cube_3_quat_w = cube_3.data.root_quat_w

    pos_cube_1_base, quat_cube_1_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, cube_1_pos_w, cube_1_quat_w
    )
    pos_cube_2_base, quat_cube_2_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, cube_2_pos_w, cube_2_quat_w
    )
    pos_cube_3_base, quat_cube_3_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, cube_3_pos_w, cube_3_quat_w
    )

    ee_pos_w = ee_frame.data.target_pos_w[:, 0, :]
    ee_quat_w = ee_frame.data.target_quat_w[:, 0, :]
    ee_pos_base, ee_quat_base = math_utils.subtract_frame_transforms(root_pos_w, root_quat_w, ee_pos_w, ee_quat_w)

    return torch.cat(
        (
            pos_cube_1_base,
            quat_cube_1_base,
            pos_cube_2_base,
            quat_cube_2_base,
            pos_cube_3_base,
            quat_cube_3_base,
            ee_pos_base,
            ee_quat_base,
        ),
        dim=1,
    )

def ee_frame_pose_in_base_frame(
    env: ManagerBasedRLEnv,
    ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
    return_key: Literal["pos", "quat", None] = None,
) -> torch.Tensor:
    """
    The end effector pose in the robot base frame.
    """
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    ee_frame_pos_w = ee_frame.data.target_pos_w[:, 0, :]
    ee_frame_quat_w = ee_frame.data.target_quat_w[:, 0, :]

    robot: Articulation = env.scene[robot_cfg.name]
    root_pos_w = robot.data.root_pos_w
    root_quat_w = robot.data.root_quat_w
    ee_pos_in_base, ee_quat_in_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, ee_frame_pos_w, ee_frame_quat_w
    )

    if return_key == "pos":
        return ee_pos_in_base
    elif return_key == "quat":
        return ee_quat_in_base
    else:
        return torch.cat((ee_pos_in_base, ee_quat_in_base), dim=1)

Similarly, the cubes_stacked termination condition is updated to ensure the success criteria check both gripper joints before concluding the task.

Below is the complete code, with relevant modifications in Lines 81-109.
stack/mdp/terminations.py

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

"""Common functions that can be used to activate certain terminations for the lift task.

The functions can be passed to the :class:`isaaclab.managers.TerminationTermCfg` object to enable
the termination introduced by the function.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import torch
from isaaclab.assets import Articulation, RigidObject
from isaaclab.managers import SceneEntityCfg

if TYPE_CHECKING:
    from isaaclab.envs import ManagerBasedRLEnv

def cubes_stacked(
    env: ManagerBasedRLEnv,
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg | None = SceneEntityCfg("cube_3"),
    xy_threshold: float = 0.04,
    height_threshold: float = 0.005,
    height_diff: float = 0.0468,
    atol: float = 0.0001,
    rtol: float = 0.0001,
) -> torch.Tensor:
    robot: Articulation = env.scene[robot_cfg.name]
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]

    pos_diff_c12 = cube_1.data.root_pos_w - cube_2.data.root_pos_w

    # Compute cube position difference in x-y plane
    xy_dist_c12 = torch.norm(pos_diff_c12[:, :2], dim=1)

    # Compute cube height difference
    h_dist_c12 = torch.norm(pos_diff_c12[:, 2:], dim=1)

    # Check cube positions
    stacked = xy_dist_c12 < xy_threshold
    stacked = torch.logical_and(h_dist_c12 - height_diff < height_threshold, stacked)
    stacked = torch.logical_and(pos_diff_c12[:, 2] < 0.0, stacked)

    if cube_3_cfg is not None:
        cube_3: RigidObject = env.scene[cube_3_cfg.name]
        pos_diff_c23 = cube_2.data.root_pos_w - cube_3.data.root_pos_w

        # Compute cube position difference in x-y plane
        xy_dist_c23 = torch.norm(pos_diff_c23[:, :2], dim=1)

        # Compute cube height difference
        h_dist_c23 = torch.norm(pos_diff_c23[:, 2:], dim=1)

        # Check cube positions
        stacked = torch.logical_and(xy_dist_c23 < xy_threshold, stacked)
        stacked = torch.logical_and(h_dist_c23 - height_diff < height_threshold, stacked)
        stacked = torch.logical_and(pos_diff_c23[:, 2] < 0.0, stacked)

    # Check gripper positions
    if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
        surface_gripper = env.scene.surface_grippers["surface_gripper"]
        suction_cup_status = surface_gripper.state.view(-1)  # 1: closed, 0: closing, -1: open
        suction_cup_is_open = (suction_cup_status == -1).to(torch.float32)
        stacked = torch.logical_and(suction_cup_is_open, stacked)

    else:
        if hasattr(env.cfg, "gripper_joint_names"):
            gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
            assert len(gripper_joint_ids) == 2, "Terminations only support parallel gripper for now"

            # Support per-joint open values (list/tuple) or single scalar (applied to all joints)
            if isinstance(env.cfg.gripper_open_val, (list, tuple)):
                open_val_0 = env.cfg.gripper_open_val[0]
                open_val_1 = env.cfg.gripper_open_val[1]
            else:
                open_val_0 = env.cfg.gripper_open_val
                open_val_1 = env.cfg.gripper_open_val

            stacked = torch.logical_and(
                torch.isclose(
                    robot.data.joint_pos[:, gripper_joint_ids[0]],
                    torch.tensor(open_val_0, dtype=torch.float32).to(env.device),
                    atol=atol,
                    rtol=rtol,
                ),
                stacked,
            )
            stacked = torch.logical_and(
                torch.isclose(
                    robot.data.joint_pos[:, gripper_joint_ids[1]],
                    torch.tensor(open_val_1, dtype=torch.float32).to(env.device),
                    atol=atol,
                    rtol=rtol,
                ),
                stacked,
            )
        else:
            raise ValueError("No gripper_joint_names found in environment config")

    return stacked

3.2.4 Piper Joint Position Configuration (piper_stack_joint_pos_env_cfg.py)

This file maps the Piper asset to the environment.

Major changes include:

• Default Pose: Adjusted to [0.0, 1.57, -1.57, 0.0, 1.22, 0.0, 0.0, 0.0] (6 ARM joints + 2 Gripper joints).
• Gripper Command Mapping:
  • Open: gripper_joint1: 0.05, gripper_joint2: -0.05
  • Close: 0.0 for both.
• End-Effector (EE) Offset: Set to [0.0, 0.0, 0.14] to account for the Piper gripper’s length from the base link to the Tool Center Point (TCP).

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

from isaaclab.assets import RigidObjectCfg
from isaaclab.managers import EventTermCfg as EventTerm
from isaaclab.managers import SceneEntityCfg
from isaaclab.sensors import FrameTransformerCfg
from isaaclab.sensors.frame_transformer.frame_transformer_cfg import OffsetCfg
from isaaclab.sim.schemas.schemas_cfg import RigidBodyPropertiesCfg
from isaaclab.sim.spawners.from_files.from_files_cfg import UsdFileCfg
from isaaclab.utils import configclass
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR

from agx_teleop.tasks.manager_based.manipulation.stack import mdp
from agx_teleop.tasks.manager_based.manipulation.stack.mdp import piper_stack_events
from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import StackEnvCfg

##
# Pre-defined configs
##
from isaaclab.markers.config import FRAME_MARKER_CFG  # isort: skip
from agx_teleop.assets.piper import PIPER_GRIPPER_CFG  # isort: skip

@configclass
class EventCfg:
    """Configuration for events."""

    init_piper_arm_pose = EventTerm(
        func=piper_stack_events.set_default_joint_pose,
        mode="reset",
        params={
            "default_pose": [0.0, 1.57, -1.57, 0.0, 1.22, 0.0, 0.0, 0.0],   # 6机械臂+2夹爪，夹爪关闭
        },
    )

    randomize_piper_joint_state = EventTerm(
        func=piper_stack_events.randomize_joint_by_gaussian_offset,
        mode="reset",
        params={
            "mean": 0.0,
            "std": 0.02,
            "asset_cfg": SceneEntityCfg("robot"),
        },
    )

    randomize_cube_positions = EventTerm(
        func=piper_stack_events.randomize_object_pose,
        mode="reset",
        params={
            "pose_range": {"x": (0.3, 0.5), "y": (-0.10, 0.10), "z": (0.0203, 0.0203), "yaw": (-1.0, 1, 0)},
            "min_separation": 0.12,
            "asset_cfgs": [SceneEntityCfg("cube_1"), SceneEntityCfg("cube_2"), SceneEntityCfg("cube_3")],
        },
    )

@configclass
class PiperCubeStackEnvCfg(StackEnvCfg):
    """Configuration for the Piper Cube Stack Environment."""

    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        # Set events
        self.events = EventCfg()

        # Set Piper as robot
        self.scene.robot = PIPER_GRIPPER_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
        self.scene.robot.spawn.semantic_tags = [("class", "robot")]

        # Add semantics to table
        self.scene.table.spawn.semantic_tags = [("class", "table")]

        # Add semantics to ground
        self.scene.plane.semantic_tags = [("class", "ground")]

        # Set actions for the specific robot type (piper)
        self.actions.arm_action = mdp.JointPositionActionCfg(
            asset_name="robot", joint_names=["joint.*"], scale=0.5, use_default_offset=True
        )
        self.actions.gripper_action = mdp.BinaryJointPositionActionCfg(
            asset_name="robot",
            joint_names=["gripper_joint.*"],
            open_command_expr={
                "gripper_joint1": 0.05,
                "gripper_joint2": -0.05,    
            },
            close_command_expr={
                "gripper_joint1": 0.0,
                "gripper_joint2": 0.0,
            },
        )
        # utilities for gripper status check
        self.gripper_joint_names = ["gripper_joint.*"]
        self.gripper_open_val = [0.05, -0.05]
        self.gripper_threshold = 0.005

        # Rigid body properties of each cube
        cube_properties = RigidBodyPropertiesCfg(
            solver_position_iteration_count=16,
            solver_velocity_iteration_count=1,
            max_angular_velocity=1000.0,
            max_linear_velocity=1000.0,
            max_depenetration_velocity=5.0,
            disable_gravity=False,
        )

        # Set each stacking cube deterministically
        self.scene.cube_1 = RigidObjectCfg(
            prim_path="{ENV_REGEX_NS}/Cube_1",
            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.4, 0.0, 0.0203], rot=[1, 0, 0, 0]),
            spawn=UsdFileCfg(
                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/blue_block.usd",
                scale=(1.0, 1.0, 1.0),
                rigid_props=cube_properties,
                semantic_tags=[("class", "cube_1")],
            ),
        )
        self.scene.cube_2 = RigidObjectCfg(
            prim_path="{ENV_REGEX_NS}/Cube_2",
            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.55, 0.05, 0.0203], rot=[1, 0, 0, 0]),
            spawn=UsdFileCfg(
                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/red_block.usd",
                scale=(1.0, 1.0, 1.0),
                rigid_props=cube_properties,
                semantic_tags=[("class", "cube_2")],
            ),
        )
        self.scene.cube_3 = RigidObjectCfg(
            prim_path="{ENV_REGEX_NS}/Cube_3",
            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.60, -0.1, 0.0203], rot=[1, 0, 0, 0]),
            spawn=UsdFileCfg(
                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/green_block.usd",
                scale=(1.0, 1.0, 1.0),
                rigid_props=cube_properties,
                semantic_tags=[("class", "cube_3")],
            ),
        )

        # Listens to the required transforms
        marker_cfg = FRAME_MARKER_CFG.copy()
        marker_cfg.markers["frame"].scale = (0.1, 0.1, 0.1)
        marker_cfg.prim_path = "/Visuals/FrameTransformer"
        self.scene.ee_frame = FrameTransformerCfg(
            prim_path="{ENV_REGEX_NS}/Robot/base_link",
            debug_vis=False,
            visualizer_cfg=marker_cfg,
            target_frames=[
                FrameTransformerCfg.FrameCfg(
                    prim_path="{ENV_REGEX_NS}/Robot/gripper_base",
                    name="end_effector",
                    offset=OffsetCfg(
                        pos=[0.0, 0.0, 0.14],   # gripper_base 到 TCP 的距离
                    ),
                ),
                FrameTransformerCfg.FrameCfg(
                    prim_path="{ENV_REGEX_NS}/Robot/gripper_link1",
                    name="tool_rightfinger",
                    offset=OffsetCfg(
                        pos=(0.0, 0.0, 0.0),  
                    ),
                ),
                FrameTransformerCfg.FrameCfg(
                    prim_path="{ENV_REGEX_NS}/Robot/gripper_link2",
                    name="tool_leftfinger",
                    offset=OffsetCfg(
                        pos=(0.0, 0.0, 0.0), 
                    ),
                ),
            ],
        )

3.2.5 Inverse Kinematics Configuration (piper_stack_ik_rel_env_cfg.py)

This file configures the Differential IK controller. We use a high-gain PD controller (PIPER_GRIPPER_HIGH_PD_CFG) to ensure precise tracking during teleoperation or inference.

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

from isaaclab.controllers.differential_ik_cfg import DifferentialIKControllerCfg
from isaaclab.devices.device_base import DeviceBase, DevicesCfg
from isaaclab.devices.keyboard import Se3KeyboardCfg
from isaaclab.devices.openxr.openxr_device import OpenXRDeviceCfg
from isaaclab.devices.openxr.retargeters.manipulator.gripper_retargeter import GripperRetargeterCfg
from isaaclab.devices.openxr.retargeters.manipulator.se3_rel_retargeter import Se3RelRetargeterCfg
from isaaclab.envs.mdp.actions.actions_cfg import DifferentialInverseKinematicsActionCfg
from isaaclab.managers import SceneEntityCfg
from isaaclab.managers import TerminationTermCfg as DoneTerm
from isaaclab.utils import configclass

from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import mdp

from . import piper_stack_joint_pos_env_cfg

##
# Pre-defined configs
##
from agx_teleop.assets.piper import PIPER_GRIPPER_HIGH_PD_CFG  # isort: skip

@configclass
class PiperCubeStackEnvCfg(piper_stack_joint_pos_env_cfg.PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        # Set Piper as robot
        # We switch here to a stiffer PD controller for IK tracking to be better.
        self.scene.robot = PIPER_GRIPPER_HIGH_PD_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")

        # Set actions for the specific robot type (piper)
        self.actions.arm_action = DifferentialInverseKinematicsActionCfg(
            asset_name="robot",
            joint_names=["joint.*"],
            body_name="gripper_base",
            controller=DifferentialIKControllerCfg(command_type="pose", use_relative_mode=True, ik_method="dls"),
            scale=0.5,
            body_offset=DifferentialInverseKinematicsActionCfg.OffsetCfg(pos=[0.0, 0.0, 0.14]),
        )

        self.teleop_devices = DevicesCfg(
            devices={
                "handtracking": OpenXRDeviceCfg(
                    retargeters=[
                        Se3RelRetargeterCfg(
                            bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT,
                            zero_out_xy_rotation=True,
                            use_wrist_rotation=False,
                            use_wrist_position=True,
                            delta_pos_scale_factor=10.0,
                            delta_rot_scale_factor=10.0,
                            sim_device=self.sim.device,
                        ),
                        GripperRetargeterCfg(
                            bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT, sim_device=self.sim.device
                        ),
                    ],
                    sim_device=self.sim.device,
                    xr_cfg=self.xr,
                ),
                "keyboard": Se3KeyboardCfg(
                    pos_sensitivity=0.05,
                    rot_sensitivity=0.05,
                    sim_device=self.sim.device,
                ),
            }
        )

@configclass
class PiperCubeStackRedGreenEnvCfg(PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        self.terminations.success = DoneTerm(
            func=mdp.cubes_stacked,
            params={"cube_1_cfg": SceneEntityCfg("cube_2"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
        )

@configclass
class PiperCubeStackRedGreenBlueEnvCfg(PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        self.terminations.success = DoneTerm(
            func=mdp.cubes_stacked,
            params={
                "cube_1_cfg": SceneEntityCfg("cube_2"),
                "cube_2_cfg": SceneEntityCfg("cube_3"),
                "cube_3_cfg": SceneEntityCfg("cube_1"),
            },
        )

@configclass
class PiperCubeStackBlueGreenEnvCfg(PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        self.terminations.success = DoneTerm(
            func=mdp.cubes_stacked,
            params={"cube_1_cfg": SceneEntityCfg("cube_1"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
        )

@configclass
class PiperCubeStackBlueGreenRedEnvCfg(PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        self.terminations.success = DoneTerm(
            func=mdp.cubes_stacked,
            params={
                "cube_1_cfg": SceneEntityCfg("cube_1"),
                "cube_2_cfg": SceneEntityCfg("cube_3"),
                "cube_3_cfg": SceneEntityCfg("cube_2"),
            },
        )

3.2.6 Environment Registration (stack/config/piper/__init__.py)

Finally, register the task in Gymnasium so it can be launched via the IsaacLab runner:

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
import gymnasium as gym

# from isaaclab_tasks.manager_based.manipulation.stack.config.franka import agents
from agx_teleop.tasks.manager_based.manipulation.stack.config.piper import agents

##
# Register Gym environments.
##

gym.register(
    id="Isaac-Stack-Cube-Piper-IK-Rel-v0",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    kwargs={
        "env_cfg_entry_point": f"{__name__}.piper_stack_ik_rel_env_cfg:PiperCubeStackEnvCfg",
        "robomimic_bc_cfg_entry_point": f"{agents.__name__}:robomimic/bc_rnn_low_dim.json",
    },
    disable_env_checker=True,
)

Step 4.Teleoperation & Data Collection

4.1 Teleoperation Configuration

Create the tool directory and script files within the agx_teleop plugin:

cd agx_teleop
mkdir -p scripts/tools
touch scripts/tools/record_demos.py \
      scripts/tools/repaly_demos.py

Copy the corresponding IsaacLab source code and modify the imports for tasks. Required sections are listed below:

from isaaclab.envs import DirectRLEnvCfg, ManagerBasedRLEnvCfg
from isaaclab.envs.mdp.recorders.recorders_cfg import ActionStateRecorderManagerCfg
from isaaclab.envs.ui import EmptyWindow
from isaaclab.managers import DatasetExportMode

# import isaaclab_tasks  # noqa: F401
import agx_teleop.tasks  # noqa: F401
from isaaclab_tasks.utils.parse_cfg import parse_env_cfg

scripts/tools/repaly_demos.py:

if args_cli.enable_pinocchio:
    import isaaclab_tasks.manager_based.locomanipulation.pick_place  # noqa: F401
    import isaaclab_tasks.manager_based.manipulation.pick_place  # noqa: F401

# import isaaclab_tasks  # noqa: F401
import agx_teleop.tasks  # noqa: F401
from isaaclab_tasks.utils.parse_cfg import parse_env_cfg

4.2 Teleoperation Configuration

# First, create the dataset directory
mkdir -p datasets

# Command for teleoperated data collection; keyboard teleoperation used here. Available devices: spacemouse, keyboard, handtracking
python scripts/tools/record_demos.py \
    --task Isaac-Stack-Cube-Piper-IK-Rel-v0 \
    --device cpu --teleop_device keyboard \
    --dataset_file ./datasets/dataset.hdf5 \
    --num_demos 10

# Replay the collected dataset
python scripts/tools/replay_demos.py \
    --task Isaac-Stack-Cube-Piper-IK-Rel-v0 \
    --device cpu --dataset_file ./datasets/dataset.hdf5

Keyboard Teleoperation Method:

Keyboard Controller for SE(3): Se3Keyboard
   Reset all commands: R
   Toggle gripper (open/close): K
   Move arm along x-axis: W/S
   Move arm along y-axis: A/D
   Move arm along z-axis: Q/E
   Rotate arm along x-axis: Z/X
   Rotate arm along y-axis: T/G
   Rotate arm along z-axis: C/V

Q & A

Q1: Why is the End-Effector (EE) offset set to [0.0, 0.0, 0.14] in the IK configuration?

A: This offset accounts for the physical length of the PiPER gripper. It shifts the Tool Center Point (TCP) from the gripper_base link to the actual center of the fingertips. Without this body_offset, the Inverse Kinematics (IK) controller would attempt to align the robot’s “wrist” with the target cube, causing the physical fingers to overshoot and collide with the objects.

Q2: How does the workflow ensure the quality and validity of the recorded .hdf5 datasets?
A: The workflow employs a two-layer validation system:

• Automated Filtering: The record_demos.py script uses DatasetExportMode.EXPORT_SUCCEEDED_ONLY, meaning only episodes that trigger the success termination criteria (cubes stacked for a set number of steps) are saved.
• Manual Verification: The replay_demos.py script allows developers to re-simulate recorded actions to confirm that the recorded trajectories consistently result in a successful task outcome before starting training.

Q3: What is the reasoning behind recommending --device cpu during the teleoperation recording phase?

A: Teleoperation is highly sensitive to input latency. When recording demonstrations, the GPU is often heavily taxed by real-time rendering and physics calculations. Running the simulation logic on the CPU can reduce “input lag” and jitter, providing the human operator with a smoother control experience, which leads to higher-quality, more natural trajectory data for imitation learning.

Q4:In piper_stack_ik_rel_env_cfg.py, why choose the relative pose mode with use_relative_mode=True?
A: The Relative Mode is more intuitive for human teleoperation: the input from the keyboard or joystick is mapped to “increments of the current end position” rather than “absolute coordinates”. During training, relative control can also reduce the difficulty for the model to learn spatial features, as it focuses on motion trends rather than fixed global coordinates, which helps improve the model’s generalization ability across different environmental positions.

Q5: Why is the --fix-base parameter required during the URDF to USD conversion process?
A: PiPER robotic arms are usually fixed on workbenches. In Isaac Sim/Isaac Lab, if the robot base is not fixed (Fix), the robotic arm will displace or tip over in the virtual space due to reaction forces when performing high-dynamic actions. The --fix-base parameter rigidly connects the base_link in the URDF to the global coordinate system (World Frame) of the simulation world, ensuring the stability of the robotic arm base.

Have Question?

If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.

1 post - 1 participant

Read full topic

Hey ROS community,

We’ve been building something called **BlackBox** — think of it like a flight data recorder for your ROS 2 robots.

The idea: every time your robot does something unexpected in the field, you shouldn’t have to dig through raw logs hoping you can reproduce it. BlackBox lets you record **episodes** (a bounded window of what your robot was doing), tag failures, monitor `/diagnostics`, and see fleet-wide patterns over time — without major changes to your existing ROS 2 setup.

We’re genuinely at the stage where we want to know if we’re solving the right problem.

**How you can connect:**

- **ROS 2 node** — the primary integration, works today

- **REST API** — for non-ROS systems or custom pipelines

- **RabbitMQ & MQTT** — for event-driven and IoT setups *(under active development)*

If your stack doesn’t speak ROS 2 natively, the API/MQTT path is coming — and knowing what people actually need here would help us prioritize.

**Try it without signing up first:**

- Live sandbox with real demo data: BlackBox Robotics — Failure Intelligence for Robotics | ROS 2 Data Platform

- Analyze view: BlackBox Robotics — Failure Intelligence for Robotics | ROS 2 Data Platform

**Or create a free account and connect your own robot:**

Sign up at https://bbrobotics.in — it takes 2 minutes, no credit card. You can hook up a ROS 2 node, upload an episode, and see how it looks on your actual data. That’s probably the most useful way to poke holes in it.

**What we want from you:**

- Is this a problem you actually have in your deployments?

- What’s obviously missing that would make this useful?

- What would make you never use something like this?

- If you’ve dealt with hard-to-reproduce field failures, how are you handling it today?

- Would MQTT/RabbitMQ support change anything for your setup?

We’d rather hear “this is pointless because X” now than after 6 more months of building. Roasts, edge cases, and skeptical takes all welcome.

1 post - 1 participant

Read full topic

Hi everyone,

I wanted to share a project I’ve been working on called BAGEL (BAG ExpLoration). It’s an open-source, fully static web application designed to let you inspect, scrub, and visualize ROS 1 and ROS 2 bag files locally in your browser with zero installation, zero configuration, and no cloud uploads.

Live App: https://bagel-ros2.vercel.app

GitHub Repository: GitHub - Hussain004/BAGEL: ROS Bag Visualizer · GitHub

Demo Playlist: https://youtube.com/playlist?list=PLHZhJb0NUk4Jv26hmARxgLa_dVw6Ab5j8&si=zSZQnYMwOmESiOgq

Why build another visualizer?

While tools like RViz2 are great for live debugging, standing up a full ROS environment just to quickly sanity-check a data log on a laptop or to share results with a non-ROS collaborator can be friction-heavy.

BAGEL is built entirely on a client-side architecture. Your data never leaves your machine. It utilizes web workers and streaming chunk decoders to handle heavy robotics telemetry data directly in the browser thread.

Key Features

• Multi-Format Ingestion: Seamless drag-and-drop or file-selection for ROS2 `.mcap` (Humble/Jazzy compatibility), ROS2 `.db3` (SQLite), and legacy ROS1 `.bag` files.

• TF-Aware 3D Panel (Three.js): Decodes and visualizes `sensor_msgs/PointCloud2`, `sensor_msgs/LaserScan`, `geometry_msgs/PoseStamped`, `nav_msgs/OccupancyGrid` SLAM maps, and `visualization_msgs/MarkerArray`. It natively handles action semantics (`ADD`, `MODIFY`, `DELETEALL`) and lifetime fade-outs.

• Geographical Grounding: The trajectory panel parses `sensor_msgs/NavSatFix` and projects the coordinates using an opt-in OpenStreetMap tile underlay with an in-memory LRU cache.

• Remote URL Streaming via HTTP Range: You can paste a public URL (S3 bucket, GitHub release asset, etc.). For MCAP and ROS1 bags, the tool only streams the precise chunks you scrub through, allowing a multi-gigabyte log to open in under a second.

• Custom Schema On-the-Fly Injection: Since `.db3` files do not embed message schemas, anything outside the standard packages usually fails to decode. BAGEL adds a runtime paste flow backed by `@foxglove/rosmsg` validation and `l@foxglovecalSto@foxgloveage` persistence, allowing you to paste your custom `.msg` definitions once and decode proprietary planner or vendor packages instantly.

• Blazing-Fast Time-Series: Uses `uPlot` to instantly graph any numerical primitive from any message payload with zero layout lag.

Technical Stack

The frontend is built using React 19, TypeScript, and Vite. Heavy parsing workloads are offloaded to worke@mca@mcaps:

• @mcap/c@foxglov@foxglovere & @foxglove/rosbag@mcap for low-level@mc@foxglove@foxgloveparsing.
• @foxglove sql.js (WebAssembly) to@foxglovequery unindexed `.db3` tables in memory.
• seek-bzip & lz4js running in web workers to handle compressed ROS1 chunks natively without blocking UI rendering loops.

Feedback Welcome!

This is a fully open-source effort aimed at making data sharing and quick field log triage as low-friction as possible.

I would love to hear feedback from the community regarding performance benchmarks, missing primitive types you rely on, or general UX enhancements. Feel free to open issues or submit PRs on the repository!

6 posts - 4 participants

Read full topic

Hi ROS Community,

We’re excited to announce LinkForge v1.4.0, a major architectural update that transforms LinkForge into a programmable Intermediate Representation (IR) for robot descriptions.

If you’ve ever debugged a crash in Gazebo caused by a bad inertia tensor, or spent hours manually resolving namespace collisions in URDFs and MoveIt 2 SRDFs when attaching a manipulator to a mobile base, this is built for you.

Key Features for ROS Developers:

• Lossless URDF/XACRO Parsing: Preserves unknown metadata/tags, allowing seamless round-trips.
• Automated SRDF & MoveIt 2: Programmatically compose subsystems and LinkForge will automatically merge planning groups and self-collision matrices into a unified SRDF.
• ROS 2 Control & Gazebo: Auto-generates ros2_control hardware interface tags based on high-level Python definitions.
• Physical Linter: Calculates rigorous inertia tensors from geometry and catches non-physical masses before you launch your simulation.

We’d love the community’s feedback on how well it integrates with your existing ROS 1 / ROS 2 workflows!

GitHub: LinkForge
Pypi: pip install linkforge-core

3 posts - 2 participants

Read full topic

Introducing Vulcanexus HRI: Human-Robot Interaction for ROS 2

As robots become increasingly present in workplaces, public environments and everyday services, the ability to perceive, communicate and interact naturally with people is becoming more important than ever.

We are happy to introduce Vulcanexus HRI, a new framework within the Vulcanexus ecosystem designed to help developers integrate Human-Robot Interaction capabilities into ROS 2 applications.

Vulcanexus HRI provides a modular set of tools for perception, communication and interaction, including:

• Face detection

• Human pose recognition

• Emotion recognition

• Speech-to-text

• Text-to-speech

• Visual interaction components

• Shared HRI message definitions

The framework is designed to work seamlessly within existing Vulcanexus and ROS 2 systems, allowing developers to enrich robotic applications with human-related information without redesigning the rest of their architecture.

The different modules can be used independently or combined depending on the application requirements, making Vulcanexus HRI suitable both for experimentation and for more advanced interactive robotic systems.

Documentation and Getting Started guide: 8. Vulcanexus HRI Overview -

We have also prepared a short video showing VulcanBot interacting using Vulcanexus HRI capabilities.

We would love to hear feedback from the ROS community and discuss possible use cases, integrations and future improvements.

1 post - 1 participant

Read full topic

Embodied Intelligence Upgraded: Self-Improving Robot Policies via RISE World Model Reasoning

Vision-Language-Action (VLA) models struggle with contact-rich tasks such as dynamic sorting and flexible packing, where tiny execution deviations often result in failure. Traditional real-world reinforcement learning (RL) faces barriers to scaling: high hardware costs, manual environment resets, and slow serial interactions.

The RISE framework (RSS 2026), developed by CUHK MMLab, HKU OpenDriveLab, Tsinghua University and so on, solves this with an “imagination-based self-evolution” paradigm. By training in simulated “imaginary space” instead of the physical world, it eliminates costly trial-and-error and achieves a 95% success rate on complex manipulation tasks using the AgileX Piper robotic arm.

References & Links

• Project Page: https://opendrivelab.com/RISE/
• Paper Link: https://arxiv.org/pdf/2602.11075
• Hardware Used: AgileX PiPER 6-DoF Robotic Arm
• Project HomePage：RISE: Self-Improving Robot Policy with Compositional World Model
• Open Source: GitHub - OpenDriveLab/RISE: [RSS 2026] Code for RISE: Self-Improving Robot Policy with Compositional World Model · GitHub

1. Real-World Robot Learning Still Struggles to Scale

Modern Vision-Language-Action (VLA) models can perform basic robotic manipulation through imitation learning (IL), but they still struggle with contact-rich tasks involving dynamic objects, deformable materials, and bimanual coordination. Even small execution errors can lead to task failure.

While reinforcement learning (RL) offers a path toward autonomous robot learning, real-world training remains limited by:

• high hardware costs

• low training efficiency

• manual environment resetting

• safety and reliability risks

Researchers have long relied on simulation and world models to improve scalability, but challenges such as the sim-to-real gap, unstable action generation, and slow robot planning continue to limit real-world deployment in embodied AI and autonomous robotics.

2.Building Self-Improving Robot Policies with Compositional World Models

The core idea behind RISE is simple: instead of relying entirely on expensive real-world robot training, robots improve themselves inside an imagined environment powered by compositional world models.

RISE separates robot learning into two key components:

1.Controllable Dynamics Model

A fast and controllable world model predicts future robot interactions and manipulation outcomes.

• Built on the Genie Envisioner video diffusion model

• Generates multi-view future robot trajectories within seconds

• Uses lightweight action encoders to ensure actions remain physically consistent and controllable

• Pretrained on large-scale robot datasets such as Agibot World and Galaxea for realistic robot manipulation prediction

2.Progress Value Model

A value prediction model continuously evaluates robot behavior during manipulation tasks.

• Built on the π0.5 Vision-Language-Action (VLA) framework

• Combines progress regression and temporal-difference (TD) learning

• Detects subtle manipulation failures such as object slipping or unstable contact

• Outputs real-time advantage scores for autonomous robot policy optimization

Together, these components enable scalable self-improving robot learning for embodied AI, contact-rich manipulation, and sim-to-real robotic systems.

3.Closed-Loop Self-Improving Robot Learning in Imagined Environments

By using the AgileX PiPER Robot Arm, RISE enables autonomous robot policy improvement entirely within imagined environments, reducing the need for large-scale real-world trial-and-error training.

The self-improving robot learning pipeline consists of three stages:

1. Policy Warm-Up

The robot policy is initialized using a small amount of offline robot data, including demonstrations and successful or failed manipulation rollouts, allowing the system to learn basic robot manipulation skills.

2. Imagined Rollout

The robot policy generates actions, while the compositional world model predicts future robot interactions and manipulation trajectories. At the same time, the value model evaluates action quality and estimates real-time advantage values.

3. Policy Optimization

High-advantage robot actions are reinforced, while low-quality behaviors are gradually filtered out through iterative policy optimization, enabling continuous self-improving robot learning.

RISE performs the entire optimization process in virtual environments without requiring repeated real-world robot interaction. During inference, the world model is no longer involved, meaning the system introduces no additional runtime computation cost for real-world robot deployment.

4.Three Challenging Robot Manipulation Tasks, Significant Performance Gains

RISE was evaluated on three high-difficulty real-world robot manipulation tasks, including dynamic brick sorting, flexible bag packing, and precision box assembly, significantly outperforming existing imitation learning (IL) and reinforcement learning (RL) baselines.

Benchmark Results

Benchmark evaluation on the AgileX PiPER Robot Arm indicates a substantial improvement in manipulation success rate, as shown in the results below.

• Dynamic Brick Sorting

Success rate improved from 50% → 85%

• Flexible Bag Packing

Success rate improved from 40% → 85%

• Precision Box Assembly

Success rate improved from 60% → 95%

Compared with online reinforcement learning methods such as PPO and DSRL, RISE demonstrated significantly more stable robot policy optimization without training collapse.

Compared with offline RL approaches such as RECAP, RISE continuously expands robot training distributions through imagined world-model rollouts, greatly improving generalization and reducing overfitting in contact-rich manipulation tasks.

5.Key Design Choices Behind RISE

Ablation studies show that every core module in RISE plays a critical role in stable robot learning and contact-rich manipulation performance.

Key Findings

• Removing dynamics model pretraining reduced dynamic sorting accuracy by 32%

• Removing the task-centric batch strategy decreased overall task success by 30%

• Removing temporal-difference (TD) learning from the value model weakened failure detection and reduced success rates by 35%

• An offline-to-online data ratio of 0.6 achieved the best balance between robot policy stability and autonomous exploration

6.Toward Scalable and Low-Cost Self-Improving Robot Learning

RISE demonstrates that a well-trained compositional world model can directly serve as an online reinforcement learning environment for real-world robot manipulation.

This brings three major advantages for embodied AI and autonomous robotics:

1. Lower Training Cost

RISE shifts expensive real-world robot trial-and-error into scalable computation, making high-performance robot policy learning more accessible for smaller robotics teams.

2. Higher Training Efficiency

Unlike traditional real-world RL, imagined robot interactions can run in parallel, dramatically accelerating robot learning and policy optimization.

3. Safer Robot Learning

By performing large-scale trial-and-error inside virtual environments, RISE reduces physical risks and prevents damage to real robotic systems during training.

At the same time, several challenges still remain:

• small sim-to-real inconsistencies in rare manipulation scenarios

• manual tuning of offline-to-online data ratios

• high computational cost for large-scale world model training

Future research will likely focus on uncertainty-aware world models, physics-constrained robot prediction, and more efficient embodied AI training pipelines, enabling robots to solve increasingly complex real-world manipulation tasks through autonomous imagined learning.

Have Question?

If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.

1 post - 1 participant

Read full topic

Please come and join us for this coming meeting at Mon, Jun 1, 2026 4:00 PM UTC→Mon, Jun 1, 2026 5:00 PM UTC, where we plan to continue writing the survey questions for a new State of Cloud Robotics survey. The last survey was in 2024 (see https://cloudroboticshub.github.io/survey), and we’d like to refresh the results as of this year. We made good progress last session in updating our old questions and introducing new ones, so this coming session will be further refining the questions and grouping them in a logical way.

If you’re interested in watching the previous session, the meeting recording is available on YouTube.

The meeting link for next meeting is here, and you can sign up to our calendar or our Google Group for meeting notifications or keep an eye on the Cloud Robotics Hub.

Hopefully we will see you there!

1 post - 1 participant

Read full topic

Hi ROS 2 Community,

To go with the Lyrical Luth release, RTI has updated the Connext Robotics Toolkit with Connext 7.7 LTS. The toolkit provides a simple single-step Debian installation of Lyrical and Connext DDS RMW on Ubuntu Linux, now supporting both amd64 and arm64 including Raspberry Pi and NVIDIA Orin platforms. It remains free-of-charge for prototyping, research, non-commercial and academic use.

Connext 7.7 brings several enhancements for ROS users looking to move from prototype to production:

Improved ROS–DDS interoperability. New Topic and Type Aliasing automatically maps between ROS-mangled and conventional DDS names, making it far easier to integrate ROS 2 systems with native DDS applications based on standards like AUTOSAR, GVA and UMAA, without any code changes or naming gateways.

Improved performance, scalability and security. Faster discovery with less network overhead, reduced memory footprint, and enhanced security capabilities.

To learn more, see installation instructions and try the new aliasing feature hands-on, visit the full announcement here: https://community.rti.com/forum-topic/connext-77-ros-lyrical-luth-now-available

Feel free to reach out if you have any questions.

1 post - 1 participant

Read full topic

Quick video promo: https://youtu.be/yYms177Rx3w?si=qH2vG1ggwfYdtumr
Datasets are available on Kaggle: Telekinesis | Kaggle
Telekinesis Agentic Skill library: https://docs.telekinesis.ai/

1 post - 1 participant

Read full topic

ICRA 2026 in Vienna Open Thread

Hi Everyone,

Now that the Lyrical release and Open Hardware Summit are behind me (ask @KimMcG or @adityakamath about their experiences at OHS ) I want to take a minute to talk about ICRA 2026 in Vienna.

@mrpollo and I have scrambled to put together not one, but two open source robotics meetups at ICRA 2026:

• Our ROS meetup for ICRA will be on Tuesday Tue, Jun 2, 2026 4:00 PM UTC (6pm local time) and you can RSVP for that meetup here.
• Our PX4 meetup for ICRA will be held on Thursday, Thu, Jun 4, 2026 5:00 PM UTC (7pm local time) and you can register for that event here.

Both @mrpollo and I will be at ICRA all week so feel free to message us if you want to meet and chat about what you are working on.

If you or your colleagues happen to be presenting a ROS-related talk at ICRA feel free to shamelessly self promote it in the thread below. We want to hear about what you are working on!

p.s. My suitcase is full of Lyrical stickers fresh from the printer. I’ll be handing them out so come and find me.

3 posts - 2 participants

Read full topic

Hiroz (High-performance Interoperable Robotics on Zenoh) is a project by ZettaScale Technology, the company supporting Eclipse Zenoh — and the new name for what was previously known as ROS-Z.

Hiroz is a robotics stack built on top of Zenoh, implemented in pure Rust. The core idea is to provide full ROS 2 compatibility while enabling roboticists to go beyond what the ROS architecture allows — with less coupling to ROS internals, faster iteration, and a foundation for enriching ROS itself.

Relationship with ROS 2 and rmw_zenoh

Hiroz is not a replacement for rmw_zenoh. It is a distinct layer that sits on top of Zenoh directly:

• rmw_zenoh is a Tier-1 RMW implementation based on Zenoh and using it with specific key expressions and encoding (CDR). It can be used as a shared library via the ROS 2 rcl, rclcpp, rclpy and rcl-rs API
• Hiroz is a Zenoh-native platform — not bound to the RMW interface, but still interoperable with ROS 2 by using the same key expressions and encoding

Both can coexist and communicate over the same Zenoh infrastructure, meaning you can mix your regular ROS 2 Nodes with enhanced Hiroz Nodes in a same system.

Features extending beyond ROS

• Multiple serialization formats: CDR (ROS-compatible), Protobuf, and more to come
• Multiple language bindings: Rust (native), Python, and Go (Go bindings sponsored by Dexory)
• Zero-copy and GPU memory support: Zenoh’s buffer-aware pub/sub enables CPU and CUDA memory backends coming soon
• Full-Rust stack: memory safety and predictable performance by design; fits naturally into AI-assisted development workflows
• Framework interoperability: communicate between ROS 2 nodes and non-ROS systems over the same Zenoh infrastructure

For more details, see the full feature comparison page

Supported ROS 2 distributions

Distribution	Status
Humble	Supported
Jazzy	Supported
Kilted	Supported
Lyrical	Very soon

Links

• Repository: https://github.com/ZettaScaleLabs/hiroz
• Documentation: https://zettascalelabs.github.io/hiroz

Feedback, questions, and contributions are welcome. If you have been following ROS-Z, everything carries over — just under the new name.

At ZettaScale, we are committed to keep supporting rmw_zenoh and also to evolving Hiroz in close collaboration with partners and customers. If you have specific needs — performance, interoperability, custom transports, new language bindings — feel free to reach out.

— The ZettaScale team

5 posts - 3 participants

Read full topic

6 posts - 4 participants

Read full topic

Hello everyone,

My name is Marussa Metocharaki. I’m an independent researcher from Greece working on QERRA-v2 Classical, a fully explainable ethical evaluation engine designed for robotics and autonomous systems.

The system uses the SEMEV-12 framework (12 ethical vectors) and returns a transparent score with reasoning. It is completely classical — no neural networks involved.

I recently released v1.8.8, where all 12 vectors are now active. The public API on Hugging Face has also been updated to this version.

I’ve written a short technical brief for the robotics community here:
QERRA_FOR_ROBOTICS.md

I would really appreciate some guidance from people with more experience in ROS 2 and Behavior Trees.

My main question is: How would you recommend integrating something like QERRA as a Condition node in a Behavior Tree?
Are there any common patterns, best practices, or things to watch out for when using an external ethical check before executing an action?

I’m especially interested in real-world testing ideas or examples from people who have worked with decision-making nodes in ROS 2.

Any feedback or suggestions would be very helpful.

I’m still learning and would really value input from the community.

hank you in advance!

Best regards,
Marussa Metocharaki

1 post - 1 participant

Read full topic

With the upcoming release of ROS 2 Lyrical Luth in May 2026, Fast DDS will continue serving as the default middleware implementation for ROS 2.

For more than 11 years, Fast DDS has accompanied the ROS 2 ecosystem through every distribution, continuously evolving together with the community to provide scalable, reliable, and high-performance communications.

At eProsima, we are extremely proud to continue contributing to the ROS ecosystem and to support the growing adoption of ROS 2 in robotics, industrial automation, autonomous systems, defense, and many other domains.

Lyrical Luth also brings interesting communication-related improvements, including Native Buffers support in the Fast DDS RMW implementation, opening the door to more efficient accelerated-memory and GPU-oriented transport workflows.

We would like to thank the entire ROS community, Open Robotics, and all contributors involved in making ROS 2 stronger with every release.

Looking forward to Lyrical Luth.

2 posts - 2 participants

Read full topic

Hi all,

rclnodejs 2.0.0 is out today, on the heels of the ROS 2 Lyrical Luth GA on May 22, 2026. This release adds rclnodejs/web — a typed browser SDK plus a small runtime that lets a web page talk to ROS 2 over a single WebSocket, with the same capabilities also reachable over plain HTTP for curl, Postman, and AI agents.

New in 2.0.0:

• A typed SDK (rclnodejs/web) for browsers and Node — one ROS 2 type name in, request / reply / message all typed.
• A capability runtime (npx rclnodejs-web) that only exposes what’s declared in web.json or on the CLI; anything else is rejected with code: 'not_exposed' before any ROS 2 API runs.
• An HTTP fallback for call and publish — every capability is also reachable over plain POST /capability/<verb>/<name>, so curl, Postman, and AI-agent tool-use work without a hand-written shim.

Carried over from 2.0.0-beta.0 (May 2026): the rosocket WebSocket gateway, ROS 2 Lyrical Luth (Ubuntu 26.04) support, and Linux x64 / arm64 N-API prebuilds — one artifact built against Node.js 20.20.2 runs unchanged on every Node.js ≥ 20.20.2, including 24.x and 26.x — across the full distro matrix (Humble, Jazzy, Kilted, Lyrical, Rolling).

Browser side, in 5 lines

import { connect } from 'rclnodejs/web';

const ros = await connect('ws://robot.local:9000/capability');

const reply = await ros.call<'example_interfaces/srv/AddTwoInts'>(
  '/add_two_ints', { a: '2n', b: '40n' }
);
console.log(reply.sum); // '42n', typed as `${number}n`

Pass a ROS 2 type name as the generic and the corresponding request, reply, or message shape is typed for you.

Server side, no JavaScript needed

source /opt/ros/lyrical/setup.bash
npx -p rclnodejs rclnodejs-web --port 9000 --http-port 9001 \
  --call /add_two_ints=example_interfaces/srv/AddTwoInts
# rclnodejs/web listening on ws://localhost:9000/capability (1 capability)
#                also http://localhost:9001/capability (call/publish only)

Or pass a web.json config file. Either way, the browser only reaches what the server has declared — anything else is rejected with code: 'not_exposed' before any ROS 2 API runs. With --http-port on, every call / publish is also reachable from plain HTTP:

curl -X POST -H 'content-type: application/json' \
  -d '{"a":"2n","b":"40n"}' \
  http://localhost:9001/capability/call/add_two_ints
# => {"sum":"42n"}

Tutorial + demos

• SDK guide: web/README.md
• JS demo (no toolchain): demo/web/javascript/
• TS + Vite demo: demo/web/typescript/

Feedback very welcome — particularly on the TypeScript surface, since the wire protocol locks in this release.

For those new to the project: rclnodejs is the Node.js client library for ROS 2, maintained under the Robot Web Tools umbrella.

Cheers,
Minggang

1 post - 1 participant

Read full topic

Paste a URDF/Xacro file and instantly:
• Preview the robot in 3D
• Validate structure live
• Share public robot links
• Export to Isaac Sim / MuJoCo / SDF
• Auto-fix common URDF problems

No ROS install required.

The goal is to make robot development feel more like modern web development tools.

Playground:

7 posts - 4 participants

Read full topic

Hi everyone,

We are thrilled to announce that ROSCon China 2026 is officially on the horizon! As one of the most vibrant regional ROS events, ROSCon China continues to bring together developers, researchers, and industry leaders to share the latest advancements in the ROS ecosystem, embodied AI, robotics technologies and commercial robotics deployment.

Whether you are a seasoned core contributor, an enterprise, or a student diving into robot, this event is designed for you.

Event Details

• Dates: October 16-17, 2026

• Location: Shanghai, China

• Format: In-person with live-streaming available

Call for Proposals (CFP)

The official website, speaker application portal, and early-bird registration link will be launched very soon. We are looking for technical presentations, case studies, and lightning talks!

Stay tuned for the official links which we will append to this post. In the meantime, feel free to drop any questions or suggestions in the thread below.

Looking forward to seeing the ROS community gather in China!

Best regards,

Xinyu Zhang
ROS Education Foundation in China

1 post - 1 participant

Read full topic

MuJoCo Simulation and ROS2 Control Workflow | Piper Robotic Arm for Embodied AI

This project is a MuJoCo-based simulation foundation for the Piper robotic arm. It is designed to provide a general-purpose MuJoCo simulation environment for the Piper arm, with support for robot control testing and simulation workflow development.

Project has already completed

• the launch of the C++ based simulation environment
• control testing of the simulated robot using a test ROS 2 control plugin

Recommended Environment

• Operating System: Ubuntu 22.04 or compatible Linux distribution
• ROS Version: ROS 2 Humble
• Compiler: C++17 or later

Reference & Resources

• Reference Project: https://github.com/unitreerobotics/unitree_mujoco
• PiPER robotic arm platform: https://global.agilex.ai/products/piper

The following sections provide step-by-step guidance for building the current version of the Piper MuJoCo simulation environment and using the ROS 2 test plugin to control the robot in simulation.

1.Project File Structure

agilex@agilex-Bellator-N176B:~/project/agilex_arm_mujoco$ tree -L 3
├── agilex_arm
│   └── agilex_piper
│       ├── assets
│       ├── piper_body.xml
│       ├── piper.png
│       ├── piper.xml
│       └── scene.xml
├── images
│   ├── mujoco_piper.png
│   └── piper.png
├── README_EN.md
├── readme.md
└── simulate
    ├── CMakeLists.txt
    ├── config.yaml
    ├── mujoco
    │   ├── bin
    │   ├── include
    │   ├── lib
    │   ├── model
    │   ├── sample
    │   ├── simulate
    │   └── THIRD_PARTY_NOTICES
    └── src
        ├── lodepng
        ├── main.cc
        ├── param.h
        ├── plugin_manager.cc
        ├── plugin_manager.h
        ├── ros2_control_plugin.cc
        ├── ros2_control_plugin.h
        └── sim_plugin.h

2.Project Setup & Build

2.1 Build the agilex_arm_mujoco Project

The build process for the agilex_arm_mujoco project follows the same workflow as the unitree_mujoco project.
This section provides a step-by-step guide to setting up the simulation base, including optional instructions for enabling the ROS 2 control plugin.

Step 1.Clone the Repository

First, create a working directory and clone the project repository:

mkdir agilex_arm && cd agilex_arm
git clone https://github.com/yanyuze1/agilex_arm_mujoco.git

By default, the project includes MuJoCo 3.3.0 for ROS 2 control testing.
If you want to use a different MuJoCo version, download and replace it manually before building.

Step 2. Two Build Modes

The agilex_arm_mujoco project supports two build modes:

• Option 1: Build without the ROS 2 plugin
• Option 2: Build with the ROS 2 plugin enabled

Choose the mode based on your development needs.

Option 1: Without the ROS 2 Plugin

• Step 1: Edit agilex_mujoco/simulate/CMakeLists.txt, change the default enabled ros2 plugin mode to OFF

# from
option(AGILEX_ENABLE_ROS2_CONTROL "Build ROS2 control plugin" ON)
# change to
option(AGILEX_ENABLE_ROS2_CONTROL "Build ROS2 control plugin" OFF)

• Step 2: Build the Project. After editing the file, build the simulation base:

cd agilex_arm_mujoco/simulate
mkdir build && cd build
cmake ..
make -j4

Option 2: With the ROS 2 Plugin Enabled

If you want to test robot control through the ROS 2 plugin, keep the default configuration and build the required ROS 2 dependencies first.

• Step 1: Clone the Dependency Workspace
  Create or enter the agilex_arm directory and clone the repository:

cd agilex_arm
git clone https://github.com/yanyuze1/agilex_ws.git

• Step 2: Build ROS 2 Dependencies
  Build the required packages in the ROS 2 workspace:

cd agilex_ws
colcon build --symlink-install --packages-up-to mujoco_ros2_control agilex_piper_mujoco
source install/setup.bash

• Step 3: Build the Simulation Base
  After the dependency build is complete, compile the simulation project:

cd agilex_mujoco/simulate
mkdir build && cd build
cmake ..
make -j4

2.2 Run the agilex_arm_mujoco Project

The agilex_arm_mujoco project provides two runtime modes:

• Start the simulation base only
• Start the simulation base with the ROS 2 plugin enabled

Choose the mode according to your development and testing needs.

2.2.1 Start the Simulation Base Only

If you only want to launch the simulation environment without ROS 2 integration, use the following command:

./agilex_mujoco -h
./agilex_mujoco -r piper -s scene.xml
./agilex_mujoco -r piper -s scene.xml -p ros2_control

After successful execution, the MuJoCo simulation environment will launch automatically, displaying the Piper robotic arm in the MuJoCo simulation scene.

2.2.2 Start the Simulation Base with the ROS 2 Plugin Enabled

If you only want to launch the simulation environment with ROS 2 integration, use the following command:

cd agilex_ws
source install/setup.bash
ros2 launch agilex_piper_mujoco bringup_agilex_mujoco_cartesian_motion_controller.launch.py   agilex_mujoco_exec:=/home/agilex/project/piper/agilex_mujoco/simulate/build/agilex_mujoco   # The current project uses absolute paths. Please modify them according to your own local path.

When the launch is successful, a MuJoCo simulation window will open and display the Piper robotic arm in the simulation environment.

Once the environment is fully loaded, you can start sending ROS 2 control commands.

Usage Situation

1. Send a Target End-Effector Pose
Use the following command to publish a target pose for the end effector:

ros2 topic pub --once /agilex_piper_cartesian_motion_controller/target_frame \
  geometry_msgs/msg/PoseStamped "{
    header: {frame_id: 'base_link'},
    pose: {
      position: {x: 0.2, y: 0.0, z: 0.2},
      orientation: {x: 0.0, y: 1.0, z: 0.0, w: 0.0}
    }
  }"

2. Control the Gripper

# Open the Gripper
ros2 topic pub --once /agilex_piper_gripper_position_controller/commands \
  std_msgs/msg/Float64MultiArray "{data: [0.035, -0.035]}"

# Close the Gripper
ros2 topic pub --once /agilex_piper_gripper_position_controller/commands \
  std_msgs/msg/Float64MultiArray "{data: [0.0, 0.0]}"

3. Check the Robot State
To view the current end-effector pose, use:

ros2 topic echo /agilex_piper_cartesian_motion_controller/current_pose

3.Afterword

This project is built upon and integrated from several open-source repositories. The main credit goes to the original authors and the open-source community.

We would like to sincerely thank the open-source community for making these technologies accessible through shared code and practical experience.

MuJoCo + ROS2 Robotic Manipulation FAQ

1. Why use MuJoCo for the Piper robotic arm instead of Gazebo or Isaac Sim?

MuJoCo provides a lightweight and high-performance physics simulation workflow, making it ideal for rapid robotics prototyping, control validation, and future reinforcement learning research. Compared with heavier simulators, it can run efficiently on more accessible hardware.

2. Can this simulation workflow be transferred to a real Piper robotic arm?

Yes. The ROS2 control architecture is designed to align simulation and real hardware interfaces, making future sim-to-real deployment more practical.

3. Why build a custom MuJoCo + ROS2 workflow instead of using existing robotics simulators?

This project focuses on providing a simpler and more flexible robotics development workflow with lower hardware requirements and easier customization for developers and researchers.

4. Is this MuJoCo robotic arm project suitable for reinforcement learning (RL)?

Yes. The simulation framework can serve as a foundation for reinforcement learning workflows such as PPO training, imitation learning, grasping policy development, and sim-to-real robotic manipulation research.

5. How does ROS2 Control work with MuJoCo simulation?

The project integrates ROS2 Control plugins into the MuJoCo simulation environment, allowing developers to send ROS2 motion commands directly to the simulated robotic arm and receive real-time robot state feedback.

Still Have Question?

If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.

8 posts - 6 participants

Read full topic

Hi community.

I have published two articles on useful tools for ROS 2 on Medium. First article on TUI based packages and second article is about web based packages. These articles are presented as short overview and hands-on tutorials on several packages from specific category.

Looking for your comments. Hope it will be useful for everyone in the community.

1 post - 1 participant

Read full topic

Hi all,

I’m from Open AR Cloud (openarcloud.org), a volunteer-run non-profit promoting open and interoperable spatial computing. Our initial focus was around world-scale augmented reality and key enablers like spatial discovery and visual positioning.

More recently, we’ve created SpatialDDS (spatialdds.org), a protocol built on DDS for sharing spatial data (detections, poses, maps, zones) across domains — robotics, autonomous vehicles, IoT, digital twins, 6G sensing. It defines typed messages (Detection3D, FramedPose, GeoPose, MapAlignment, etc.) plus spatial discovery and multi-operator namespacing. As an example, it would allow multiple autonomous vehicle fleets to share detections and planned trajectories at an intersection, fused into a model no single fleet could build alone.

There’s a ROS 2 bridge (sensor_msgs, vision_msgs, tf2 frame mapping), plus bridges for MCAP, MQTT, and WebSocket. The multi-operator fusion demo runs with docker compose up.

Spec: spatialdds.org Demo: github.com/OpenArCloud/SpatialDDS-demo

We’d love to get feedback from the ROS community!

Cheers,

James

1 post - 1 participant

Read full topic

How to Build a Robot Arm RL Grasping System in Isaac Lab | NERO Arm

This project presents a reinforcement learning workflow for Embodied AI manipulation built on the Nero robotic arm, SO-ARM101, and NVIDIA Isaac Lab. It establishes a simulation-driven framework for training and evaluating robotic manipulation policies, with a focus on preparing the system for simulation-to-real transfer.

Project Summary

Tech Stack

• RL training pipeline
• policy validation process
• robotic manipulation task configuration
• simulation-to-real transfer preparation

Key Specifications

• Programming Language: Python 3.8+
• Hardware: Nero Robotic Arm https://global.agilex.ai/products/nero
• Base Framework: SO-ARM101 https://github.com/MuammerBay/isaac_so_arm101
• Simulation Platform: NVIDIA Isaac Lab
• Open-source implementation:https://github.com/agilexrobotics/Agilex-College/tree/master/isaac_sim/agx_arm_IsaacLab

1. Project Setup and Environment Preparation

1.1 Install Isaac Lab

Follow the official guide to install Isaac Lab:
Isaac Lab Pip Installation Guide

We use the pip-based installation method (recommended).

Environment:

• Conda virtual environment
• Python development environment
• NVIDIA Isaac Lab
• Nero robotic arm project dependencies

1.2 Install the uv Package Manager

This project uses uv as its Python package manager.

As a fast, next-generation tool, uv delivers:

• Faster package installation
• Efficient dependency resolution
• Built-in virtual environment management

Compared to traditional tools like pip, uv streamlines setup and reduces environment issues in Python-based robotics and embodied AI workflows.

First, install uv with a single command:

curl -LsSf https://astral.sh/uv/install.sh | sh

After installation, restart your terminal or run the following command to activate the uv environment:

source $HOME/.cargo/env

1.3 Clone the Repository and Install Dependencies

Next, clone the project repository, enter the project directory, and use uv to install all required dependencies with one command:

git clone https://github.com/smalleha/isaac_so_arm101.git
cd isaac_so_arm101
uv sync

uv will automatically create a virtual environment and install all necessary dependency packages. The entire process usually takes only a few minutes and is significantly faster than traditional pip-based installation workflows.

2. Environment Validation

To validate the setup for tasks, we first verify that the required simulation environments for the Nero robotic arm and Piper are properly registered:

uv run list_envs

The expected output should include Isaac-Nero-Reach-v0 and Isaac-Piper-Reach-v0, confirming that the environments have been installed successfully.

Next, run a simulation test with a zero-action agent to validate environment execution and ensure the robotic control pipeline works as expected:

# Test the Piper environment with a zero-action command
uv run zero_agent --task Isaac-SO-ARM100-Reach-v0

If the simulation window launches and the robotic arm behaves as intended, the environment is confirmed to be ready.

3. Project File Structure

isaac_so_arm101/
├── CITATION.cff                # Citation metadata for academic referencing
├── CONTRIBUTING.md             # Contribution guidelines (PR process, standards)
├── CONTRIBUTORS.md             # List of project contributors
├── LICENSE                     # BSD-3-Clause open-source license
├── README.md                   # Main project documentation (setup, tasks, usage)
├── pyproject.toml              # Python project metadata (dependencies, build config)
├── uv.lock                     # Dependency lockfile for reproducible environments
└── src/
    └── isaac_so_arm101/
        ├── __init__.py          # Python package initialization
        ├── robots/              # Robot models: SO-ARM100/101 simulation configs
        ├── scripts/             # Executable scripts: training, testing, playback
        ├── tasks/               # RL task definitions (reach, lift)
        └── ui_extension_example.py  # Omniverse UI extension example

This directory structure provides a clear overview of the project organization, making it easy to extend with new use cases such as Nero robotic arm example.

4. Download the URDF Model

This project uses the Nero URDF model from the agx_arm_urdf repository. After cloning the repository,
copy the nero directory into the robots folder of the isaac_so_arm101 project:

git clone https://github.com/agilexrobotics/agx_arm_urdf.git
cd agx_arm_urdf/
cp -r nero/ isaac_so_arm101/robots

Once the model has been copied, modify nero_description.urdf to make it compatible with Isaac Lab. Since the original URDF uses ROS-style package paths, these references must be converted to relative paths so that the link and mesh files can be correctly resolved. The base_link configuration is shown below as an example.

Before Edit

    <link name="base_link">
        <inertial>
            <origin xyz="-0.00319465997 -0.00005467608 0.04321758463" rpy="0 0 0"/>
            <mass value="1.06458435"/>
            <inertia ixx="0.00102659855152" ixy="0.00000186219753" ixz="-0.00000295298037" iyy="0.00114399299508" iyz="-0.00000078763492" izz="0.00090872933022"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="package://agx_arm_description/agx_arm_urdf/nero/meshes/dae/base_link.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="package://agx_arm_description/agx_arm_urdf/nero/meshes/base_link.stl"/>
            </geometry>
        </collision>
    </link>

After Edit

    <link name="base_link">
        <inertial>
            <origin xyz="-0.00319465997 -0.00005467608 0.04321758463" rpy="0 0 0"/>
            <mass value="1.06458435"/>
            <inertia ixx="0.00102659855152" ixy="0.00000186219753" ixz="-0.00000295298037" iyy="0.00114399299508" iyz="-0.00000078763492" izz="0.00090872933022"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/base_link.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/base_link.stl"/>
            </geometry>
        </collision>
    </link>

5.Configuring Isaac Lab Files

Step 1.Importing the URDF Model

After modifying the URDF file, you need to write a Python script to import the URDF model and configure the robotic arm’s motor properties, including stiffness, damping, and other relevant parameters.

This script is typically placed at:
src/isaac_so_arm101/robots/nero/nero.py

The content of this file is shown below:

from pathlib import Path

import isaaclab.sim as sim_utils
from isaaclab.actuators import ImplicitActuatorCfg
from isaaclab.assets.articulation import ArticulationCfg

TEMPLATE_ASSETS_DATA_DIR = Path(__file__).resolve().parent

##
# Configuration
##

NERO_CFG = ArticulationCfg(
    spawn=sim_utils.UrdfFileCfg(
        fix_base=True,
        replace_cylinders_with_capsules=True,
        asset_path=f"{TEMPLATE_ASSETS_DATA_DIR}/urdf/nero_gripper.urdf",
        activate_contact_sensors=False,  # Disable contact sensors until capsule collision implementation is complete
        rigid_props=sim_utils.RigidBodyPropertiesCfg(
            disable_gravity=False,
            max_depenetration_velocity=5.0,
        ),
        articulation_props=sim_utils.ArticulationRootPropertiesCfg(
            enabled_self_collisions=True,
            solver_position_iteration_count=8,
            solver_velocity_iteration_count=0,
        ),
        joint_drive=sim_utils.UrdfConverterCfg.JointDriveCfg(
            gains=sim_utils.UrdfConverterCfg.JointDriveCfg.PDGainsCfg(stiffness=0, damping=0)
        ),
    ),
    init_state=ArticulationCfg.InitialStateCfg(
        rot=(1.0, 0.0, 0.0, 0.0),
        joint_pos={
            "joint1": 0.0,
            "joint2": 0.0,
            "joint3": 0.0,
            "joint4": 2.0,
            "joint5": 0.0,
            "joint6": 0.0,
            "joint7": 0.0,
            "gripper_joint1": 0.05,
            "gripper_joint2": -0.05
        },
        # Set initial joint velocities to zero
        joint_vel={".*": 0.0},
    ),
    actuators={
        "arm": ImplicitActuatorCfg(
            joint_names_expr=["joint.*"],
            effort_limit=25.0,  # Moderate effort limit to prevent instantaneous impact shocks
            velocity_limit=1.5,
            
            # Stiffness: Optimized for the lightweight Piper robotic arm; prioritizes stability over maximum rigidity
            stiffness={
                "joint1": 200.0,
                "joint2": 170.0,
                "joint3": 120.0,
                "joint4": 80.0,
                "joint5": 50.0,
                "joint6": 20.0,
                "joint7": 10.0
            },
            
            # Damping: Critical damping strategy with ratio set to approximately 10%
            damping={
                "joint1": 100.0,
                "joint2": 60.0,
                "joint3": 70.0,
                "joint4": 24.0,
                "joint5": 20.0,
                "joint6": 10.0,
                "joint7": 5,
            },
        ),
        "gripper": ImplicitActuatorCfg(
            joint_names_expr=["gripper_joint1","gripper_joint2"],
            effort_limit_sim=22,  # Increased from 1.9 to 2.5 for stronger grip
            velocity_limit_sim=1.5,
            stiffness=800.0,  # Increased from 25.0 to 60.0 for more reliable closing
            damping=20.0,  # Increased from 10.0 to 20.0 for stability
        ),
    },

    soft_joint_pos_limit_factor=0.9,
)

Next, create an init.py file to initialize the directory as a Python module.

Step 2.Create Task Configuration Files

In the tasks/lift directory, create the following files:

• nero_joint_pos_env_cfg.py
• nero_lift_env_cfg.py

The nero_joint_pos_env_cfg.py file defines the environment configuration for joint position control, including the controllable joints, the robot end-effector link, and the basic task parameters.

# Copyright (c) 2024-2025, Muammer Bay (LycheeAI), Louis Le Lay
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
#
# Copyright (c) 2022-2025, The Isaac Lab Project Developers.
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

import isaaclab_tasks.manager_based.manipulation.lift.mdp as mdp
from isaaclab.assets import RigidObjectCfg

# from isaaclab.managers NotImplementedError
from isaaclab.sensors.frame_transformer.frame_transformer_cfg import (
    FrameTransformerCfg,
    OffsetCfg,
)
from isaaclab.sim.schemas.schemas_cfg import RigidBodyPropertiesCfg
from isaaclab.sim.spawners.from_files.from_files_cfg import UsdFileCfg
from isaaclab.utils import configclass
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
from isaac_so_arm101.robots import SO_ARM100_CFG, SO_ARM101_CFG  # noqa: F401
# from isaac_so_arm101.tasks.lift.lift_env_cfg import LiftEnvCfg
from isaac_so_arm101.tasks.lift.nero_lift_env_cfg import LiftEnvCfg
from isaaclab.markers.config import FRAME_MARKER_CFG  # isort: skip
# from isaac_so_arm101.robots.piper_description.piper import PIPER_CFG
from isaac_so_arm101.robots.nero_description.nero import NERO_CFG
@configclass
class NeroLiftCubeEnvCfg(LiftEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        # Set so arm as robot
        self.scene.robot = NERO_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")

        # override actions
        self.actions.arm_action = mdp.JointPositionActionCfg(
            asset_name="robot",
            joint_names=["joint1", "joint2", "joint3", "joint4", "joint5", "joint6","joint7" ],
            scale=0.5,
            use_default_offset=True,
        )
        self.actions.gripper_action = mdp.BinaryJointPositionActionCfg(
            asset_name="robot",
            joint_names=["gripper_joint1","gripper_joint2"],
            open_command_expr={"gripper_joint2": -0.05,"gripper_joint1":0.05},
            close_command_expr={"gripper_joint2": -0.001,"gripper_joint1":0.0},
        )   
        # Set the body name for the end effector
        self.commands.object_pose.body_name = ["gripper_base"]

        # Set Cube as object
        self.scene.object = RigidObjectCfg(
            prim_path="{ENV_REGEX_NS}/Object",
            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.2, 0.0, 0.015], rot=[1, 0, 0, 0]),
            spawn=UsdFileCfg(
                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/DexCube/dex_cube_instanceable.usd",
                scale=(0.5, 0.5, 0.5),
                rigid_props=RigidBodyPropertiesCfg(
                    solver_position_iteration_count=16,
                    solver_velocity_iteration_count=1,
                    max_angular_velocity=1000.0,
                    max_linear_velocity=1000.0,
                    max_depenetration_velocity=5.0,
                    disable_gravity=False,
                ),
            ),
        )

        # Listens to the required transforms
        marker_cfg = FRAME_MARKER_CFG.copy()
        marker_cfg.markers["frame"].scale = (0.05, 0.05, 0.05)
        marker_cfg.prim_path = "/Visuals/FrameTransformer"
        self.scene.ee_frame = FrameTransformerCfg(
            prim_path="{ENV_REGEX_NS}/Robot/base_link",
            debug_vis=True,
            visualizer_cfg=marker_cfg,
            target_frames=[
                FrameTransformerCfg.FrameCfg(
                    prim_path="{ENV_REGEX_NS}/Robot/gripper_base",
                    name="end_effector",
                    offset=OffsetCfg(
                        pos=[0.0, 0.0, 0.125],
                    ),
                ),
            ],
        )


@configclass
class NeroLiftCubeEnvCfg_PLAY(NeroLiftCubeEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()
        # make a smaller scene for play
        self.scene.num_envs = 50
        self.scene.env_spacing = 2.5
        # disable randomization for play
        self.observations.policy.enable_corruption = False

The nero_lift_env_cfg.py file provides the base environment configuration for the lifting task. It specifies the task reward, penalties, policy setup, target point position, and block position, which together define the behavior and objective of the environment.

# Copyright (c) 2024-2025, Muammer Bay (LycheeAI), Louis Le Lay
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
#
# Copyright (c) 2022-2025, The Isaac Lab Project Developers.
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

from dataclasses import MISSING

import isaaclab.sim as sim_utils

# from . import mdp
import isaac_so_arm101.tasks.lift.mdp as mdp
from isaaclab.assets import (
    ArticulationCfg,
    AssetBaseCfg,
    DeformableObjectCfg,
    RigidObjectCfg,
)
from isaaclab.envs import ManagerBasedRLEnvCfg
from isaaclab.managers import CurriculumTermCfg as CurrTerm
from isaaclab.managers import EventTermCfg as EventTerm
from isaaclab.managers import ObservationGroupCfg as ObsGroup
from isaaclab.managers import ObservationTermCfg as ObsTerm
from isaaclab.managers import RewardTermCfg as RewTerm
from isaaclab.managers import SceneEntityCfg
from isaaclab.managers import TerminationTermCfg as DoneTerm
from isaaclab.scene import InteractiveSceneCfg
from isaaclab.sensors.frame_transformer.frame_transformer_cfg import FrameTransformerCfg
from isaaclab.sim.spawners.from_files.from_files_cfg import GroundPlaneCfg, UsdFileCfg
from isaaclab.utils import configclass
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR

# from isaaclab.utils.offset import OffsetCfg
# from isaaclab.utils.noise import AdditiveUniformNoiseCfg as Unoise
# from isaaclab.utils.visualizer import FRAME_MARKER_CFG
# from isaaclab.utils.assets import RigidBodyPropertiesCfg


##
# Scene definition
##


@configclass
class ObjectTableSceneCfg(InteractiveSceneCfg):
    """Configuration for the lift scene with a robot and a object.
    This is the abstract base implementation, the exact scene is defined in the derived classes
    which need to set the target object, robot and end-effector frames
    """

    # robots: will be populated by agent env cfg
    robot: ArticulationCfg = MISSING
    # end-effector sensor: will be populated by agent env cfg
    ee_frame: FrameTransformerCfg = MISSING
    # target object: will be populated by agent env cfg
    object: RigidObjectCfg | DeformableObjectCfg = MISSING

    # Table
    table = AssetBaseCfg(
        prim_path="{ENV_REGEX_NS}/Table",
        init_state=AssetBaseCfg.InitialStateCfg(pos=[0.5, 0, 0], rot=[0.707, 0, 0, 0.707]),
        spawn=UsdFileCfg(usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Mounts/SeattleLabTable/table_instanceable.usd"),
    )

    # plane
    plane = AssetBaseCfg(
        prim_path="/World/GroundPlane",
        init_state=AssetBaseCfg.InitialStateCfg(pos=[0, 0, -1.05]),
        spawn=GroundPlaneCfg(),
    )

    # lights
    light = AssetBaseCfg(
        prim_path="/World/light",
        spawn=sim_utils.DomeLightCfg(color=(0.75, 0.75, 0.75), intensity=3000.0),
    )


##
# MDP settings
##


@configclass
class CommandsCfg:
    """Command terms for the MDP."""

    object_pose = mdp.UniformPoseCommandCfg(
        asset_name="robot",
        body_name=MISSING,  # will be set by agent env cfg
        resampling_time_range=(5.0, 5.0),
        debug_vis=True,
        ranges=mdp.UniformPoseCommandCfg.Ranges(
            pos_x=(0.3, 0.35),
            pos_y=(-0.2, 0.2),
            pos_z=(0.2, 0.35),
            roll=(0.0, 0.0),
            pitch=(0.0, 0.0),
            yaw=(0.0, 0.0),
        ),
    )


@configclass
class ActionsCfg:
    """Action specifications for the MDP."""

    # will be set by agent env cfg
    arm_action: mdp.JointPositionActionCfg | mdp.DifferentialInverseKinematicsActionCfg = MISSING
    gripper_action: mdp.BinaryJointPositionActionCfg = MISSING


@configclass
class ObservationsCfg:
    """Observation specifications for the MDP."""

    @configclass
    class PolicyCfg(ObsGroup):
        """Observations for policy group."""

        joint_pos = ObsTerm(func=mdp.joint_pos_rel)
        joint_vel = ObsTerm(func=mdp.joint_vel_rel)
        object_position = ObsTerm(func=mdp.object_position_in_robot_root_frame)
        target_object_position = ObsTerm(func=mdp.generated_commands, params={"command_name": "object_pose"})
        actions = ObsTerm(func=mdp.last_action)

        def __post_init__(self):
            self.enable_corruption = True
            self.concatenate_terms = True

    # observation groups
    policy: PolicyCfg = PolicyCfg()


@configclass
class EventCfg:
    """Configuration for events."""

    reset_all = EventTerm(func=mdp.reset_scene_to_default, mode="reset")

    reset_object_position = EventTerm(
        func=mdp.reset_root_state_uniform,
        mode="reset",
        params={
            "pose_range": {"x": (0.1, 0.2), "y": (-0.1, 0.2), "z": (0.0, 0.0)},
            "velocity_range": {},
            "asset_cfg": SceneEntityCfg("object", body_names="Object"),
        },
    )


@configclass
class RewardsCfg:
    """Reward terms for the MDP."""

    reaching_object = RewTerm(func=mdp.object_ee_distance, params={"std": 0.05}, weight=1.0)

    lifting_object = RewTerm(func=mdp.object_is_lifted, params={"minimal_height": 0.025}, weight=15.0)

    object_goal_tracking = RewTerm(
        func=mdp.object_goal_distance,
        params={"std": 0.3, "minimal_height": 0.025, "command_name": "object_pose"},
        weight=16.0,
    )

    object_goal_tracking_fine_grained = RewTerm(
        func=mdp.object_goal_distance,
        params={"std": 0.05, "minimal_height": 0.025, "command_name": "object_pose"},
        weight=5.0,
    )

    # action penalty
    action_rate = RewTerm(func=mdp.action_rate_l2, weight=-1e-4)

    joint_vel = RewTerm(
        func=mdp.joint_vel_l2,
        weight=-1e-4,
        params={"asset_cfg": SceneEntityCfg("robot")},
    )


@configclass
class TerminationsCfg:
    """Termination terms for the MDP."""

    time_out = DoneTerm(func=mdp.time_out, time_out=True)

    object_dropping = DoneTerm(
        func=mdp.root_height_below_minimum, params={"minimum_height": -0.05, "asset_cfg": SceneEntityCfg("object")}
    )


@configclass
class CurriculumCfg:
    """Curriculum terms for the MDP."""

    action_rate = CurrTerm(
        func=mdp.modify_reward_weight, params={"term_name": "action_rate", "weight": -1e-1, "num_steps": 10000}
    )

    joint_vel = CurrTerm(
        func=mdp.modify_reward_weight, params={"term_name": "joint_vel", "weight": -1e-1, "num_steps": 10000}
    )


##
# Environment configuration
##


@configclass
class LiftEnvCfg(ManagerBasedRLEnvCfg):
    """Configuration for the lifting environment."""

    # Scene settings
    scene: ObjectTableSceneCfg = ObjectTableSceneCfg(num_envs=4096, env_spacing=2.5)
    # Basic settings
    observations: ObservationsCfg = ObservationsCfg()
    actions: ActionsCfg = ActionsCfg()
    commands: CommandsCfg = CommandsCfg()
    # MDP settings
    rewards: RewardsCfg = RewardsCfg()
    terminations: TerminationsCfg = TerminationsCfg()
    events: EventCfg = EventCfg()
    curriculum: CurriculumCfg = CurriculumCfg()

    def __post_init__(self):
        """Post initialization."""
        # general settings
        self.decimation = 2
        self.episode_length_s = 5.0
        self.viewer.eye = (2.5, 2.5, 1.5)
        # simulation settings
        self.sim.dt = 0.01  # 100Hz
        self.sim.render_interval = self.decimation

        self.sim.physx.bounce_threshold_velocity = 0.2
        self.sim.physx.bounce_threshold_velocity = 0.01
        self.sim.physx.gpu_found_lost_aggregate_pairs_capacity = 1024 * 1024 * 4
        self.sim.physx.gpu_total_aggregate_pairs_capacity = 16 * 1024
        self.sim.physx.friction_correlation_distance = 0.00625

Then, the nero reach task needs to be registered in src/isaac_so_arm101/tasks/reach/__init__.py

# Copyright (c) 2024-2025, Muammer Bay (LycheeAI), Louis Le Lay
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
#
# Copyright (c) 2022-2025, The Isaac Lab Project Developers.
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

import gymnasium as gym

from . import agents

##
# Register Gym environments.
##

gym.register(
    id="Isaac-SO-ARM100-Lift-Cube-v0",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    kwargs={
        "env_cfg_entry_point": f"{__name__}.joint_pos_env_cfg:SoArm100LiftCubeEnvCfg",
        "rsl_rl_cfg_entry_point": f"{agents.__name__}.rsl_rl_ppo_cfg:LiftCubePPORunnerCfg",
    },
    disable_env_checker=True,
)

gym.register(
    id="Isaac-SO-ARM100-Lift-Cube-Play-v0",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    kwargs={
        "env_cfg_entry_point": f"{__name__}.joint_pos_env_cfg:SoArm100LiftCubeEnvCfg_PLAY",
        "rsl_rl_cfg_entry_point": f"{agents.__name__}.rsl_rl_ppo_cfg:LiftCubePPORunnerCfg",
    },
    disable_env_checker=True,
)

gym.register(
    id="Isaac-SO-ARM101-Lift-Cube-v0",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    kwargs={
        "env_cfg_entry_point": f"{__name__}.joint_pos_env_cfg:SoArm101LiftCubeEnvCfg",
        "rsl_rl_cfg_entry_point": f"{agents.__name__}.rsl_rl_ppo_cfg:LiftCubePPORunnerCfg",
    },
    disable_env_checker=True,
)

gym.register(
    id="Isaac-SO-ARM101-Lift-Cube-Play-v0",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    kwargs={
        "env_cfg_entry_point": f"{__name__}.joint_pos_env_cfg:SoArm101LiftCubeEnvCfg_PLAY",
        "rsl_rl_cfg_entry_point": f"{agents.__name__}.rsl_rl_ppo_cfg:LiftCubePPORunnerCfg",
    },
    disable_env_checker=True,
)

gym.register(
    id="Isaac-Nero-Lift-Cube-v0",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    kwargs={
        "env_cfg_entry_point": f"{__name__}.nero_joint_pos_env_cfg:NeroLiftCubeEnvCfg",
        "rsl_rl_cfg_entry_point": f"{agents.__name__}.rsl_rl_ppo_cfg:LiftCubePPORunnerCfg", 
        "rl_games_cfg_entry_point": f"{agents.__name__}:rl_games_ppo_cfg.yaml",

    },
    disable_env_checker=True,
)

6. An Isaac Lab Training and Evaluation Pipeline

First, activate the Conda environment:

conda activate env_isaaclab

Then switch to the project directory:

cd isaac_so_arm101

Start training the Isaac-Nero-Lift-Cube-v0 task in headless mode to reduce GPU and display overhead:

uv run train --task Isaac-Nero-Lift-Cube-v0 --headless

If your hardware is strong enough, you can also run training with visualization enabled to observe the learning process directly:

uv run train --task Isaac-Nero-Lift-Cube-v0

This task is trained for 1,000 iterations. Once training is finished, use the following command to evaluate the learned policy:

uv run play --task Isaac-Nero-Lift-Cube-v0

7. Summary

Key outcomes of this work include:

• URDF Model Integration
• Reproducible RL Training Pipeline
• Environment Validation

The workflow can also be extended to more complex manipulation scenarios, including multi-object grasping and obstacle avoidance, with the Nero robotic arm serving as the deployment target for simulation-to-real transfer.

FAQ

What is inverse kinematics in robotics?
Inverse kinematics (IK) calculates the required joint angles for a robotic arm to reach a target position and orientation.

How does ROS2 help with robot arm control?
ROS2 provides communication, motion control, and integration tools for robotic manipulators, including MoveIt2 and RViz.

What is the difference between FK and IK?
Forward kinematics calculates the end-effector position from joint angles, while inverse kinematics calculates joint angles from a target pose.

Why use MoveIt2 for robot arms?
MoveIt2 simplifies robot arm motion planning, collision checking, and trajectory execution in ROS2 environments.

Can this IK solver run on a real robot arm?
Yes. The parametric IK solver can be deployed on physical robot arms after proper calibration and controller integration.

Does NERO Arm support Gazebo simulation?
Yes. The NERO Arm can be simulated in Gazebo and visualized in RViz for testing and development.

Still Have Question?
If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.

3 posts - 2 participants

Read full topic

PR #1419 was merged into introlab/rtabmap_ros last week, which was co-authored by Mathieu Labbe. It adds a TurtleBot3 Gazebo Harmonic demo showing a feedback loop between FusionCore (IMU + wheel UKF) and icp_odometry.

The architecture is worth describing because the feedback direction is non-obvious:

/imu  ──────────────────────┐
/odom (wheel) ──────────────┤──> FusionCore (UKF)
/rtabmap/icp_odometry ──────┘    │
    ^                            │ odom -> base_footprint TF
    │                            │ /fusion/odom
    │        guess_frame_id: odom│
    └──── icp_odometry <─────────┘
             │
             └──> rtabmap SLAM ──> map -> odom TF

FusionCore runs at 100 Hz and owns the odom frame. icp_odometry uses that frame as the initial guess for scan matching via guess_frame_id. A stable initial guess means scan matching succeeds more consistently and with lower residual error. The ICP result feeds back into FusionCore as a second velocity source (encoder2), tightening the UKF state estimate. rtabmap handles loop closure and map correction on top.

Each node tightens the other. Neither is strictly downstream.

To run it on Jazzy:

sudo apt install ros-jazzy-fusioncore-ros ros-jazzy-rtabmap-ros \
                 ros-jazzy-turtlebot3-gazebo ros-jazzy-nav2-bringup
export TURTLEBOT3_MODEL=waffle
ros2 launch rtabmap_demos turtlebot3_sim_fusioncore_icp_demo.launch.py

Full architecture notes and topic/TF table are in the demo README: https://github.com/introlab/rtabmap_ros/tree/ros2/rtabmap_demos/launch/turtlebot3/fusioncore

rtabmap_ros/rtabmap_demos at ros2 · introlab/rtabmap_ros

If you are using rtabmap outdoors with GPS, FusionCore also handles that separately: https://github.com/manankharwar/fusioncore

1 post - 1 participant

Read full topic

Hello ROS Community!

My name is Hariss Abdraman Tahir, and I’m a final-year Computer Engineering student at the University of Maiduguri with a strong passion for robotics, autonomous systems, and embedded hardware.

I recently joined the ROS ecosystem and am currently learning ROS 2 Jazzy Jalisco while building hands-on projects that combine software, hardware, and autonomous systems engineering.

My areas of interest include:

• ROS 2 and robotic software architecture

• Autonomous ground robots and navigation (Nav2)

• Drone development for agricultural and surveillance applications

• Computer vision and perception systems

• Embedded systems and hardware-software integration

• Simulation with Gazebo

• PX4 and MAVSDK for drone autopilot systems

Current Project

For my final-year project, I am designing and building a prototype automated navigation wheelchair for clinical environments. The system integrates autonomous navigation, real-time obstacle avoidance, safety-critical design, and human-machine interaction using a ROS 2-based architecture.

I’m genuinely excited to be part of this community and look forward to learning from experienced developers, contributing to open-source projects, and growing alongside others who share the same passion for robotics.

Any tips, learning resources, or project feedback are always welcome.

Thank you for having me!

5 posts - 4 participants

Read full topic