Module 5: Sim-to-Real Transfer & Best Practices

Module 5 focuses on the sim-to-real transfer problem: why systems that work in your digital twin sometimes fail on real hardware, and what you can do about it. You will learn system identification, validation protocols, and capstone‑oriented practices for moving from Gazebo/Isaac Sim into the real lab safely.

5.1 The Sim-to-Real Transfer Challenge

The Reality Gap

Even with careful tuning, simulations are approximations. Differences show up in:

• Visual appearance:
  • Lighting and shadows
  • Texture detail and wear
  • Sensor lens characteristics
• Physics and dynamics:
  • Friction coefficients and contact behavior
  • Joint backlash, compliance, and stiction
  • Cable routing, wiring drag, and unmodeled masses
• Sensors and actuators:
  • Noise levels, biases, drift
  • Latencies and update rates
  • Nonlinearities and saturation

These differences can cause:

• Controllers tuned in sim to oscillate or become unstable on real hardware
• Vision models to fail on real camera feeds while acing synthetic data
• SLAM pipelines to drift more in real environments than in simulation

Types of Transfer Strategies

You will use several strategies, often in combination:

• Dynamics randomization:

  • Vary friction, mass, center of mass, joint damping within reasonable ranges
  • Train controllers to succeed across many plausible dynamics

• Visual randomization:

  • Change materials, lighting, textures, and backgrounds
  • Encourage perception models to focus on geometry and structure, not specific textures

• Noise injection:

  • Add realistic noise and latency to sensors and actuators
  • Ensure algorithms tolerate missing, delayed, or corrupted data

• System identification:

  • Measure real robot parameters and tune simulation to match them more closely

5.2 System Identification: Measuring Reality

What to Identify

For your humanoid, important physical parameters include:

• Mass and inertia of links:
  • Total mass of each limb
  • Inertia tensor around joint axes
• Friction:
  • Surface friction coefficients for feet and hands
  • Joint friction (static and dynamic)
• Damping and compliance:
  • Joint damping (viscous and Coulomb)
  • Flexibility in gearboxes, belts, or cables
• Delays:
  • Sensor latency (camera, IMU, encoders)
  • Actuation delay (command to motor response)

Identification Methods

You will conceptually use:

• Direct measurement:

  • Weigh components, measure dimensions
  • Use CAD models for nominal inertias

• Experimental tests:

  • Free‑fall or pendulum tests for inertia and damping
  • Sliding tests for friction (measure acceleration/decay)
  • Step responses for actuator delays and bandwidth

• Optimization:

  • Define a cost function between real and simulated trajectories
  • Adjust parameters to minimize this error (grid search, gradient‑free methods, Bayesian optimization)

The end goal is not perfection, but a simulation whose predictions are accurate enough for the tasks you care about (e.g., stable walking, precise grasping).

5.3 Validation Protocol: Before Going to Hardware

Staged Deployment

You will follow a conservative, staged process:

1. Simulation‑only validation:
   • Controllers and planners run in Gazebo/Isaac Sim
   • All basic tasks (standing, walking short distances, simple grasps) are stable
2. Static hardware tests:
   • Robot powered on, standing or mounted in a safe configuration
   • Low‑amplitude commands to verify directions and scaling
3. Slow motion tests:
   • Execute trajectories at 10–20% speed
   • Monitor tracking error, joint temperatures, and unexpected vibrations
4. Nominal speed tests:
   • Run at full intended speed in controlled conditions
   • Keep emergency stop readily accessible
5. Edge cases and disturbance tests:
   • Gentle external pushes, minor obstacles, or sensor occlusions
   • Only after nominal behavior is robust

Validation Checklist

Before moving from simulation to hardware, you should be able to answer:

• Physics validation:

  • Does the simulated robot fall or slip under conditions where the real robot does not (or vice versa)?
  • Are contact forces and torques within safe, plausible ranges?

• Control validation:

  • Are tracking errors bounded and acceptable in simulation?
  • Do controllers gracefully handle sensor drops or latency?

• Perception validation:

  • Do models trained on synthetic data perform adequately on real sensor feeds?
  • Are failure cases understood and bounded?

• Safety validation:

  • Are there clear, tested emergency stop behaviors?
  • Are joint limits, torque limits, and velocity limits enforced?

5.4 Capstone Integration: Digital Twin for the Autonomous Humanoid

End-to-End Digital Twin Requirements

For your capstone, your digital twin should support:

• Humanoid model:
  • Accurate URDF in ROS 2
  • SDF/USD representation in Gazebo/Isaac Sim
• Physics simulation:
  • Stable standing and walking
  • Realistic contact with floors and objects
• Sensors:
  • Cameras, depth, LiDAR (as applicable)
  • IMU and joint state outputs
  • Realistic noise and latency
• ROS 2 integration:
  • Same topics/services/actions used for both sim and real robot
  • Configuration that can switch between use_sim_time and real time
• Ground truth:
  • Accessible for validation and training

Transition to Hardware

The ideal sim‑to‑real transition looks like:

• Re‑use the same:
  • ROS 2 nodes (perception, planning, control)
  • Message definitions
  • Launch files (with minor substitutions)
• Swap:
  • Gazebo/Isaac sensor drivers for real camera/LiDAR/IMU drivers
  • Simulated joint controllers for hardware motor controllers
  • use_sim_time → system time

The more your architecture respects this separation from the beginning, the easier your final deployment will be.

5.5 Hands-On Lab: End-to-End Validation Experiment

Scenario

You will design a navigation validation experiment in Gazebo for a humanoid in a simple obstacle course, as a dress rehearsal for your capstone system:

• World: Floor, walls, and several obstacles forming a corridor or maze
• Robot: Humanoid with walking controller and simulated RGB‑D camera
• System: ROS 2 nodes for perception (SLAM), planning (Nav2), and control

Tasks

• Build a Gazebo world:
  • Floor plane and boundary walls
  • Obstacles (boxes, cylinders) in the path
  • Start and goal positions for the robot
• Configure the perception stack:
  • Simulated RGB‑D camera aligned with your target real sensor (e.g., RealSense)
  • SLAM node estimating robot pose
  • Obstacle detection based on depth or point clouds
• Configure planning and control:
  • Nav2 or equivalent planner generating walking paths
  • Walking controller that follows planned paths without falling
• Run 10 repeatable trajectories:
  • Same start and goal
  • Deterministic conditions (no randomization)
  • Log:
    • Planned vs executed trajectories
    • Control tracking error
    • Obstacle avoidance events

Analysis

• Compute:
  • Success rate (reaching the goal without collision)
  • Average and max tracking error along trajectories
  • Planning time per trajectory
• Identify:
  • Failure modes that would be dangerous on real hardware
  • Aspects where simulation is obviously easier (perfect state estimation, ideal friction)
  • Priorities for additional system identification or randomization

Deliverables

• Gazebo world and launch files
• ROS 2 configuration (nodes, topics, parameters) for the full stack
• Recorded rosbag logs for all 10 runs
• A short validation report that:
  • Summarizes metrics
  • Highlights sim‑to‑real risks
  • Proposes concrete next steps before deploying on physical robots

By the end of Module 5—and Chapter 3 as a whole—you will have a capstone‑ready digital twin: not only a visually compelling and physically plausible simulation, but one that is validated, instrumented, and structured to support safe, systematic transfer to real humanoid hardware.