Module 3: Sensor Simulation, Validation & Debugging

Module 3 dives deep into sensor simulation, ground truth generation, and the validation of your digital twin against reality. You will learn how to configure realistic cameras and LiDARs, extract perfect ground truth from simulation, compare it to real data, and systematically debug simulations when they misbehave.

3.1 Deep Dive: Camera and LiDAR Simulation

Camera Models

Realistic camera simulation requires:

• Intrinsics:
  • Focal lengths ( f_x, f_y )
  • Principal point ( c_x, c_y )
  • Aspect ratio and skew
• Distortion:
  • Radial coefficients ( k_1, k_2, k_3 )
  • Tangential coefficients ( p_1, p_2 )
• Sensor noise:
  • Shot noise (photon noise)
  • Read noise (electronics)
  • Quantization noise (bit depth)
• Temporal effects:
  • Motion blur (integration time vs motion)
  • Rolling shutter vs global shutter

In Gazebo and Isaac Sim you will:

• Set camera intrinsics based on your RealSense or similar hardware
• Enable reasonable distortion and noise
• Configure frame rate and exposure time for realism

Depth Cameras (e.g., RealSense)

Depth cameras use:

• Structured light or time‑of‑flight to estimate distance
• Per‑pixel depth measurements with:
  • Distance‑dependent error (increasing with range)
  • Invalid pixels on reflective/transparent surfaces

You will:

• Simulate depth images aligned with RGB
• Model typical depth noise (e.g., Gaussian with variance proportional to distance squared)
• Evaluate:
  • Missing data patterns (holes)
  • Systematic biases (plane fitting against known surfaces)

LiDAR Simulation

LiDAR is modeled as:

• Beams with:
  • Field of view (vertical/horizontal)
  • Angular resolution
  • Max/min range
  • Beam divergence and intensity
• Returns:
  • Single return or multi‑echo (first, strongest, last)
  • Intensity based on material reflectance and incidence angle

You will set up:

• 2D or 3D LiDARs with realistic FOV and resolution
• Noise models that:
  • Drop returns on glass or highly absorptive materials
  • Add jitter to ranges
  • Model adverse conditions (rain, fog, dust) at least qualitatively

3.2 Ground Truth and Evaluation Metrics

What is Ground Truth in Simulation?

Simulation provides access to perfect internal state that you can never fully observe on a real robot:

• Robot base pose (position + orientation)
• Joint positions and velocities
• Object poses (6D) and identities
• Exact depth and segmentation labels
• Camera intrinsics and extrinsics

You will:

• Export ground truth alongside sensor data
• Use it to:
  • Train supervised models
  • Compute evaluation metrics
  • Diagnose failure modes of SLAM and detection algorithms

Core Evaluation Metrics

You will use metrics such as:

• Pose estimation:
  • Translation error (e.g., RMSE in meters)
  • Rotation error (e.g., degrees between quaternions)
• Object detection:
  • Precision, recall, F1 score
  • Average Precision (AP) at different IoU thresholds
• Segmentation:
  • Intersection‑over‑Union (IoU) per class
  • Mean IoU across classes
• SLAM:
  • Absolute Trajectory Error (ATE)
  • Relative Pose Error (RPE)

Simulation ground truth makes these metrics exact, allowing you to quickly test algorithmic changes.

3.3 Validation: Comparing Sim to Real

Validation Pipeline

To validate your digital twin:

1. Collect real data:
   • Run your humanoid (or a sensor rig) through a real environment
   • Record rosbag logs for all relevant topics
2. Recreate scenario in simulation:
   • Model the environment geometry in Gazebo/Isaac Sim (walls, floors, obstacles)
   • Match sensor placement and intrinsics
   • Match lighting conditions as closely as possible
3. Replay identical behaviors:
   • Use the same planned trajectories or teleop inputs
   • Record simulated sensor data and ground truth
4. Compare algorithms:
   • Run SLAM, detection, or control algorithms on both real and simulated data
   • Compare performance metrics side‑by‑side

Matching Simulation to Reality

You will iteratively tune:

• Sensor models:
  • Noise distributions and magnitudes
  • Distortion parameters
  • Latencies and frame rates
• Physics parameters:
  • Friction coefficients
  • Restitution (bounciness)
  • Damping and joint compliance
• Visual fidelity:
  • Materials and textures
  • Light sources

Stopping criteria:

• Key metrics (e.g., SLAM ATE, detection AP) in simulation and reality are within acceptable deltas
• Qualitative comparisons (images, point clouds) look similar to the naked eye

3.4 Debugging Simulations: When Things Go Wrong

Common Failure Modes

Typical issues you will encounter:

• Robot “explodes” or flies away:

  • Often due to bad inertia tensors or overlapping collision geometries
  • Fix by:
    • Checking masses and inertias (no negative or tiny values)
    • Removing self‑intersections in collision meshes
    • Reducing time step or increasing solver iterations

• Joints jitter or oscillate:

  • Usually overly aggressive PID gains or insufficient damping
  • Fix by:
    • Lowering proportional gains
    • Adding joint damping
    • Increasing control frequency carefully

• Robot falls through the floor:

  • Missing or mis‑configured collision on floor or feet
  • Incorrect gravity or physics engine settings

• Sensor data looks wrong or is missing:

  • Mis‑named topics or frames
  • Incorrect noise or intrinsics
  • Bridge plugins not loaded or misconfigured

Debugging Techniques

You will use:

• Visualization:
  • Gazebo GUI and Isaac Sim viewport for contacts and link frames
  • RViz for TF, point clouds, and trajectories
• Logging and plotting:
  • ROS 2 logs (/rosout)
  • ros2 topic echo/hz/bw and rqt_plot
  • Custom logging within plugins and nodes
• Ground truth comparisons:
  • Plot planned vs executed trajectories
  • Compare measured vs true poses or forces

In this module, you should start to treat debugging as data‑driven: collect evidence from simulation and logs, form hypotheses, and test them systematically.

3.5 Advanced: Custom Plugins and Sensor Extensions

Why Custom Plugins?

Built‑in sensors and controllers are sufficient for many tasks, but you may need:

• Specialized LiDAR behavior (e.g., multi‑echo, material‑dependent returns)
• Custom force/torque sensors or contact detectors
• Application‑specific logging or scenario control (e.g., scripted disturbances)

Custom plugins give you fine‑grained control over simulation behavior:

• Gazebo plugins in C++
• Isaac Sim extensions in Python

You will not implement full custom plugins in this chapter, but you will:

• Read and understand basic plugin structures
• Know when to reach for a plugin versus composing existing tools

3.6 Hands-On Lab: Complete Digital Twin Validation

Scenario

You will create and validate a digital twin of a humanoid arm + gripper performing a reach‑and‑grasp task in Gazebo:

• Robot: Upper body/arm URDF from Chapter 2
• Environment: Table and graspable object (e.g., a box or cylinder)
• Sensors:
  • Camera (RGB or RGB‑D) viewing the scene
  • Optional depth or LiDAR for 3D perception
  • Joint state and force/torque readings for feedback

Tasks

• Build a Gazebo world with:
  • Table and object placed in front of the robot
  • Proper physics settings (gravity, friction for table and gripper)
• Integrate the arm and gripper URDF into the world
• Attach sensors and ensure:
  • Joint states are published
  • Camera images show the table and object
  • Force/torque sensor (real or simulated) provides contact information
• Implement a simple reach‑and‑grasp motion:
  • Use ROS 2 to send a planned trajectory to the arm
  • Close the gripper around the object
• Record ground truth:
  • Robot base and end‑effector pose over time
  • Object pose over time
  • Contact events (when gripper touches object)

Validation Metrics

• Grasp success rate: Object ends up stably in the gripper (no slipping)
• Trajectory tracking error: Difference between planned and executed end‑effector pose
• Contact forces: Within safe and plausible ranges
• Sensor quality: Camera images and depth maps enable reliable object detection/pose estimation

Deliverables

• Gazebo world file with the arm, table, and object
• ROS 2 launch files for:
  • Spawning the world and robot
  • Starting controllers and sensor nodes
• Scripts or nodes for:
  • Executing reach‑and‑grasp motion
  • Logging joint states, sensor data, and ground truth
• Validation report with:
  • Metrics and visualizations (plots, screenshots)
  • Discussion of differences you expect when moving to real hardware

By completing Module 3, you will have not just a digital twin, but a validated one—with a documented understanding of when and how it diverges from reality, and a toolkit for debugging those gaps.