Topic 2: Shared Perception, Mapping & World Models

Topic 2 shows how multiple robots can build and maintain a shared understanding of the world. Instead of each robot holding a private map and perception stack, you will design systems where agents exchange observations, merge maps, and synchronize a global world model that supports fleet-level planning and coordination.

2.1 From Local Maps to Shared World Models

In Chapter 4, you built perception and SLAM pipelines for individual robots. Each robot maintained:

• Local sensor streams (RGB, depth, LiDAR, IMU)
• A local pose estimate
• A local map (e.g., occupancy grid, point cloud, or 3D reconstruction)

For multi-robot systems, we care about:

• Consistency across robots
  • If two robots see the same hallway, they should agree on its shape and key landmarks.
• Coverage
  • Multiple robots can map different regions in parallel and then stitch their maps.
• Shared state
  • Planners and commanders need a global view of obstacles, free space, and relevant objects.

The goal of this topic is to move from “my robot’s map” to our shared map, and to understand the trade-offs in how this is built and maintained.

2.2 Multi-Robot SLAM: Parallel Mapping and Map Merging

Multi-robot SLAM extends single-robot SLAM by:

• Running SLAM on multiple robots in parallel
• Exchanging:
  • Poses and trajectories
  • Keyframes or submaps
  • Loop closure candidates
• Producing:
  • A global map that covers a larger area than any single robot

Common approaches:

• Centralized mapping server
  • Each robot runs a local front-end (feature extraction, scan matching).
  • A central back-end receives submaps and pose graphs and optimizes a global pose graph.
  • Robots periodically receive updated global poses and maps.
• Distributed/peer-to-peer mapping
  • Robots share map fragments and constraints directly.
  • Consensus or distributed optimization techniques reconcile differences.

Design questions:

• How often do robots exchange map data?
• Do they send raw sensor data, downsampled point clouds, or compressed submaps?
• How do you detect when two maps overlap and should be merged?

For this course, you will focus on conceptual design and simulation prototypes, rather than full research-grade multi-robot SLAM, but you should understand the typical data flows and constraints.

2.3 Shared Sensor Fusion and Target Sharing

Even when robots do not fully merge maps, they can still share perception outputs:

• Object detections (class, pose, confidence)
• Landmarks and features
• Regions of interest (ROIs) or “hotspots”

Shared perception enables:

• Improved situational awareness
  • A robot entering a room already knows about previously detected obstacles or objects.
• Distributed target tracking
  • Multiple robots can localize the same moving object from different viewpoints.
• Task guidance
  • One robot can tell another, “There is an unvisited area here,” or “This item was seen on shelf B.”

Key design aspects:

• Message schemas
  • Standardize how detections and landmarks are represented (e.g., ROS 2 messages with pose, covariance, and labels).
• Confidence handling
  • Combine or compare detections using confidence scores and consistency checks.
• Latency and refresh rates
  • Decide how often to broadcast world model updates; too frequent can overload the network, too sparse can make data stale.

In practice, many fleets maintain a shared state database (e.g., a central world-state server) fed by robot perception nodes.

2.4 Digital Twins for Fleets

Digital twins (Chapter 3) can represent entire fleets, not just individual robots:

• Simulated world contains:
  • Multiple robot instances
  • Shared environment and objects
  • Global state that all simulated robots interact with
• The digital twin becomes the ground truth for:
  • Where robots are
  • What they see
  • How the environment changes

Fleet-level digital twins support:

• End-to-end testing of coordination strategies
  • Map merging, task allocation, and conflict resolution can be evaluated safely.
• Visualization and debugging
  • Developers can observe all robots’ paths, sensor FOVs, and world model updates in one place.
• Training and synthetic data
  • Generate multi-view, multi-robot datasets for perception and mapping.

Your goal is to think of the simulator not as a toy, but as a shared virtual world where fleet-level behaviors can be iterated on before hardware deployment.

2.5 Consistency, Conflicts, and Partial Knowledge

In real systems, shared world models are never perfectly consistent:

• Robots may see the same area at different times under different conditions.
• Occlusions and sensor noise can produce conflicting observations.
• Network delays can cause outdated information to persist longer than expected.

Common issues:

• Overlapping but misaligned maps
  • Small pose errors can cause double walls or mis-registered obstacles.
• Stale information
  • A pallet moved minutes ago still appears in another robot’s map.
• Contradictory detections
  • One robot sees an object; another fails to see it in the same place.

Strategies to manage this:

• Track timestamps and frame IDs for all data.
• Use confidence and recency weighting for world model updates.
• Prefer local sensing for safety-critical decisions (e.g., collision avoidance) even if global maps are imperfect.
• Periodically recompute or compact maps to resolve long-lived inconsistencies.

The key insight: shared world models are approximations, and fleet behaviors must tolerate partial, noisy, and evolving information.

2.6 How Shared World Models Support Fleet Coordination

Shared maps and world models are the substrate for:

• Multi-robot path planning
  • Avoiding collisions and congestion in shared corridors.
  • Coordinating use of narrow passages and shared resources.
• Task allocation and coverage
  • Assigning robots to unexplored or high-priority regions.
  • Ensuring no areas are accidentally ignored or double-covered.
• Commander-level reasoning
  • LLM-based or rule-based commanders query the world model to:
    • Understand which tasks are done or pending.
    • Determine which robots are best placed to take new tasks.
    • Evaluate mission progress and adjust plans.

In later topics, you will see how world models, planners, and LLM Commanders interact to create coherent fleet behaviors that go beyond what any single robot could achieve alone.