Showing Posts From

Swarm robotics

The conveyor belt is dead. Long live the swarm. For decades, industrial logistics relied on fixed infrastructure—conveyors, AS/RS cranes, and predetermined guidepaths—to enforce order on chaos. That paradigm hits a hard ceiling the moment SKU proliferation explodes or a black-friday surge demands instantaneous reconfiguration. Enter the agentic swarm: heterogeneous fleets of mobile manipulators (MoMas) that perceive, decide, and act without a central puppet master. But decentralization introduces a brutal constraint: how do you maintain global optimality—throughput, collision avoidance, load balancing—when every agent runs a local policy trained on a simulation that never perfectly matches the dusty, dynamic reality of a 100,000 sq ft fulfillment center? The answer lies not in better planners, but in better adaptation. Recent breakthroughs in real-to-sim-to-real pipelines and noise-space policy steering are finally giving swarms the shared world model and the local plasticity they need to operate at scale.Digital Twin Fidelity as the Swarm's Shared Consciousness The foundational bottleneck for any decentralized system is state estimation. If Agent A believes the pallet rack is at coordinate (x, y) but Agent B—relying on a stale SLAM map—thinks it's at (x+0.3m, y), the resulting deadlock isn't a planning failure; it's a consensus failure. Traditional LiDAR-based SLAM struggles with the textureless, repetitive corridors of modern warehouses and the dynamic occlusion caused by other robots. This is where the Splat2Act pipeline fundamentally shifts the architecture. By leveraging 3D Gaussian Splatting (3DGS) reconstructed from a simple handheld phone scan, it generates a photorealistic, metrically accurate digital twin of the entire navigable area—median 104.87 m² in the study, but scalable to building-scale environments—complete with collision geometry and, crucially, semantic receptacle discovery. For a swarm, this digital twin serves as the "ground truth" simulation substrate. Instead of each robot maintaining a local, noisy map, the fleet operates on a unified, high-fidelity simulation instance hosted on the edge. The Splat2Act pipeline automatically discovers 852 receptacles (tabletop and non-tabletop) across 101 scenes, constructing feasible pickup tasks without human annotation. This means the swarm's centralized training environment—the "gym" where decentralized policies are forged—is geometrically identical to the deployment reality. The pipeline generates 68,000+ demonstrations via a privileged simulation oracle. This massive dataset allows us to pre-train a Vision-Language-Action (VLA) model, specifically π₀.₅, not just on generic manipulation, but on the specific geometry of this warehouse's mezzanine, this facility's narrow aisles, and this dock's loading bays. The result? A policy that enters the real world with a 66.7% simulated success rate on target tasks, tracking closely to 63.9% on a real Stretch 3 robot. That <3% sim-to-real gap is the trust margin required to decentralize execution. Decentralized Policy Adaptation via Noise Space Learning A perfect digital twin solves the modeling problem, but logistics is stochastic. A forklift drops a pallet; a human blocks an aisle; a conveyor jams. A static policy, even one trained on a perfect twin, fails here. Retraining the VLA backbone (billions of parameters) on-edge is computationally impossible. Freezing the backbone and fine-tuning adapters (LoRA) is faster but still requires backward passes and gradient synchronization across the fleet—violating decentralization. Noise Space Learning (NSL) provides the elegant escape hatch. The core insight: Flow-matching VLAs generate actions by denoising a latent noise vector $z \sim \mathcal{N}(0, I)$. Instead of adapting the denoising network $\theta$, NSL learns a context-conditioned distribution $q_\phi(z | c)$ over the input noise space, keeping $\theta$ frozen. The policy becomes $\pi(a|o) = \text{Denoise}\theta(z \sim q\phi(\cdot|o), o)$. Adaptation reduces to learning a lightweight context encoder $\phi$ that biases the noise distribution toward successful trajectories for the current observation $o$. This is mathematically trivial to decentralize. Each robot runs its own $\phi$ (a tiny MLP or Transformer), updated via PPO, SAC, or AWR on its local replay buffer. No gradient sharing, no parameter server, no consensus protocol on weights. The paper demonstrates this on LIBERO and real-robot tasks under strict interaction and compute budgets. For a swarm, this means Robot 7, stuck behind a fallen box, adapts its noise distribution to "push slightly left then rotate" within 50 local interactions, while Robot 42, cruising an open aisle, keeps its noise distribution near the prior $\mathcal{N}(0, I)$. They diverge behaviorally while sharing the same frozen backbone $\theta$—the ultimate parameter-efficient, communication-free adaptation. The information-theoretic analysis in the paper confirms NSL minimizes the KL divergence to the optimal adapted policy while constraining the mutual information $I(z; c)$, preventing catastrophic forgetting of the generalist priors encoded in $\theta$. Sim-to-Real Transfer: Closing the Reality Gap for Fleet Deployment The "Sim-to-Real Gap" is usually treated as a domain randomization problem: randomize friction, lighting, sensor noise, and hope the policy covers the real world. Splat2Act and NSL attack this differently: Geometry First, Dynamics Second. Splat2Act eliminates geometric domain gap. The collision mesh in the simulator is the mesh extracted from the 3DGS reconstruction of the real building. The receptacles are the actual shelves. The visual rendering is photorealistic because it is the real view (novel view synthesis). This removes the need for massive visual domain randomization. The remaining gap is dynamics (actuator latency, wheel slip, payload inertia). This is where NSL shines. Because NSL adapts in the noise space of a flow-matching policy, it effectively performs residual physics learning. The pretrained VLA $\theta$ provides the "nominal" kinematic trajectory (move to pose, grasp). The adapted noise distribution $q_\phi$ injects the corrective stochasticity needed to handle the specific wheel slip of Robot 12 carrying a 15kg tote on a polished concrete floor. The empirical validation is striking. The Splat2Act policy on Stretch 3 achieves 63.9% real-world success on a 12-task subset, closely tracking the 66.7% sim success. NSL variants (PPO-NSL, SAC-NSL, AWR-NSL) show consistent improvement over frozen baselines on real-robot manipulation under compute constraints. For logistics, this translates to: Deploy the frozen VLA backbone fleet-wide. Let each agent run NSL-AWR (Advantage Weighted Regression) locally for 100 steps during a "calibration shift" using only proprioception and wrist-cam data. The fleet homogenizes capability without homogenizing parameters. This is the architectural pattern for 2026+ logistics: Centralized Geometry Acquisition (Splat2Act) -> Centralized Backbone Training -> Decentralized Noise Adaptation (NSL). Heterogeneous Fleet Coordination: From Manipulation Primitives to Logistics Flows A warehouse swarm isn't homogeneous. You have AMRs (Autonomous Mobile Robots) for transport, MoMas for picking, case-handling robots for depalletizing, and drones for cycle counting. They share the same digital twin (Splat2Act output) but have vastly different action spaces. The VLA paradigm—specifically the π₀.₅ architecture used in the paper—handles this via language conditioning. The policy $\pi(a | o, l)$ takes a language instruction $l$ ("transport tote T-404 to station B", "pick SKU-X from bin 3, place in tote T-404"). Decentralized pathfinding in this context isn't A* on a grid. It's Next-Best-Action Prediction conditioned on Global Intent. The "Global Intent" is broadcast as a lightweight token stream (e.g., MQTT topics: fleet/task/allocation/robot_12 -> "navigate:zone_4; manipulate:pick_sku_X"). The local VLA+NSL policy executes the how. The pathfinding is the manipulation policy. When the MoMa approaches the shelf, the "path" becomes a 6-DoF trajectory to the grasp pose. When the AMR approaches the congestion zone, the "path" becomes a velocity command sequence avoiding Robot 42. This collapses the stack: Perception -> VLA (Backbone + NSL) -> Low-level Control. No separate Global Planner -> Local Planner -> Controller hierarchy. The "planner" is the denoising process of the flow-matching model. The "local planner" is the noise adaptation. The Splat2Act pipeline's automatic receptacle discovery (7 receptacles median per scene) provides the semantic keypoints ("bin_3_top_left", "conveyor_entry_2") that ground the language tokens in the 3DGS geometry. The swarm coordinates via Stigmergy on the Digital Twin: Robot 12 updates the twin's semantic state ("bin_3_top_left: occupied_by_tote_T404"). Robot 42 queries the twin ("find nearest empty bin for SKU-Y") and receives a geometrically valid target. The twin is the pheromone trail; the 3DGS is the substrate. Scalable Orchestration: Kubernetes at the Edge for Robot Swarms Running VLA inference (even quantized π₀.₅) + NSL adaptation + 3DGS rendering (for local view synthesis/verification) requires serious compute. A typical MoMa might carry an NVIDIA Jetson Orin AGX (275 TOPS) or an industrial PC with an RTX 6000 Ada. Managing this fleet as "robots" is the wrong abstraction. They are Edge Compute Nodes with Actuators. The orchestration layer must be Kubernetes (k3s/k0s) running on the robot, managed by a GitOps flux controller from the cloud/edge gateway. The container image contains: The Frozen VLA Backbone (ONNX/TensorRT engine), The NSL Adapter Head (PyTorch, <10MB), The 3DGS Renderer (CUDA kernels for splatting), The ROS 2 Humble Bridge (cyclonedds). The ConfigMap mounts the Scene-Specific Assets: The 3DGS ply/ckpt file (geometry + appearance), The Receptacle JSON (semantic map from Splat2Act), The NSL Adapter Weights (updated nightly via OTA from the training pipeline). # docker-compose.edge.yml - MoMa Edge Runtime Stack version: '3.8' services: vla-inference: image: registry.internal/logistics/vla-pi05-trt:2026.10 runtime: nvidia deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu, compute, utility] environment: - MODEL_PATH=/models/pi05_frozen.engine - ADAPTER_PATH=/adapters/nsl_head_latest.pt - SCENE_GAUSSIANS=/assets/warehouse_04_gaussians.ply - RECEPTACLE_MAP=/assets/warehouse_04_receptacles.json volumes: - model_store:/models:ro - adapter_store:/adapters:rw # NSL writes updated weights here - scene_assets:/assets:ro ipc: host network_mode: host privileged: true # Required for GPU DMA / ROS 2 realtime healthcheck: test: ["CMD", "grpc_health_probe", "-addr=localhost:50051"] interval: 10s timeout: 3s retries: 3 nsl-trainer: image: registry.internal/logistics/nsl-trainer:2026.10 runtime: nvidia depends_on: vla-inference: condition: service_healthy volumes: - adapter_store:/adapters:rw - replay_buffer:/data/replay:rw environment: - ALGO=AWR # Advantage Weighted Regression for stability - BATCH_SIZE=32 - UPDATE_FREQ=50 # Steps - KL_B### Related Insights- [Cappadocia Unveiled: Soaring Above Fairy Chimneys and Sleeping in Stone Dreams](/blog/cappadocia-unveiled-soaring-above-fairy-chimneys-and-sleeping-in-stone-dreams) - [Captive Traitor King (Braided Fate, #2) Book Review and Summary](/blog/captive-traitor-king-braided-fate-2-brigid-kemmerer) - [**Dark Sky** Book Review and Summary](/blog/dark-sky)

We use cookies (including third-party cookies) to provide you with the best possible experience and to serve personalized ads. By clicking "Accept", you consent to the use of all cookies. Visit our Privacy Policy for more information.