NVIDIA expands Cosmos with a modular Cosmos Framework for world model training, optimization, and inference, designed to support multiple models and Hugging Face compatibility.

The Cosmos Framework includes agent skills and recipes for world model training, multimodal data processing, optimization, and inference, with integration across Cosmos 3 and compatibility with Alpamayo, GR00T, Qwen, pi0.5, and Hugging Face models.

Reference runs demonstrate multi-view 720p video outputs and LiDAR sweeps, with guidance on input data, sample outputs, and verification of World Scenario Map alignment across views.

NIM supports live streaming VLM inference, achieving sub-second latency reductions, automatic deployment configuration, performance validation, licensing, and quarterly updates.

Concrete deployment steps include selecting model and config, installing NVIDIA drivers and Docker, logging into NGC, launching the Cosmos NIM container, checking readiness and metadata, and choosing latency versus throughput profiles based on the use case, with sample commands and environment variables.

Additional resources include links to technical reports, inference guides, and livestream events, along with getting started notes and recommended workflows.

The framework supports online and offline reinforcement learning pathways (DPO, GRPO, PPO, OPD, DiffusionNFT) and connects simulators for experience collection.

Reinforcement learning capabilities cover online/offline paths with DPO, GRPO, PPO, OPD, and DiffusionNFT, with rollout, reward, and training able to run on shared GPUs or separate pools, and simulator integration for data collection.

Cosmos simulation workflows cover autonomous vehicles, multi-camera driving, surgical policy evaluation, and Nano simulation recipes (Cosmos3-Nano-Transfer-Auto, Cosmos3-Nano-Sim-Auto, Cosmos3-Nano-Sim-Transfer, Cosmos3-Nano-Sim-Bimanual) for synchronized camera and LiDAR data with prompts and controls like World Scenario Map, HD-map controls, and weather/lighting variation.

Simulation recipes address autonomous vehicles, robotics, and surgery, including Cosmos3-Nano-Transfer for offline synthetic data and Cosmos3-Nano-Sim-Auto for closed-loop driving with incremental observations.

NVIDIA plans a livestream on November 5 to demonstrate Cosmos Framework usage and notes Cosmos NIM deployment on more than twenty GPUs.

A GPU memory usage table outlines reasoning and generation requirements for Cosmos3-Edge, Cosmos3-Nano, and Cosmos3-Super across BF16 and FP8 formats.