Databricks’ AI Research team leverages Databricks AI Runtime (AIR) to accelerate experimentation, scale training workloads, and support agent-led research across multiple projects.

Readers are guided to start using AIR via the AI Runtime documentation, with examples of ongoing work in faster search, reinforcement learning, and structured document understanding.

AIR offers built-in GPU resiliency, including preflight checks and runtime monitoring to detect issues and cordon problematic nodes, reducing wasted GPU hours on long-running training like KARL with models over 100 billion parameters.

MLflow integration with AIR automates experiment tracking, enabling monitoring of status, logs, and metrics and providing agents with necessary results for decision-making.

Agent-led (autoresearch) training has increased experimentation volume eightfold, as agents propose changes, launch trials in parallel, evaluate results, and update plans accordingly.

Unity Catalog and FUSE mounting provide governed, seamless access to research data across workspaces, preserving governance while enabling distributed training workflows.

A continuous agent-driven cycle—Propose, Launch trials, Learn, Update the plan—drives faster iteration through AIR’s quick job launches, reliability, and scalable execution to support larger models.

AIR includes workload prioritization and preemption to allocate GPU capacity across projects, enabling high-priority experiments while using spare capacity for exploratory work.

The AIR CLI and custom containers allow fast, programmatic workload launches, consistent environments, and rapid feedback loops, with Docker images registered in Databricks Artifact Registry for reuse across experiments.