ZenteiQ deployed Google Kubernetes Engine (GKE) and GKE Inference Gateway powered by llm-d to scale data generation, achieving a 4.07x increase in inference throughput and increasing daily tokens generated per 1,000 accelerators from 1.47 billion to 5.97 billion.

The data-generation bottlenecks stemmed from uneven engine queues and redundant prefix computation in a round-robin routing setup as model sizes grew to 9B- and 35B-parameter schedules.

With the improved throughput, datasets and pre-training runs stay on schedule, enabling ZenteiQ to scale from 9B to 35B parameter models and extend the approach into post-training RL and automated numerical solver evaluation loops.

Future steps include applying the same GKE Inference Gateway and llm-d setup to broader workloads and exploring the open-source llm-d project for distributed inference.

GKE Inference Gateway, with llm-d, delivers load- and cache-aware routing by monitoring real-time vLLM queue depths and KV cache utilization, routing requests to available nodes and to nodes with warm caches to reuse prompt prefixes.

This routing shift produced tangible gains: about 4x higher fleet throughput (from 16,964 to 69,114 tokens per second per 1,000 accelerators), plus 3.5x–5.5x higher gains across model pools and a 14-percentage-point increase in prefix-cache hit rate.

ZenteiQ is an IISc-spun AI startup and IndiaAI Mission participant that built a large-scale batch synthetic data-generation pipeline to pre-train scientific foundation models capable of understanding physical laws and numerical solvers.