Glacier.Inference delivers substantial efficiency gains over a Go+C++ CUDA-based local daemon across several models and hardware, cutting binary size, speeding cold-starts, and reducing sampling overhead by about 250x, while remaining competitive with CUDA-based pipelines.

Direct3D 12 compute targets integrated GPUs to avoid display-driver resets and optimize memory via UMA, featuring 32-token GEMM tiling and register-based tiling to keep dequantized vectors in fast memory.

The project eliminates CUDA runtime bloat by launching kernels directly through nvcuda.dll with P/Invoke, packaging everything into a self-contained ~15 MB native AOT executable that replaces a multigigabyte toolchain.

The takeaway is that managed languages can handle low-level compute workloads by integrating zero-copy weight access, Direct3D 12 compute, direct P/Invoke, and on-device reductions to deliver bare-metal throughput with a portable single binary.

On-device in-VRAM argmax reduction is fused with a 512-thread warp-shuffle kernel to minimize PCIe transfers, cutting per-token data movement from 608 KB to 4 bytes.

Glacier.Inference presents a pure C# .NET 10 bare-metal approach to local LLM inference, avoiding CUDA toolkits and native C++ DLLs by using direct driver interfaces and memory-mapped GGUF models.

Core components include a memory-mapped zero-copy GGUF reader, a bare-metal SASS engine using direct nvcuda.dll, Direct3D 12 compute via HLSL Wave32, and an adaptive unmanaged KV-cache with FP16/FP8 dynamics.

Speculative decoding (VerifyBatch) reduces memory bandwidth by batch-verifying candidate tokens in one pass, achieving higher throughput (72–104+ tokens/sec on RTX 4060 laptop) than serial reads.

The article provides a GitHub repository with Glacier.Inference code, benchmarks, and standalone binaries.

Results show strong performance: for a 4.68 GB model, ~41.92 tokens/sec serial and ~72.5–104.8 tokens/sec with speculative generation on RTX 4060, and ~21.68 tokens/sec on AMD 890M for a 30B MoE model, roughly 24x outperforming AVX-512 CPU, with a binary footprint around 15 MB versus a multi-GB toolchain.