Disaggregating the Disaggregated Model
Fast accelerators improve speculative decoding interactivity. Separating draft prefill from draft decode preserves that speed while opening up gains in time to first token and energy efficiency.
Fast accelerators improve speculative decoding interactivity. Separating draft prefill from draft decode preserves that speed while opening up gains in time to first token and energy efficiency.
At Gimlet, we are building a self-optimizing multi-silicon inference cloud. As agents optimize their own inference efficiency, they enter a recursive self-optimization loop where they can optimize for an ever-longer tail of use cases. And in adapting to new hardware, they accumulate knowledge that improves their ability to make future optimizations more effective.
Agentic workloads repeatedly re-enter model inference over growing state. We characterize how trajectories, KV-cache retention, and tool use shape serving demand.
AI workloads are shifting from simple LLM inference to complex, multi-model workflows. To run them efficiently at scale, we need a system that can dynamically decompose workloads, plan and schedule them, and map execution to the right hardware.
Separating prefill and decode stages of LLM inference improves token throughput because their resource needs differ. Although most deployments use NVIDIA hardware for both stages, multivendor disaggregation can actually improve efficiency while maintaining SLAs. Based on our models using NVIDIA B200s and Intel Gaudi 3, common workloads can see 1.7X TCO improvement compared to single-vendor disaggregation.