Gimlet Labs

Posts tagged with Efficiency

Recursive Self-Optimization at Gimlet

At Gimlet, we are building a self-optimizing multi-silicon inference cloud. As agents optimize their own inference efficiency, they enter a recursive self-optimization loop where they can optimize for an ever-longer tail of use cases. And in adapting to new hardware, they accumulate knowledge that improves their ability to make future optimizations more effective.

Splitting LLM inference across different hardware platforms

Separating prefill and decode stages of LLM inference improves token throughput because their resource needs differ. Although most deployments use NVIDIA hardware for both stages, multivendor disaggregation can actually improve efficiency while maintaining SLAs. Based on our models using NVIDIA B200s and Intel Gaudi 3, common workloads can see 1.7X TCO improvement compared to single-vendor disaggregation.