At Gimlet, we are building a self-optimizing multi-silicon inference cloud. As agents optimize their own inference efficiency, they enter a recursive self-optimization loop where they can optimize for an ever-longer tail of use cases. And in adapting to new hardware, they accumulate knowledge that improves their ability to make future optimizations more effective.
Fred Kjolstad, an Assistant Professor in the Stanford CS Department, Director of the Stanford Center for Portable Accelerated Learning, and compiler researcher, has joined Gimlet Labs as Chief Scientist. Fred will help formulate this vision and develop the research agenda through technical collaboration, advice, and mentorship.
Hardware accelerators are currently swinging towards the specialization end of Makimoto’s semiconductor pendulum, and there is no Moore’s law to halt their swing (see our a16z blog post). There are at least a dozen viable accelerators in today’s market, and in a year they will be replaced with a new generation of hard-to-program hardware. It seems like every microprocessor company, hyperscaler, and frontier lab these days is developing their own AI accelerator, and there are many exciting hardware startups with no end currently in sight.
Models and inference workloads are also likely to turn more diverse from increased specialization. In the last few years, models have consolidated into the general capabilities of giant GPT-style LLMs, but efficiency eventually rears its head. We expect the future to hold room for many models that are optimized for a long tail of diverse workloads. The promise of recursive self-improvement makes this development more likely, as the agents themselves can generate cheap specialized models for different use cases.
Gimlet is in a unique position to tame the wilderness of specialization shown in Figure 1—of hardware, models, and workloads. Our inference cloud is built from the ground up for heterogeneous hardware and supports models expressible in PyTorch. We partner with many hardware vendors and are actively building support for their accelerators.
In the past, the growth of specialized hardware was stunted by the large cost and complexity of building the system software, the kernels, and the compilers required to make it usable. We appear to now find ourselves at a crossroads, where agentic approaches to compilation may finally break through the barrier and make specialized hardware widely usable. At Gimlet, we have long embraced agentic kernel generation through our KForge compiler, and believe the time is right for even more aggressive uses of automatic agentic code generation.
We are now seeking to develop an inference cloud that continuously optimizes itself by rewriting its own source code in response to changing runtime conditions and through zero-day deployment on new hardware. Fast code is specialized code. To automate specialization, the system will continuously profile running models and collect runtime telemetry, feeding this data into agentic systems that automatically re-architect the inference stack, from low-level kernels to high-level routing patterns.
Our vision, depicted in Figure 2, centers on building an agentic auto-research loop that continuously regenerates and optimizes components of the inference stack using runtime data. An inference stack is essentially a virtual machine for executing models. It combines a runtime system that schedules requests, manages parallelism, and orchestrates KV-cache movement with high-performance compute kernels that are pre-generated where possible and JIT-compiled otherwise.
Compute kernels abstract accelerators into functions that operate on tensors, such as FlashAttention and bulk data movement, and are the primary battleground for portability. Different operations, fused combinations of operations, hardware architectures, data types, and data layouts create a combinatorial explosion of specialized kernel variants. There are too many variants for humans to implement and optimize, so we need automation. There are too many variants even to store, so we also need a way to discover which ones are needed as new models run.
Gimlet’s KForge project has successfully built agentic harnesses that let agents generate kernels for new hardware with modest human involvement. We are now building the auto-research loop that will let them operate autonomously. The key to effective agentic kernel generation is specification, instruction, abstraction, and verification. Agents need to understand what they are supposed to do, know how to go about it, have performance models and languages that make them productive, and know when they have achieved their goal. For simple kernels on familiar hardware, they can draw on knowledge encoded in their weights and available on the internet. But effective and efficient exploration requires narrowing their search space and providing them with simple models to guide their search. These are classic research questions in compiler and programming systems.
Moving beyond agentic kernel generation, we also seek to equip agents with the ability to generate components of the inference stack, and eventually the entire stack. That is, instead of building the inference stack, we seek to build agentic meta-systems that generate it. This requires developing specifications, instructions, and tools that let agents generate and verify the different components. Our envisioned StackForge system will also incorporate a harness that supports an auto-research loop in which agents autonomously optimize the inference stack and incorporate new ideas. For example, StackForge could explore new ways to disaggregate models or experiment with new KV cache orchestration policies without requiring engineers to architect the code in advance to support such experiments.
As agents take on more of the implementation work, the focus of our engineering effort shifts toward developing the specifications, knowledge, and tools that enable them to succeed (Figure 3). What makes an engineer effective today is the knowledge, intuition, and judgment they apply throughout the engineering process. If we want agents to take on more of that work, we need to capture that knowledge in a way that is explicit and accessible to them. Some of this involves organizing what we already know about hardware accelerators, optimization, performance modeling, verification, and distributed systems into forms that agents can use. But realizing our vision also requires developing new understanding. Which optimization strategies work best on a new accelerator, and why? What specifications and verification methods let an agent regenerate a component of the inference stack while preserving its required behavior? Answering such questions requires investigation and experimentation, followed by distilling what we learn into clear, reusable forms that agents can apply.
The shift toward knowledge development in engineering, combined with the speed of innovation in inference, has made research an everyday part of how we build products at Gimlet. Ideas originate throughout the technical organization, and anyone can help formulate and pursue a consequential problem aligned with our mission. We organize teams around these problems rather than separating research from engineering. Some teams focus entirely on engineering, others on research, and still others on an evolving combination. Teams form, evolve, and disband as the work requires. A team developing a new capability may begin with exploratory research, then assume development and operational responsibilities as its ideas become useful.
Strong technical organizations are strong mentorship organizations. Gimlet has considerable technical talent capable of carrying out research projects. Experienced researchers increasingly work alongside them, conducting investigations together and engaging deeply with technical content. Through these interactions, strong engineers with less research background learn to formulate consequential problems, assess the technical feasibility of new ideas, and eventually develop the judgment to originate and lead research agendas independently. Meanwhile, research-oriented members of the technical staff learn how to engineer inference systems, making them better equipped to identify and pursue high-impact research.
Our research focuses on the hardest problems that arise from inference serving, and we hold ourselves to a high technical standard. We recognize contributions based on their ultimate product impact, valuing both the research that creates a capability and the work that brings it into reliable use. We are also committed to an open culture, sharing our best ideas and knowledge with the community through blog posts, open-source software, and publications. For us, the test of whether an idea is worth publishing is its quality and potential for broad impact, not concerns about secrecy and competitive moats.
Gimlet has a mature heterogeneous inference stack supporting an increasingly diverse range of hardware. Our next ambition is to build systems that continually improve it: agents that generate kernels and re-engineer the stack for new workloads, hardware, and research ideas. That ambition makes developing specifications, knowledge, and tools a central engineering responsibility. We are thus building research teams into our engineering organization, because developing new understanding and ideas is increasingly becoming part of how we build products. If you're excited about rethinking how inference systems are built and pushing the boundaries of what's possible, join us at Gimlet.