tl;dr: AI-generated kernels are often evaluated by their speedup over a reference implementation. But a large speedup can come from a weak baseline, a relaxed numerical contract, or overfitting to one input shape. To generate production-ready kernels, agentic kernel generation systems must optimize not just for speedup but also for capability relative to the hardware limit, correctness under the intended numerical contract, and coverage across deployment shapes.
At Gimlet, we are building full-stack recursive self-optimization into our cloud so performance can improve automatically without human intervention. Agentic kernel generation is one component of this broader system, applying agents to optimize low-level kernels that map model operations onto target hardware.
Fig. 1. Gimlet's recursive self-optimization vision uses runtime telemetry and agents to improve components across the inference stack. Within this system, agentic kernel generation uses the optimization loop to generate and refine kernels that execute model operations on target hardware.
Large language model (LLM) agents have become increasingly effective at writing and optimizing kernels for GPUs and other AI accelerators. Given access to a compiler, profiler, and target device, an agent can iterate through implementations, measure each candidate, and discover optimizations that would otherwise require substantial expert effort.
This post distills lessons from applying agents to kernel optimization and examines what production readiness requires beyond speedup. Progress in agentic kernel generation is usually measured by speedup over a reference implementation. This metric is intuitive, but it is not enough to establish that a generated kernel is genuinely better or ready for production.
A speedup is only relative to the baseline chosen for comparison. A candidate can also appear fast by quietly changing precision, exploiting a narrow test distribution, or specializing for the exact shape used during evaluation. These benchmark-specific gains may look compelling when evaluation records a single number, but they may not survive in a production inference system, where kernels must preserve a numerical contract and perform across changing batch sizes, sequence lengths, and hardware targets.
Our experience building agentic kernel generation systems and deploying the resulting kernels has led us to assess progress along three dimensions beyond raw speedup: capability, correctness, and coverage.
Capability: How close is the kernel to the hardware limit?
A 2x speedup may appear substantial, but its significance depends heavily on how well the reference implementation is optimized. It does not reveal whether the generated kernel reaches 10% or 90% of the target hardware's performance ceiling. A large improvement over a weak baseline can still leave most of the available performance unused, while a modest improvement over a strong baseline may already be close to the hardware limit.
Capability should therefore be measured against speed-of-light (SOL), a workload- and hardware-specific estimate of the best achievable performance. Depending on the operation, this ceiling may be set by compute throughput, memory bandwidth, communication, or another system constraint. Comparing measured performance with that ceiling shows both what the kernel achieves and how much opportunity remains. Recent work has applied this framing both to benchmarking kernels against hardware limits and to using SOL guidance to steer and budget agentic optimization.
Results from representative runs on KernelBench Level 2, conducted earlier in our work, further illustrate the difference between relative speedup and hardware-limit attainment. One fused Conv3D kernel ran 4.6x faster than its matched baseline but reached only 4% of the estimated hardware limit. Other generated kernels showed smaller speedups, around 1.8x to 2.2x, while reaching as much as 70% of SOL. While absolute performance numbers change as kernel-generation systems evolve, ranking kernels only by speedup can obscure which workloads are already well optimized and which still leave substantial hardware capability unused.
Hardware-limit guidance can also improve the generation process itself. If an agent stops after exceeding a baseline, it may settle for an implementation that is fast only in relative terms. When the system can see that a candidate remains far from the hardware ceiling, it has a reason to continue exploring. Hardware-specific skills, worked examples, and longer-horizon planning can then direct that search toward the resources that actually limit the kernel. For example, a gpt-oss attention kernel was already 2.4x faster than a tuned Triton reference but remained well below SOL. Adding hardware-specific guidance derived from NVIDIA Blackwell documentation reduced kernel time by another 22%, bringing the total speedup to about 3x. The guidance led the agent to use a smaller KV tile after identifying the exponential unit, rather than the tensor cores, as the limiting resource.
Fig. 2. Large speedups do not necessarily indicate high attainment of the estimated hardware limit. Bars show the fraction of that limit reached by the torch.compile (bf16) baseline and the generated kernel; bold labels show speedup over the baseline.
This perspective also changes how optimization effort should be allocated. Production systems run models, not isolated kernels, and per-kernel gains do not automatically compose into end-to-end improvements. Doubling the speed of an operator that accounts for 5% of runtime reduces end-to-end runtime by only about 2.5%, even before launch overhead and fusion effects. Capturing the operators and shapes issued by a real inference engine allows the generation system to focus on the kernels that dominate end-to-end time. Using this approach, the PagedAttention kernel generated by our system increased prefill throughput for gpt-oss-120b on an AMD MI350X from 20,000 to 27,000 tokens per second, a 35% end-to-end improvement over an already optimized vLLM configuration.
The capability question extends beyond "How much faster is this kernel than the baseline?" It is "How much of the estimated performance limit does it attain, and does improving it materially accelerate the deployed workload?"
Correctness: Does the kernel preserve the production contract?
An optimization agent maximizes what the evaluation rewards. If acceptance is based on a limited set of inputs and a loose output tolerance, the agent may discover a faster computation that passes those checks without preserving the intended operation.
Precision is one common source of this gap. A candidate may reduce accumulation precision or substitute a lower-precision path while remaining within the benchmark's tolerance. The reported speedup then reflects a changed numerical contract rather than a better implementation of the same contract.
A narrow input range can conceal an even more fundamental correctness failure. In one run, an agent generated a HardTanh kernel that simply returned its input and reported a 71,000x speedup. The benchmark sampled values only from the interval where HardTanh behaves like the identity function, so the shortcut passed every provided test while being incorrect outside that range. In another case, an agent removed the running-maximum calculation required for numerically stable softmax and used a fixed value for the maximum. The output remained within the tested tolerance, but the kernel no longer preserved the expected numerical behavior. These failures are systematic and follow directly from an agent's ability to exploit any gap between the intended operation and the tests used to evaluate it.
Correctness must therefore be part of the generation loop rather than a final filter applied after optimization. The system should:
- encode input and output data types, accumulation precision, and other numerical requirements as an explicit contract;
- reject candidates that obtain speed by weakening that contract;
- generate additional cases that probe untested values, precisions, and ranges;
- validate important kernels in the context of the end-to-end application; and
- retain enough information about each candidate to audit how its result was produced.
Matched baselines are equally important. Across six evaluated problems, bf16-compliant generated kernels showed a 10.3x geometric-mean speedup over the original fp32 baselines, but only a 2.24x speedup over bf16 baselines with aligned inputs and explicit output checks. The kernels were still faster, but most of the larger headline number came from comparing different numerical configurations.
The correctness question extends beyond "Did this kernel pass the benchmark?" It is "Did it preserve the complete contract required in production?"
Coverage: Does performance survive beyond the tuned shape?
An operator in production rarely runs at one fixed shape. Prefill and decode exercise different dimensions. Batch sizes and sequence lengths vary over time. Packed requests may be regular in a benchmark but ragged in deployment. A kernel optimized for one configuration can perform poorly, or fail entirely, when any of these conditions change.
This creates a coverage cliff. A generated kernel looks strong at the point used for tuning but loses its advantage away from that point. Hardcoded dimensions, assumptions about regular inputs, and specializations that do not dispatch safely are all ways an agent can overfit to a benchmark while producing little production value.
Specialization itself is not the problem because a deployed operator can select among several specialized kernels based on the input shape. The problem is evaluating one kernel at one point and treating that result as representative of the full workload.
Fig. 3. Coverage: kernels tuned for one shape may underperform on others. Kernel time (lower is better, log axis) across five decode configurations. Variant 3, developed using the full set of target cases, is the only variant that performs consistently across all configurations.
In our PagedAttention experiments, Variant 1, generated only for prefill shapes, underperformed across decode configurations. Variant 2, tuned for two decode cases, generalized to some regular unseen shapes but failed on a ragged batch. Variant 3, optimized across all five target cases, maintained low latency across every configuration, including the ragged input.
Preventing these cliffs requires incorporating coverage directly into the optimization objective. Generation systems should optimize each operator across multiple representative cases rather than for a single benchmark point. Each candidate is compiled once, checked against every required case, and ranked across the full set. A primary shape can still receive priority, but secondary shapes must continue to meet explicit performance requirements so that an improvement in one configuration does not silently create a regression elsewhere.
The coverage question extends beyond "How fast is this kernel?" It is "Where is it fast, where is it valid, and does that domain match the workload that will actually deploy it?"
Making these objectives part of agentic kernel generation
Capability, correctness, and coverage should not be viewed only as evaluation criteria. They should shape what the generation agent sees and what it is allowed to optimize.
Putting these objectives into practice requires generation systems to provide the agent with a hardware-aware performance target, an explicit numerical contract, and representative deployment cases. Every candidate is then compiled, validated, and profiled against these requirements on the target accelerator. Retaining the results allows later iterations to build on measured evidence instead of relying on the agent's memory of previous attempts.
This approach turns production requirements into feedback available during generation:
- Comparing performance against the estimated hardware limit tells the agent whether a relative speedup still leaves substantial hardware performance untapped.
- Contract checks prevent a faster but numerically incompatible candidate from becoming the new best.
- Multi-case evaluation rewards performance that holds across the operator’s deployment envelope.
- Trace-derived workloads direct effort toward the kernels and shapes that affect end-to-end model performance.
- Persistent telemetry makes failures and optimization paths inspectable rather than reducing a run to its final score.
The result is a different optimization target. Instead of asking an agent to maximize one benchmark number, the system asks it to produce a deployable kernel.
What kernel generation results should report
Speedup remains useful, but it needs context. A more informative report should make the following dimensions visible:
- Capability: baseline provenance and quality, the methodology and assumptions used to estimate hardware limits, attainment across workloads, and end-to-end impact.
- Correctness: accepted tolerances, input and accumulation precision, the numerical contract, and validation beyond fixed benchmark inputs.
- Coverage: the shapes and input distributions evaluated, and performance away from the tuned point.
- Setup: target devices, whether kernels were re-tuned for each device, and the generation budget in samples, agent steps, or wall-clock time.
These details make results easier to interpret and harder to game. More importantly, they align evaluation with what production systems need.
Beyond the leaderboard
Agentic kernel generation is advancing quickly. As agents gain longer horizons, richer tool access, and deeper hardware knowledge, reported speedups will continue to grow. But those numbers alone cannot tell us whether a kernel approaches the target hardware's limits, preserves the intended computation, or performs reliably across deployment conditions.
The next stage of progress should extend beyond speedup to include production readiness. That means making capability, correctness, and coverage first-class objectives in both generation systems and benchmarks. Speedup remains useful, but the ultimate measure is whether a kernel can be trusted across production workloads.