Inside Agentic AI Workloads

tl;dr: In collaboration with researchers at UIUC and Intel, we evaluated ReAct-style agent workloads across complete trajectories. Serving demand depends on how each trajectory unfolds, whether cached state survives, and how tool use evolves. Serving systems must therefore manage scheduling, cached state, and resources across the complete trajectory rather than optimize individual model calls in isolation.

Traditional LLM serving is built around individual requests. A prompt comes in, the model generates a response, and the request completes. Each request is treated independently. The serving stack is designed to batch as many of these independent requests as possible, keep the GPUs busy, and optimize for metrics such as tokens per second and time to first token.

Agentic AI breaks that assumption. An agent’s execution is inherently iterative. It repeatedly invokes the model, makes tool calls, incorporates observations, and maintains an increasingly large context throughout the task. As AI agents become an important part of software development, data analysis, and research, these workflows account for a growing share of inference demand. Serving them efficiently requires understanding the complete execution trajectory rather than treating each model invocation independently.

In our new paper, Agentic AI Workload Characterization, presented at the 2026 IEEE International Symposium on Workload Characterization, Gimlet Labs collaborated with UIUC’s Yichao Yuan and Professor Nishil Talati, and Intel’s Souvik Kundu. The study examines how agents behave, accumulate state, and consume inference resources, and what these patterns imply for serving infrastructure.

We traced and analyzed ReAct-style agents running through Claude Code across five workload domains. These included data analytics and data-agent tasks from ADE-Bench and DABStep, software engineering from SWE-bench Pro, terminal-based tasks from Terminal-Bench 2.0, and general-assistant tasks from GAIA. For DABStep and SWE-bench Pro, we randomly sampled 100 tasks from each benchmark. We compared Gemma4-31B and Qwen3.6-27B with thinking enabled and disabled, which we refer to as their thinking and instant configurations. The models were served on two NVIDIA H100 GPUs, and we linked each complete agent trajectory to its request-level vLLM traces. We looked at three connected layers of the execution stack: agent execution behavior and evolving context, runtime serving performance, and tool invocations external to language models. Four findings stood out.

The traces reveal two ways a task becomes demanding. Some tasks accumulate large contexts as individual turns add substantial reasoning or tool output. Others require many turns, repeatedly re-entering model inference.

For example, Gemma Instant on ADE averages 109 turns per task, with a maximum of 786, yet its mean context stays at 60K tokens. Qwen Thinking on SWE-bench Pro shows a different balance, with just 41 turns on average but 79K tokens of context. Relative to the other benchmarks, SWE-bench Pro is heavy on context for every configuration, with the highest mean across all four (69K to 80K tokens). It is also heavy on the number of turns for most configurations. A workload can be demanding because it invokes the model frequently, maintains a large context, or both.

Both these dimensions translate into serving cost. More turns mean more model calls to schedule and run, while larger contexts mean more KV-cache memory held across the task. Each turn also builds on earlier model responses and tool results, so the serving system is handling a stateful trajectory rather than a stream of independent requests.

The growing contexts described above do not have to be processed from scratch at every turn. Each invocation carries forward most of the existing trajectory and appends only new reasoning, messages, tool calls, and observations. If the corresponding KV state remains available, the serving system can reuse the existing prefix and prefill only the appended tokens. Performance therefore depends not only on context size, but also on whether cached state is retained across turns.

When memory is sufficient to avoid cache eviction, empirical cache-hit rates measured by the serving system range from 84.6% to 99.5%, closely matching theoretical ratios based on trajectory prefix reuse (Fig. 3). In calculating the theoretical ratios, the initial request and newly appended tokens are treated as cache misses because they require first-time processing. Across configurations, mean raw input reaches 559.8 tokens per output token, while mean newly appended input never exceeds 7.3 tokens per output token.

This reuse changes where execution time is spent. End-to-end runtime is split between LLM and tool execution (Fig. 4a), with the balance varying across benchmarks. Within LLM execution, decode accounts for 94.4% to 97% of execution time on average despite the large contexts (Fig. 4b). Prefill accounts for only 3% to 5.6% when the cached state remains available. Agent inference is therefore better characterized as repeated decode over growing cached state than as repeated processing of independent long prompts.

That efficiency can disappear under memory pressure. A serving instance typically runs agents for many users concurrently, and their growing KV caches compete for the same limited GPU memory. In the Terminal-Bench concurrency experiment, the prefix-cache hit rate fell from 98.85% at 15 concurrent agents to 16.78% at 20 (Fig. 5a). At 25 concurrent agents, per-request generation throughput fell from 30.08 to 10.30 tokens per second (Fig. 5b), a reduction of about 66%.

When KV state no longer fits in GPU memory, the serving system must either recompute an evicted prefix or retrieve it from a secondary memory tier such as DRAM or SSD. Our experiment uses recomputation. The resulting prefill work competes with requests already generating tokens, slowing decode across the inference batch. Offloading avoids recomputation, but retrieval can still be costly because of transfer latency and limited bandwidth between memory tiers. In our measurements, prefill remains below 10% of request time (Fig. 5c), yet generation throughput falls substantially.

Together, these results explain why KV-cache retention is crucial. When the cached state survives across turns, agents reuse most of their input and spend nearly all of their LLM time in decode. When it is evicted, that advantage disappears, and the recomputation slows every request sharing the GPU, not only the agent that lost its state. For agent serving, the KV cache is long-lived execution state, and whether it stays resident directly affects throughput across repeated model invocations and tool calls.

KV-cache retention shapes the model-execution portion of an agent’s runtime, but model execution is only one part of the complete task. Between model invocations, agents call tools that vary widely in both latency and the work they perform. File operations, shell commands, web requests, and tools that launch subagents have different execution and output characteristics. A subagent can perform an entire agent execution within one parent-level call. Two workloads with similar tool-call counts can therefore have very different execution profiles depending on the tools they use.

Tool calls also reveal how an agent’s work evolves during a task. An agent fixing a complex software bug, for example, often starts much like a human developer by exploring directories and reading files before writing patches and running tests. Categorizing tool calls as read/explore or execute/write turns this intuitive pattern into a measurable runtime signal. When averaged across all five benchmarks, all four model configurations show read/explore activity declining and execute/write activity increasing as tasks progress.

Tool use varies across tasks and over the course of execution, so tool-call counts alone do not capture the work underway. Tracking tool intent provides an observable signal of shifts in an agent’s activity that could inform scheduling and resource allocation as the task progresses. For example, when deciding whether to keep an agent’s cached context resident, a scheduler could use its phase to anticipate which tools it may call next and how long those calls may take. Agent-serving systems can benefit from tracking tool behavior alongside model execution.

Tool activity provides one view into how an agent’s work progresses, but total serving demand depends on the complete run. Each model invocation consumes the context accumulated so far, while each model response or tool result can extend that context. Reasoning and failures illustrate how these effects accumulate across a trajectory.

Reasoning adds work to every model call. Before acting, a thinking model generates tokens that an instant model skips. Averaged across the five benchmarks, thinking accounts for 53.2% of Gemma Thinking’s output and 35.1% of Qwen Thinking’s output. Generating these tokens requires additional decode work, and retaining them increases the KV-cache state carried into subsequent turns.

That additional work, however, does not necessarily produce a larger context or more steps. As Fig. 2 shows, on SWE-bench Pro, Qwen Thinking averaged 41 turns, compared with 62 for Qwen Instant, while their mean contexts remained similar at 79K and 80K tokens. Although this relationship may vary by model and workload, the result suggests that more deliberation at each step can reduce the work required later in the trajectory.

Failed actions can increase serving demand in a different way. Error messages, stack traces, and unsuccessful patches enter the agent’s history and can lead to further diagnosis and retries. These loops add model invocations and increase the context carried between them. More broadly, failed runs often accumulate more context than successful runs. On Terminal-Bench, failed Qwen runs averaged 77K to 79K context tokens, compared with 44K to 46K for successful runs. Failed trajectories can therefore be an important source of long-tail serving demand.

These results show why individual requests are insufficient for estimating serving demand. A configuration that does less work per invocation does not necessarily reduce total demand if it produces a longer trajectory. Similarly, request-level averages can conceal retry-heavy runs that consume growing resources without making progress.

Serving infrastructure therefore needs a trajectory-level control layer above the request scheduler. This layer should group model invocations by trajectory, track cumulative resource use, and support task-level accounting and safeguards. Capacity planning and decisions about when to start new work should account for the distribution of demand across complete trajectories, including its long tail, rather than relying only on average per-request cost.

Takeaway: Efficient agent serving requires optimizing the whole trajectory

The central lesson is that an individual request remains the unit of model execution, but it is no longer sufficient as the unit of optimization. Each invocation belongs to a longer trajectory shaped by accumulated context, cached state, tool activity, and prior outcomes. Agent-serving efficiency must therefore be evaluated and managed across the complete trajectory. The quantitative results will change as models, agent harnesses, reasoning strategies, and hardware evolve, but the need for trajectory-level management will persist.

For serving systems, these findings make scheduling and cache management a joint problem. The set of active trajectories determines the live KV-cache working set. Higher concurrency can enable more parallelism, but it also expands that working set and increases the risk of eviction. Losing cached state can shift execution from decode-dominated processing toward repeated prefill. Scheduling policies should therefore weigh the benefits of concurrency against the cost of losing cached state, while multi-level cache hierarchies can preserve reusable state that cannot remain in device memory.

Evaluating these policies requires benchmarks that capture trajectory-level behavior. Beyond conventional fixed input-length, output-length, and concurrency measurements, agent-serving benchmarks should replay multi-turn workloads with growing shared prefixes, bursts of concurrent model calls, and waits for tool results. They should distinguish cached input, newly processed input, and generated output, and measure when increasing concurrency expands the live KV-cache working set enough to cause eviction, reduce cache reuse, and degrade throughput.

The same trajectory-level view also changes infrastructure planning. Memory capacity limits how many active trajectories can keep their KV state resident, while bandwidth determines the performance cost of moving state between memory tiers. Prefill, decode, KV-cache residency, and tool execution place different demands on compute, memory, and I/O. At Gimlet Labs, our heterogeneous systems are designed around this variation, matching workload components with the resources best suited to them.

Research like this helps ground these serving decisions in measured agent behavior. As agents take on longer tasks and account for more inference demand, improving serving efficiency will require looking beyond individual model requests to managing state and resources across the complete trajectory.

If you're interested in this work, you can reach us here, check out the paper, or explore our open roles.