If you are building a coding agent, you will eventually hit the context ceiling. Summarizing old chat turns sounds like the fix, and for a chat agent it mostly is. For a coding agent the bloat is structural: file bodies, command output, and tool-call arguments that never leave the message list. A write_file from turn 3 is still carrying its full payload at turn 40, and no summary of the conversation touches it.

I spent a few days studying the Open SWE harness and the Deep Agents library it sits on, taking notes on what I thought was interesting. Compaction was the topic I kept coming back to. It matters more for coding agents than chat agents, and I wanted to see what they actually do about it. The first half of this post is that distillation. The second half is how I mapped it onto Genkit middleware, with a full example at the end.

Part 1: What I found interesting

The mechanic underneath is simple. An agent loop keeps one append-only messages array, and every iteration the whole thing goes back to the model:

response = await ai.generate(messages=messages, ...)

The task, every tool call, every tool result: nothing falls out on its own. A session of n turns therefore sends on the order of n² tokens overall, and the constant factor is set by your largest tool payloads: a write_file from turn 3 is still riding along at turn 40. Compaction is surgery on that array before the next generate() call. The only interesting questions are which surgery, and in what order, which is what Open SWE and Deep Agents answer.

What Open SWE does at the tool boundary

The cheapest token is the one that never enters the array, and the earliest place to enforce that is the tool boundary: the moment a tool returns, you decide what becomes a message and what does not.

Open SWE’s main agent is a create_deep_agent() call in agent/server.py with a custom middleware stack on top. The piece I kept coming back to is ToolArtifactMiddleware in agent/middleware/tool_artifact.py. Its module docstring states the design plainly:

edit_file / write_file return only a one-line summary, but the dashboard
renders a full-file diff per edit. This middleware reads the file's *before*
content from the sandbox once, computes the *after* content locally, and
stamps the result's artifact with a {"diff": {...}} payload.

A tool result has two consumers that want different things. The model needs to know the edit succeeded; one line does it. The human reviewer wants the full diff, highlighted, on every reload. The mistake is serving both from the same place, because then the model pays context for bytes only the UI reads. Open SWE splits them: the one-line ack goes into the message the model sees; the full diff goes into ToolMessage.artifact, a serialized side channel the dashboard reads directly and the model never touches. It is textbook separation of concerns (hand back a reference to the bulky thing, keep the value out of the hot path), and it is lossless, because the artifact still holds the complete diff.

If you are deciding what your own file-edit tools should return, this is the decision that matters most: everything in the result is something every future model call re-reads.

Open SWE also caps its own before-read at 20,000 lines (_MAX_DIFF_LINES) when computing those diffs. Even the bookkeeping around compaction is budgeted.

What Deep Agents does underneath

If the tool boundary is about prevention, Deep Agents is about triage once the array has already grown. It wires summarization into every agent by default: create_deep_agent adds create_summarization_middleware(model, backend) to the stack, and Open SWE rides on it without modification. The part worth copying is not any single trick but the ordering. The middleware does three things in increasing order of severity, and treating them as a hierarchy (cheapest and most reversible first, lossy last) is the lesson.

First, it clips large tool-call arguments in older messages. TruncateArgsSettings names the failure mode in its docstring:

This is a lightweight, pre-summarization optimization that fires at a lower
token threshold than full conversation compaction. When triggered, only the
args values on AIMessage.tool_calls in messages *before* the keep window
are shortened. Recent messages are left intact. Typical large arguments
include write_file content, edit_file patches, and verbose execute outputs.

The arguments to a write_file from twenty turns ago are dead weight in a precise sense: the edit already landed, the file on disk is the source of truth, and a read is idempotent, so the model can recover the exact bytes any time by reading the file again. Clipping them costs nothing you cannot get back. Notice the keep window, too: recent messages are left intact, because recency is the cheapest predictor of what the model still needs, the same bet an LRU cache or a working set makes.

Second, large tool results get offloaded to the backend under /large_tool_results/, with a pointer left in the message. Same move as the artifact side channel, applied after the fact: swap an inlined value for indirection to it.

Third, when token usage crosses a threshold, the middleware summarizes the evicted span with an LLM call and appends the full original messages to /conversation_history/{thread_id}.md on the backend, so nothing is unrecoverable. This is the only lossy step (summarization is lossy compression; you cannot reconstruct the input from the output), which is exactly why it fires last and why the verbatim transcript is archived next to it. With a known model profile, the defaults trigger at 85% of the context window and keep the most recent 10%.

What this looks like on a real transcript

All of that is theory until you see it on your own harness, so I spent some time writing a real coding agent with Genkit to find out where the hot zones actually are and how much of Open SWE’s playbook would carry over. The fastest way to build intuition was to stop reasoning about it and just look at the raw messages on the wire, the exact array that gets serialized and handed to the model, and go slot by slot asking where I would cut.

So here is one of my runs: one failing test, two file reads, a pytest run, a patch, a rewrite, a green pytest run, then a follow-up from the user. Twenty-one slots in messages. The figure is the array itself: each index drawn to token weight (taller slot = more context burned), with the hot zones highlighted.

Read it as before-and-after of the same list. The moment I lined the indices up, the shape of the problem jumped out: most slots are cheap, a few are towers, and the towers are old.

Most slots are slivers: a tool finished, the model got a one-line ack:

// messages[7]: already cheap
{ "role": "tool", "content": [
  { "toolResponse": { "name": "write_file", "output": "wrote compaction.py (341 lines)" } }
]}

The towers are tool-call arguments that still carry whole files the agent already wrote to disk:

// messages[9]: a tower, the full file rides along in the args
{ "role": "model", "content": [
  { "toolRequest": { "name": "write_file", "input": {
    "path": "compaction.py",
    "content": "# ...340 lines of source, ~14.8k chars..."
  }}}
]}

That content is dead weight. The file is on disk; the model can re-read it. So compaction clips it in place: same slot, same index, a fraction of the bytes:

// messages[9]: after wrap_generate (Part 2)
{ "role": "model", "content": [
  { "toolRequest": { "name": "write_file", "input": {
    "path": "compaction.py",
    "content": "[clipped: 14.8k chars omitted]"
  }}}
]}

That is the whole trick, repeated down the prefix. The hot zones (messages[3]/messages[6] <read_file> deliveries, messages[8] a pytest traceback, messages[9]/messages[11] file-carrying args) get clipped or truncated. The keep window (messages[15] onward) stays verbatim. The array is still 21 elements long, but the total drops from ~50.5k characters to ~9.1k, and none of it was summarized.

For the same array as a flat bar chart, easier to compare slot sizes at a glance:

Total: about 50.5k characters, roughly 12k tokens, for a task a human would describe in one sentence. By messages[20] the model needs almost none of the bytes in messages[0] through messages[14]. That ratio is the whole problem, and because every turn re-sends the whole array, it compounds with each tool call.

The stack, in order

Putting Open SWE and Deep Agents together, the layers fire cheapest first:

1. Keep bulk out of the message history at the tool boundary (short acks, diffs in a UI side channel)
2. Clip bulky arguments in old tool calls before each model call
3. Truncate or offload huge tool results, leaving a pointer the agent can follow
4. Summarize with an LLM only near the context ceiling, with the verbatim transcript archived somewhere recoverable

Summarization is the only layer that costs a model call and the only one that loses information you cannot point back to. Both projects treat it accordingly. Everything above it is structural: cheap, and in practice lossless because the files are still on disk and the archived logs still exist.

A note on where the bytes land before any of this runs: harness design matters. Open SWE and Deep Agents paginate read_file and keep the body in the tool result. If your own tools return full file bodies into the message list, compaction can shrink the stale deliveries later, but paginated reads and short write acks are the better starting point. That is the Open SWE tool-boundary lesson applied before middleware ever runs.

Part 2: Implementing this in Genkit

On PyPI genkit==0.8.1, middleware is BaseMiddleware from genkit.middleware, registered with @ai.middleware, and attached per call with use=[...]. Two hooks map onto the stack above. wrap_tool runs around each tool execution, so you can shrink a fresh oversized result before it joins the history. wrap_generate runs on each iteration of the tool loop, so you can edit params.options.messages before the next model call. There is no official Filesystem or Artifacts middleware package on that release, and GenerateMiddlewareContext has no session or artifact API. Short acks and any side channel for diffs belong in the tools you write; recoverable offload is your own disk or object store, not a framework primitive.

Open SWE’s short acks land in the tool implementation (and optional wrap_tool polish). Deep Agents’ argument clipping and keep-window truncation live in wrap_generate. Immediate shrinking of a fresh enormous tool result lives in wrap_tool. Summarization near the ceiling is another wrap_generate step, and if you want a recoverable transcript you write it somewhere yourself.

The structural half of that stack is what I care about for the transcript in Part 1: clip the towers in messages[9] and messages[11], and shrink oversized tool responses outside the keep window. That is aimed squarely at those hot zones. Summarization can wait until you are near a real context ceiling.

What a Compaction middleware looks like

Subclass BaseMiddleware with a Pydantic config, decorate it with @ai.middleware so it shows up in the local Dev UI, and override the two hooks. The sketch below is the shape, not a full recipe: clip bulky tool-call arguments outside a keep window, truncate large tool outputs in wrap_tool, and leave summarization as an optional follow-on.

from copy import deepcopy

from pydantic import BaseModel, Field

from genkit import Genkit
from genkit.middleware import (
    BaseMiddleware,
    GenerateHookParams,
    GenerateMiddlewareContext,
    MultipartToolResponse,
    ToolHookParams,
)
from genkit_google_genai import GoogleAI

ai = Genkit(
    plugins=[GoogleAI()],
    model='googleai/gemini-flash-latest',
)

class CompactionConfig(BaseModel):
    keep_recent_messages: int = 6
    max_tool_input_chars: int = 400
    max_tool_output_chars: int = 1500
    preview_chars: int = 120
    bulk_input_keys: set[str] = Field(
        default_factory=lambda: {'content', 'old_string', 'new_string', 'patch', 'diff'}
    )


@ai.middleware(
    name='compaction',
    description='Clip old tool-call args and truncate oversized tool results.',
)
class Compaction(BaseMiddleware[CompactionConfig]):
    async def wrap_tool(self, params, ctx, next_fn):
        # Aimed at a fresh tower: shrink the result before it enters history.
        result = await next_fn(params, ctx)
        text = result.output if isinstance(result.output, str) else str(result.output)
        if len(text) <= self.config.max_tool_output_chars:
            return result
        preview = text[: self.config.preview_chars]
        return MultipartToolResponse(
            output=f'{preview}\n...[{len(text)} chars truncated]',
            content=result.content,
            metadata=result.metadata,
        )

    async def wrap_generate(self, params, ctx, next_fn):
        # Aimed at bars like messages[9]/messages[11]: clip args outside the keep window.
        msgs = list(params.options.messages or [])
        keep = self.config.keep_recent_messages
        cutoff = max(0, len(msgs) - keep)
        compacted = []
        for i, msg in enumerate(msgs):
            if i >= cutoff:
                compacted.append(msg)
                continue
            compacted.append(_clip_bulk_tool_args(msg, self.config))
        new_options = params.options.model_copy(update={'messages': compacted})
        new_params = params.model_copy(update={'options': new_options})
        return await next_fn(new_params, ctx)


def _clip_bulk_tool_args(msg, cfg: CompactionConfig):
    cloned = deepcopy(msg)
    for part in cloned.content or []:
        req = getattr(part.root, 'tool_request', None) or getattr(part, 'tool_request', None)
        if req is None or not isinstance(req.input, dict):
            continue
        trimmed = dict(req.input)
        for key in cfg.bulk_input_keys:
            value = trimmed.get(key)
            if isinstance(value, str) and len(value) > cfg.max_tool_input_chars:
                trimmed[key] = (
                    f'{value[: cfg.preview_chars]}'
                    f'\n[clipped: {len(value)} chars omitted]'
                )
        req.input = trimmed
    return cloned

If you are assembling your own middleware instead of starting from this sketch, the mapping stays direct. Open SWE’s short acks are a tool-return decision. Deep Agents’ argument clipping is a wrap_generate pass over params.options.messages. You do not fork Genkit’s agent loop; you add a middleware class and list it in use.

Full example

Wire the same ai instance: define Compaction with @ai.middleware, define file tools that return short acks, then pass use=[Compaction(...)] on the call. Layer 1 of the stack is already in the tools before middleware runs.

# Continues from the Compaction class above (same `ai` instance).

from pathlib import Path

from genkit import Message, Part, Role, TextPart

workspace = Path('./workspace')
workspace.mkdir(exist_ok=True)

@ai.tool(description='Read a UTF-8 text file under the workspace.')
def read_file(path: str) -> str:
    text = (workspace / path).read_text()
    # Paginate in a real harness; full bodies are how towers start.
    if len(text) > 4000:
        return text[:4000] + f'\n...[{len(text)} chars total, truncated]'
    return text

@ai.tool(description='Write a UTF-8 text file under the workspace.')
def write_file(path: str, content: str) -> str:
    target = workspace / path
    target.parent.mkdir(parents=True, exist_ok=True)
    target.write_text(content)
    lines = content.count('\n') + 1
    return f'wrote {path} ({lines} lines)'

messages: list[Message] = [
    Message(
        role=Role.SYSTEM,
        content=[Part(root=TextPart(text=(
            'You are a coding agent. Work only inside the workspace directory. '
            'Read files before editing them.'
        )))],
    ),
]

async def run_turn(user_input: str) -> None:
    global messages
    response = await ai.generate(
        prompt=user_input,
        messages=messages,
        tools=['read_file', 'write_file'],
        max_turns=20,
        use=[
            Compaction(
                keep_recent_messages=6,
                max_tool_input_chars=400,
                max_tool_output_chars=1500,
            ),
        ],
    )
    messages = response.messages
    print(response.text)

The knobs worth tuning first are keep_recent_messages and the two size caps, against your model’s real context size. If you later add LLM summarization near the ceiling, keep it behind the structural pass so you are not paying for a summary of bytes you could have clipped for free.

What it buys you

The messages-array figure in Part 1 shows compaction slot by slot. Here is the same list aggregated into one bar per stage:

The drop is almost entirely structural. Old read deliveries shrink to short previews, the write_file argument clip removes the 15k rust bar, and the pytest traceback outside the keep window gets truncated. What remains in the tail is prose, recent reads, and small acks. On a transcript this size summarization does not fire yet; that layer kicks in near the context ceiling.

A note on the companion recipe repo: jeffdh5/python-middleware-recipes still has a fuller compaction.py, but it depends on session artifact APIs and genkit-plugin-middleware, which are not installable from PyPI genkit==0.8.1. Treat that repo as design reference for the clip/truncate ordering, not as something uv sync will run against public packages today. The sketch above is the part that works on the shipped surface.

To run the samples against PyPI genkit==0.8.1:

uv add genkit genkit-google-genai