Coding agent harness: what I learned by cracking open Codex, OpenCode and Pi

TL;DR: A harness is everything around the model: the agent loop, tools, permissions, sandbox, and context compaction. The model is stateless and remembers nothing between turns; the harness holds the state and logic. This is why two coding agents using the same model produce different results, and those results vary along several axes: whether the agent completes the task, how long it takes, and how much it costs. The difference lies in the architectural bets made by their harnesses, not in the models. Below, I examine those bets using the source code of Codex, OpenCode, Pi, and my own agent: where state lives, who holds permissions, whether there is a sandbox, and which bets will survive the next major model release versus which ones it will make obsolete.

In my previous article about prompt caching, I made a point that is actually more important than caching itself: the model remembers nothing; it is stateless, and the harness maintains all memory on its behalf. On every turn, the harness reassembles a single request containing system instructions, tool descriptions, history, and fresh input, feeds it to the model, and interprets what the model asks it to do. Caching was the pretext for that article. The real protagonist remained behind the scenes: the harness, the wrapper around the model.

I am building my own agent on top of OpenAI models. It is called agent-core, and it uses Python and Pydantic AI. I wrote the basic agent loop myself, and it started working fairly quickly: the model requests a tool, the harness executes it, the result goes back to the model, and the cycle repeats until there is an answer. Then the difficult problems began. How should context be compacted once it grows too large? How do you stop the agent from repeating work it has already completed? How do you know that it has actually finished rather than simply run out of steam? Whenever I got stuck on one of these problems, I went to see how the grown-ups handled it. Fortunately, the source code for Codex, OpenCode, and Pi is public, while Anthropic has documented the architecture of Claude Code in detail.

Here is what I found. Coding agents differ neither by the model inside them nor by the feature list on their landing pages. They differ through the architectural bets made by their harnesses, and those bets are visible directly in the code. Some are diametrically opposed: something one agent treats as foundational may be entirely absent from another. This difference is rarely shown because doing so requires opening the source code of four agents and comparing them side by side. So that is what I did. What follows is an analysis of those bets, with code examples, and an attempt to determine which ones are worth taking seriously and which will survive only until the next major model release.

I verified all the code in this article in mid-July 2026 and pinned it to specific versions: Codex 0.144.5, OpenCode 1.18.3, and Pi 0.80.10.

Harnesses by the numbers

On Habr, the phrase “the same model produces different results in different harnesses” has been repeated so often that it has become a cliché. The problem with clichés is that people stop checking whether they are true. So instead of a slogan, here are three numbers, all measured using the same fixed model.

SWE-bench Pro is designed so that the scaffold—the harness used for the benchmark run—can be fixed while changing only the scaffold itself. Claude Opus 4.5 completes 45.9% of tasks using SEAL’s standard scaffold. The same Opus, the same model, reaches 55.4% with the Claude Code harness. That is an increase of 9.5 percentage points from the harness alone. On SWE-bench Verified Mini, the gap is even more dramatic: Sonnet 4.5 solves 68% of tasks with the SWE-Agent scaffold, while the same model gets only 34% with HAL’s generalist scaffold. Half of the result depends on how the system around the model is built.

The strongest number comes from a controlled experiment on 100 SWE-bench Verified tasks, where the researchers independently varied the model and the harness and measured the variance. The variation caused by changing the harness was 7.8 times greater than the variation caused by changing the model.

We need to slow down here, or this will turn into the same cliché with numbers attached. A conflicting study from IBM Research (arXiv 2602.22953) shows the opposite: across five general-purpose agent architectures running on five models, the model explained 27.8% of the variance in quality, while the agent architecture explained only 0.5%. That is a difference of more than 50 times in the other direction. IBM used general-purpose architectures assembled according to standard recipes. The experiment reporting a 7.8× difference used harnesses optimized for the task. The rule I took away from this is: with a general-purpose harness, the model determines the result; with a harness polished for a specific task, the harness determines the result. My agent-core is currently closer to the general-purpose end than I would like. It is a product-oriented, general-purpose harness, rather than a scaffold refined around one narrow benchmark where every architectural bet can be tuned.

These numbers become outdated quickly, so do not fixate on the absolute values. The ratio matters more than the exact percentage. Newer models, Opus 4.8 and Sonnet 5, already produce different absolute results, which is why the rest of this article focuses on architectural bets rather than on who is ahead by how many percentage points today. Terminal-Bench 2.0, for example, rose from 62.9% in January to 84.7% by summer.

The formula Agent = Model + Harness was introduced by Viv Trivedi of LangChain, although half of the secondary sources attribute it to Mitchell Hashimoto, who never actually used that formula. Trivedi’s definition is broad, and it also provides the best list of dimensions for comparison: the harness is everything that is not the model. System prompts. Tools, skills, and MCP. Built-in infrastructure: the file system, sandbox, and browser. Orchestration. Hooks and middleware: compaction, continuation, and linting. The word has two meanings, and it helps not to confuse them: a broad product-level meaning, covering the entire wrapper around the model, as Anthropic and OpenAI use it; and a narrow architectural meaning, referring only to the execution loop. The Claude Code documentation states it directly in the broad sense: “Claude Code is the harness; Claude is the model inside it.” I will move between both meanings below, but it should always be clear which one applies.

Anatomy: what the framework provides and what you write yourself

The heart of every harness is the agent loop. They all do roughly the same thing: call the model, inspect what it requests, execute the tool, return the result, and repeat until there is a final answer. Anthropic describes this as three phases: gather context, take action, verify results. The real question is who writes the dispatcher for this loop and what additional logic it injects.

Pydantic AI, which underpins my agent-core, provides a graph with three nodes: a user-input node, a model-request node, and a tool-call node. I could have handed control over to the graph and stopped thinking about it. I chose not to. Instead, I wrapped it in my own loop with explicit dispatching because I needed control over every transition:

while not isinstance(node, End):
    if isinstance(node, UserPromptNode):
        node = await agent_run.next(node)
        continue

    if isinstance(node, ModelRequestNode):
        model_request_count += 1
        if model_request_count > settings.max_loop_depth:
            raise ToolLoopLimitExceeded()
        await recorder.emit(
            "run.calling_model",
            data={"label": "Calling model"},
        )
        ...

    if isinstance(node, CallToolsNode):
        ...

(agent-core, agent/runtime.py)

The graph gave me the transition structure between nodes. I wrote everything else in this while loop myself: a depth counter with a hard limit to prevent the agent from entering an infinite tool loop, an outward event stream (run.calling_model and about a dozen others), and separate accounting for web-search calls with its own cap. This is the boundary worth identifying in any harness: here is where the framework operates, and here is where the engineer begins adding what they consider necessary.

It is useful to see how the same loop is implemented by more mature systems. They all do the same basic thing—call the model, execute tools, repeat—but each makes a different bet about who holds the state between loop iterations. Codex runs an explicit Rust loop {} and decides whether to continue based on the needs_follow_up flag:

loop {
    …
    match sampling_request_result {
        Ok((sampling_request_output, sampling_request_input)) => {
            let SamplingRequestResult {
                needs_follow_up: model_needs_follow_up,
                …
            } = sampling_request_output;
            …
            let needs_follow_up = model_needs_follow_up || has_pending_input;
            …
            if !needs_follow_up {
                …
                break;
            }
            …
        }
        …
    }
}

(codex, codex-rs/core/src/session/turn.rs)

OpenCode uses a single while (true), but turns everything inside it into tasks: a subagent, compaction, and an ordinary step become three branches of the same dispatcher.

while (true) {
  …
  const task = tasks.pop()
  if (task?.type === "subtask") {
    yield* handleSubtask({ task, model, lastUser, sessionID, session, msgs })
    continue
  }
  if (task?.type === "compaction") {
    const result = yield* compaction.process({ … })
    if (result === "stop") break
    continue
  }
  …
}

(opencode, packages/opencode/src/session/prompt.ts)

Pi deliberately keeps the loop memoryless: state is moved outside into hooks, while the while loop itself knows nothing about the session.

while (true) {
  let hasMoreToolCalls = true;
  while (hasMoreToolCalls || pendingMessages.length > 0) {
    …
    const nextTurnSnapshot = await config.prepareNextTurn?.(nextTurnContext);
    …
  }
  const followUpMessages = (await config.getFollowUpMessages?.()) || [];
  if (followUpMessages.length > 0) {
    pendingMessages = followUpMessages;
    continue;
  }
  break;
}

(pi, packages/agent/src/agent-loop.ts)

Four loops, four answers to the question of where state lives: in my system, it lives in the Pydantic AI graph and manual dispatching; in Codex, in the needs_follow_up flag; in OpenCode, in the task queue; and in Pi, deliberately nowhere inside the loop, because everything lives outside it. Claude Code adds its own three phases—gather context, take action, verify results—but I have examined its loop only through Anthropic’s description rather than its source code.

Another one of my bets can be seen in where tools are actually executed. They are not executed inside agent-core. The brain—request assembly, the loop, and decision-making—lives in one service, while the hands—writing files and executing code—live in a separate agent-runtime. They communicate over HTTP with HMAC signatures. The most dangerous tools, delete_file and move_file, are explicitly marked in the manifest with the blocked_in_model_loop flag: the model cannot call them autonomously from inside the loop. I am currently moving the sandbox to Daytona, though this is still in the exploratory stage. Separating the brain from the hands is a deliberate bet, and below we will see that mature systems make very different choices here.

To keep this entire zoo straight, here is a reference matrix. The comparison dimensions are on the left, while the five harnesses I can examine in detail are across the top: four through source code, plus Claude Code through official materials.

I will not walk through the table row by row; that would turn into a neutral reference guide. Instead, I will take several problems I encountered myself and show how differently each system approaches them. That is where the real differences become visible.

The agent is convinced it has finished, but it is wrong

Early versions of my agent behaved in ways that made me want to close the laptop. You would ask it to run a script, the script would fail, and the agent would cheerfully report that it had received an error and consider the turn complete. It would not investigate or fix anything; it would simply return the error as the result and stop. I addressed this directly through the system prompt—finish the job—and it helped, but the experience left a clear impression: by default, the model is far too eager to decide that it has done enough.

It turned out that this was neither my fault nor a quirk of the particular model I was using. Caleb from the Caleb Writes Code channel shows an agent working on a long task until it reaches the edge of its context window, summarizes itself into a short note, and confidently announces that everything is complete, even though only half the work has been done.

Then I found the exact same failure mode in Anthropic’s engineering analysis. Anthropic explicitly describes two failure scenarios for long-running tasks. In the first, the agent reaches the end of its context in one pass and leaves the work unfinished. The second is subtler and worse: the next agent sees the work already completed, notices the progress, and “declares the job done” without verifying it. Models are optimistic by nature. Ask one to evaluate its own work and, in Anthropic’s words, it may “confidently praise the work even when quality is obviously mediocre.”

It is worth seeing how a tool error reaches the model in the first place. Consider Codex: a failed command does not terminate the loop or throw an exception. Its output, including the error text, is wrapped and returned to the model as an ordinary tool result.

let result = if exit_code == 0 {
    Ok(content)
} else {
    Err(FunctionCallError::RespondToModel(content))
};

(codex, codex-rs/core/src/tools/events.rs)

RespondToModel effectively means: “Here you go; deal with it yourself.” OpenCode does the same thing, marking part of the result as output-error, while Pi returns a result with isError: true. Errors are always handed back to the model, and as we have just seen, the model is entirely capable of shrugging and closing the task. This is exactly what I kept encountering during the first few weeks. Therefore, whenever an outcome can be verified, I try not to leave the decision solely to the model.

Mature systems solve this architecturally rather than by saying, “Let’s write a better prompt.” Anthropic separates the generating agent from the evaluating agent, so there is no one left to praise their own work. The evaluating agent does not take claims at face value: it opens the live application through Playwright MCP and inspects it directly. The feature list is stored in a JSON file where every feature defaults to passes: false. The agent may change the flag, but it may not delete or rewrite the checks themselves, preventing it from “completing” a task by erasing the test.

I approached this problem from a simpler angle, but with the same underlying logic: do not let a probabilistic model decide that work is complete when the result can be checked deterministically. agent-core has domain hooks, one of which catches the case where the model tries to write a file again and the tool returns a “file already exists” error. A naive agent may shrug and assume that if the file exists, the work must already be done. Mine does not assume; it reads and compares:

observed = await execute_tool(
    "read_file",
    {"path": path},
    deps,
)
…
if observed.get("content") != content:
    return None

return DeterministicStopDecision(
    outcome="already_done",
    message=f"Файл `{path}` уже содержит нужный текст, поэтому ничего менять не пришлось.",
)

(agent-core, agent/domain_hooks.py)

If the file exists and its contents match byte for byte, then yes, the work is “already done,” and that is an honest deterministic fact rather than a model’s guess. If the contents differ, nothing has been completed and there is no point pretending otherwise. It is a small piece of code, but it captures the whole idea: wherever the truth can be verified, I place a deterministic check above the probabilistic model. Anthropic does the same at a larger scale by separating roles; I do it at a smaller scale by comparing file contents.

Skills, incidentally, suffer from a similar problem: an agent may load the same skill twice and waste context on it. Mine follow Anthropic’s Agent Skills specification directly, with the same contract: a package containing SKILL.md with YAML frontmatter and adjacent references, scripts, and assets directories. Lazy loading comes from the same specification: the system prompt receives only a compact “name: description” list, while the full text is retrieved through load_skill when the skill is actually needed. I added one extra layer that is not part of the specification: a context manager that reconstructs from the conversation history which skills have already been loaded and returns an already_loaded flag. The system prompt states: “load_skill and read_skill_resource are idempotent; if either tool returns already_loaded, continue using the previously loaded instructions or resource from conversation context.” I discussed how this lazy loading interacts with prompt caching—another headache, because tools live at the beginning of the request—in my previous article about caching.

Context compaction: four answers to one question

For quite a while, I thought compaction meant, “We will just ask the model to summarize the history,” so I kept postponing it. Then I looked at how mature systems handle the problem and understood. Compaction does not fail because of the compaction technique itself. It fails because, near the edge of the context window, the agent starts to panic. Anthropic has a separate term for this: “context anxiety.” Near the context limit, the model starts wrapping up its work too early in an attempt to finish in time. With Sonnet 4.5, this behavior is so pronounced that compaction alone is insufficient; complete context resets with transferred state are required. According to Anthropic, this anxiety is “greatly diminished” in Opus 4.5 and 4.6. In other words, the harness’s behavior around compaction depends on the model inside it, and the same code may need to be rewritten for a more capable model.

The compaction mechanisms also differ substantially; this is not a single checkbox. Codex does not return compacted history as a string. Its special endpoint, /responses/compact, returns a list containing opaque encrypted_content: latent model state rather than a human-readable summary. It is visible directly in the type:

Compaction {
    encrypted_content: String,
},

(codex, codex-rs/protocol/src/models.rs)

OpenCode divides compaction into two mechanisms and represents both as tasks in its loop. prune traverses history from the end and discards old tool results while preserving the last two turns and the most recent 40,000 tokens. Compaction itself divides the history into a head and a tail, summarizes the head, and preserves the tail. Claude Code behaves similarly near the limit: it first removes old tool results and then summarizes. CLAUDE.md and memory survive compaction because they are reread from disk, and after compaction the “five most recent files” are placed back into context. Pi summarizes the head of the history while preserving the tail. Four harnesses, four different answers to one question. I also discussed how to avoid paying twice for compaction because of broken caching in my article about prompt caching, including Claude Code’s elegant solution based on preserving the same prefix. In the second part, I explained why caching is reused only for a shared prefix and can collapse because of a single extra character, going all the way down to KV tensors and the internals of vLLM.

A spectrum of bets: from Pi’s minimalism to Codex’s armor

If the four harnesses are placed on a single axis representing how much responsibility the harness assumes, Pi and Codex sit at opposite ends.

Pi describes itself as a “coding agent that does less.” This is not posturing; it is visible in the source code. There are no subagents in the core. There is no MCP at all: I checked the dependencies of every Pi package, and @modelcontextprotocol/sdk is simply absent. There are seven tools. The sandbox exists only as an example extension rather than as part of the core. Pi deliberately removes things that others treat as mandatory.

In return, Pi makes a bet that almost nobody else makes. Extensions are ordinary TypeScript files loaded through jiti, meaning that they are transpiled at runtime without a separate build step:

/**
 * Extension loader - loads TypeScript extension modules using jiti.
 */
import { createJiti } from "jiti/static";

(pi, packages/coding-agent/src/core/extensions/loader.ts)

The /reload command reloads extensions, skills, and settings without restarting the session. Put these together: the agent can write a new TypeScript tool for itself on the fly and immediately load it into the active session. The extension can import the agent core itself as a library. This is the literal embodiment of the idea that “the agent writes code as an extension of itself,” and Pi gave up MCP, subagents, and a built-in sandbox in exchange for that bet.

When I saw this, my first impulse was to tear apart my own skill system and implement the same thing dynamically. Then I remembered what Pi pays for that capability, and the impulse passed: I need both MCP and a sandbox, while Pi deliberately includes neither.

Codex sits at the other end of the axis, making the opposite bet: security is foundational rather than optional. The sandbox is embedded into the core, and the Seatbelt profile for macOS begins with one crucial line:

; start with closed-by-default
(deny default)

(codex, codex-rs/sandboxing/src/seatbelt_base_policy.sbpl)

Everything is forbidden, and only what is necessary is then explicitly allowed. On Linux, the equivalent is built from bubblewrap, Landlock, and seccomp. Network access is disabled by default. Another firm Codex bet is that it remains deliberately stateless on top of the Responses API. According to OpenAI’s documentation, it intentionally does not use previous_response_id, avoiding server-side state to support zero-data-retention mode. It therefore rebuilds the request from scratch on every turn. Where Pi strips down to the minimum for the sake of flexibility, Codex surrounds itself with armor for predictability and isolation.

To make the spectrum even clearer, consider subagents and move from one extreme to the other. Codex forks the history and starts a subagent as a genuine child thread:

/// Spawn a subagent by forking persisted history from `forked_from_thread_id`.
pub async fn spawn_subagent(
    &self,
    forked_from_thread_id: ThreadId,
    mut options: StartThreadOptions,
) -> CodexResult<NewThread> {

(codex, codex-rs/core/src/thread_manager.rs)

OpenCode invokes a subagent in the same process, as an ordinary call inside its existing loop, without separate threads:

const result = yield* ops.prompt({
  messageID: MessageID.ascending(),
  sessionID: nextSession.id,
  model: {
    modelID: model.modelID,
    providerID: model.providerID,
  },
  …
})

(opencode, packages/opencode/src/tool/task.ts)

Pi does not include subagents in the core at all. They exist only as an example extension that launches a separate pi process through spawn:

/**
 * Subagent Tool - Delegate tasks to specialized agents
 *
 * Spawns a separate `pi` process for each subagent invocation,
 * giving it an isolated context window.
 * …
 */
import { spawn } from "node:child_process";

(pi, packages/coding-agent/examples/extensions/subagent/index.ts)

One axis, four different bets: a native thread in Codex, the same process in OpenCode, a separate process launched by a Pi extension, and no subagents at all in my system yet. Claude Code gives subagents their own context and system prompt, then returns a short summary to the main conversation.

The other systems fall between these poles. OpenCode provides full MCP support: local servers launched through commands, remote servers accessed by URL, and OAuth. Claude Code places permissions at the harness level rather than the model level, which is a fundamental distinction. Its documentation states that instructions in your prompt or CLAUDE.md “don’t change what Claude Code allows.” They do not alter what the harness itself permits. Claude Code’s OS-level sandbox applies to Bash, but not to file reads and writes. My agent-core, with its external agent-runtime, sits closer to the stricter end, though it still lacks a native sandbox. This has bothered me for a long time: I separated the hands from the brain, but I am only now moving real isolation into Daytona. One question—“What must the harness guarantee instead of the model?”—produces five answers with very different levels of strictness.

How to choose, for yourself and for a team

At the code level, I examined four harnesses plus Claude Code, but the market is broader, and it helps to understand the bets made by the others. As of mid-2026, all of them remain active and under development. Amp split from Sourcegraph into an independent company in December 2025 and switches between four modes, from low to ultra. The mode changes the set of models used under the hood, while users still pay based on actual usage without an additional markup. Factory’s Droid targets enterprise deployments, including fully isolated environments with no outside access. Its persistent virtual machines grew into a separate product, Droid Computers, which preserves state between sessions. Block’s Goose moved under the Agentic AI Foundation, a new Linux Foundation organization that also brought MCP and AGENTS.md under the same umbrella in December 2025. Goose supports more than fifty providers, and its extensions are built directly on MCP. Google’s Antigravity is an agent-first environment available in several forms, from an IDE to a CLI and SDK, and Google is also folding Gemini CLI into Antigravity CLI. Since April 2026, GitHub’s Copilot CLI has supported user-provided keys and local models; with a custom provider, GitHub authentication is not even required. Cursor is rolling out cloud agents and its own Composer model, while in June SpaceX announced that it would acquire Cursor’s parent company, Anysphere, for $60 billion in stock, the largest acquisition of a venture-backed startup in history. The spread of architectural bets is the same as in the systems whose code I examined, only without the ability to look inside.

omp, the main Pi fork, stands apart. Its developer took the exact opposite approach: batteries included, a “coding agent with a built-in IDE.” It incorporates everything Pi deliberately excluded: LSP, a real debugger through DAP, persistent Python, first-class subagents, 32 tools instead of seven, and around fifty thousand lines of Rust in the core. omp appeared in late 2025 and collected 18,000 stars in six months, compared with 72,000 for Pi itself.

Team selection adds another layer, and the most revealing example again comes from Codex, specifically the team building it. According to their own account, a group that started with three engineers and grew to seven built a product containing roughly one million lines of code and around 1,500 pull requests in five months, with zero lines written by hand. All code was generated by the agent, averaging around 3.5 pull requests per engineer per day. The initial scaffold and the first AGENTS.md—around a hundred lines, closer to a table of contents than an instruction manual—were also generated by Codex. Invariants are enforced through custom linters rather than code review, while a separate background cleanup agent removes deviations. OpenAI calls this way of working harness engineering.

Birgitta Böckeler of Thoughtworks provides a useful vocabulary for thinking about this layer. She describes a harness as a cybernetic regulator with two components. Guides are proactive instructions that steer the agent before it acts. Sensors provide feedback after an action, becoming more effective as the signal is tailored more precisely to the model. A custom linter message written specifically for an LLM works better than a raw compiler error. For a team lead, this is the practical way to evaluate a harness: how developed are its sensors, and can they be adapted to the team?

The market map also needs to account for generational change. The turning point came in late autumn 2025: I switched to Claude Code myself in November, and the data supports the pattern, with Claude Code’s share of commits and survey responses rising sharply that winter. Some products were swept aside by the wave. Cline lost the core of its team—at least seven people moved to OpenAI—although the company itself remains active. Roo shut down in favor of a cloud product, and its niche was taken over by a fork called Kilo. Aider is technically still alive and continues to release updates, but it has almost disappeared from discussion, so I no longer use it as a reference point. There are also harnesses designed for work other than coding, such as OpenClaw, a bridge between messaging platforms and models, or Hermes, a self-improving personal agent. These should not be confused with coding agents; they are different tools for different tasks.

The market is therefore better understood through these bets than through landing-page promises: how the loop works, who controls permissions, whether there is a real sandbox and subagents, how the system can be extended, and whether it is locked to a single vendor. This leads to a more important question: which of these bets will survive the next model release, and which will it erase?

Build to delete: what to build with disposal in mind

All of this has an uncomfortable property. Some of the harness code you write is destined to be discarded; you simply do not yet know which part.

Philipp Schmid formulated this in early 2026 as the principle of “build to delete”: make the architecture modular, because new models will replace your logic, and be prepared to remove code. He grounds this directly in Sutton’s “bitter lesson,” which says that general methods backed by increasing computation eventually outperform handcrafted layers. Anthropic makes the same point more strictly, and the wording is worth quoting in full: “every component in a harness encodes an assumption about what the model can't do on its own, and those assumptions are worth stress testing, both because they may be incorrect, and because they can quickly go stale as models improve.” Every piece of the harness encodes an assumption about something the model cannot do independently, and that assumption needs to be tested regularly because it expires as models improve.

According to Manus’s own account, they rewrote their harness four times. They did not refactor it; they rewrote it completely because old workarounds became unnecessary with each new model. The canonical example of a harness designed for disposal is Geoffrey Huntley’s Ralph: a bare Bash while true loop that starts a fresh agent instance with clean context on every iteration, while maintaining memory externally in Git, progress.txt, and prd.json with a passes field. There is no elaborate internal context management; all complexity is moved into the loop design. In November 2025, Anthropic formalized the idea as an official ralph-loop plugin and a demonstration harness for autonomous coding. Ralph’s small repository is not the goal but a consequence: in the author’s words, it is “essential to use as little as possible.” The less context each iteration consumes, the better.

Addy Osmani adds a rule he calls the ratchet: add a restriction to the harness only after observing an actual failure, and remove it as soon as the model can operate without it. Do not build workarounds in advance, but do not try to operate without them prematurely either.

This ratchet has already clicked in my system, and I can trace the full cycle through commits. In early March, I wrote a regular expression called _needs_path_clarification. It parsed the user’s message before it reached the model and used lists of verbs in two languages—“исправ,” “обнов,” “edit,” “update”—to decide whether the request identified the target file unambiguously. If not, it demanded clarification of the path to the file that needed to be fixed, updated, or edited. Six days later, I ran into the heuristic myself and had to add an exception for .docx. The commit comment states the reason directly: the heuristic intercepts the request before the model has a chance to call read_docx_structure. Nine days after that, I removed the heuristic entirely—854 lines deleted—and delegated the decision to the model through the system prompt.

A workaround survives only until the day it causes more trouble than it prevents. My current candidate for removal is even more absurd: the model sometimes returns tool-call arguments as several identical JSON objects concatenated together, such as {"name":"docx"}{"name":"docx"}, which causes ordinary json.loads to fail. The runtime therefore contains a parser that recognizes the concatenation and collapses it into a single object when all the pieces are identical. It will disappear as soon as the provider fixes serialization.

Forty-three minutes after removing that regular expression, I moved in the exact opposite direction and restored deterministic blocking for delete_file and move_file: the same blocked_in_model_loop flag mentioned at the beginning of the article. The commit explains why: previously, both tools had been forbidden only through an instruction in the system prompt, and prompt injection can bypass an instruction. Where security is concerned, a prompt turned out to be an inadequate substitute for code.

What would I leave untouched even while accepting the build-to-delete principle? Permission boundaries and sandboxes. The event contract that supports observability. The ability to switch the underlying model. These bets define the system around the model, and they become more important as the model grows more capable. The more an agent can do autonomously, the more expensive missing boundaries become. This is the line along which I suggest evaluating a harness: treat everything that compensates for the model’s current limitations as temporary; treat everything that defines boundaries and contracts as structural.

I began by getting stuck on a few problems and looking inside other harnesses simply to understand how they worked. What I found was somewhat unexpected. A harness cannot be understood as a static object; it has no correct final form. It changes along with the model. Something that acts as a load-bearing beam today may become an unnecessary support after two major model releases. The key is knowing in advance which parts are beams and which are temporary supports. A useful question for every line of harness code is: am I doing this for the model because it cannot do it, or because it could not do it yesterday?

The source code reveals which bets were made, but says nothing about what those bets cost. That bothered me throughout the writing process, so I assembled a test bench and measured it: four tasks, seven harness–model combinations, and 56 runs. That will be the subject of the next article.

Stay curious.

I write about artificial intelligence, language models, and developer tools. I test models and services on real-world tasks and share my conclusions in my Telegram channel.