THE CHAI & AI COMMUNITY Designing AI Harnesses: Principles and Practice A framework for thinking about agents, models, and the layer in between by Rachitt Shah JUL 09, 2026 24 1 Share This article grew out of a live Chai & AI session with Rachitt Shah (AI at Accel), moderated by Akshat Kharbanda and was summarized by Aiza Hasib. The views expressed in the article are personal and do not represent those of Rachitt's employer. The vocabulary around what wraps a model keeps shifting. What used to be called an orchestrator became an agent, then a harness, and in some flavours it now goes by agentic SDLC. These terms overlap heavily, and the field has not landed on clear distinctions. A simple way to think of a harness is as extremely good software engineering practices applied to Applied AI problems. The model itself can be thought of as the unit of compute and the harness as the design philosophy that wraps around it, scaled via code. The framework that follows, walks through how to think about a Harness: The three layers of a well defined harness Whether to design from the model or from the job to be done Whether to build from scratch or compose from existing harnesses How to treat evals so they do not become a bottleneck Where harnesses sit as models get more capable 1. Map the three layers Every harness has three layers worth distinguishing. The model layer is how intelligence is consumed and how it flows through code. This covers the model itself, how it is prompted, and the inference patterns used to call it. How you use the model can matter as much as which model you use. A recent MIT paper on Recursive Language Models is a good example. The technique wraps the LLM so it treats long context as an external environment and recursively sub-calls itself over snippets. An RLM using GPT-5 scored 91% on BrowseComp-Plus, a multi-million token benchmark where standard base models scored zero. The gain came entirely from how the model was wrapped. The external data and API layer is what the harness consumes to do useful work. Existing data sources, third-party APIs, internal services, and MCPs all sit here. This is the deterministic part of the harness, and it is entirely software engineering. The verification layer is what stops bad output from cascading. It covers evals, structured handoffs between agents, and rules about when not to pass unverified data downstream. The classic example is a team of two agents. If agent A hallucinates and the handoff to agent B keeps the hallucinated data in the context, agent B is burning tokens for nothing. The verification layer catches that before it propagates further. A natural question that comes up is how agents and harnesses differ. An agent is the intersection of model and harness, given a mandate to execute end-to-end. What that intersection needs to look like depends on the task. Hard-verifiable tasks like writing code have a built-in feedback loop, since tests either pass or fail. The harness can stay relatively light because the model has a clear signal to iterate against. Soft-verifiable tasks like filing taxes have no such loop, since you cannot really tell if the outcome was correct until much later. The harness has to do more of the work here to make the result trustworthy. Either way, the agent is the tool calls plus the harness engineering. 2. Design from the job, not the model Most teams fall into one of two camps when they start building. The first is what you might call model-first. The premise is that we trust this model and we are going to build everything around it. It is a small subset of teams, but they exist. A year ago, that often meant building entirely around Claude. The better camp starts from jobs to be done. Before designing the harness, picking tools and choosing context strategies, define what the thing actually needs to do. For a deep research agent, if the job is researching the people on a call, the starting point is realising the actual work is finding LinkedIns, GitHubs, Twitters, and bios. Model choice, context strategy, and tool design all come second to that. The idea is to design backward from the outcome you need, with the job to be done as the anchor. Everything else, from context management to prompt structure and tool selection, sits downstream of it. 3. Choose to build or compose Once the jobs are clear, the next decision is whether to build or compose. Building means writing a harness from scratch for five to seven specific use cases you need to support. Because the scope is bounded, the context windows stay tighter and evals stay clearer. It is the right approach when you already know what you need. Composing means inheriting from existing harnesses. You pull in an orchestration library, use its preset context management tooling, and assemble what you need. Since you are not reinventing the wheel for problems that already have solutions, speed is an advantage here. Both approaches have trade-offs. Building takes longer up front, since you are writing more of the machinery yourself. Composing is faster to start, but you sit downstream of someone else’s release cadence, which bites in production. A cleanly designed library can still be difficult to maintain when the package keeps updating fast enough to create dependency issues. Heavier packages also affect build times and runtime, and the flexibility you get is bounded by what the abstractions allow. Since speed is everything in this space, a practical tactic, regardless of which approach you take, is to lean heavily on open source so long as the license permits it. Shipping a harness in a couple of weeks beats shipping in a couple of months, by which point the trend has usually moved on. Inherit components where they save time, and reinvent only where you have to. 4. Treat evals as a first-class citizen Only two parts of a harness are actually non-deterministic. One is context management and the other is evaluation. Everything else is basically software engineering. Context management matters for latency, cost, and the quality of downstream output. If a sub-agent hallucinates and the handoff to the next sub-agent preserves the hallucination, you are burning tokens to amplify bad output. Putting a verification step in place before the handoff prevents this, and so does designing sub-agents so that each one handles a small task rather than making any single one carry a long workflow. Without strong evals, you end up running in circles. Prompt improvements that should help do not translate into any measurable change and regressions creep in without warning. Tool calls time out, APIs return 429s and 500s, and there is no clean way to isolate whether the problem sits in your harness or somewhere else. A mix of Behavior-Driven Development (BDD) and Eval-Driven Development (EDD) is helpful here. BDD helps you anticipate user behaviour and encode it into the harness before users find the edges. EDD catches behaviour and quality regressions before users do. For the deterministic parts of the system, the external services, MCPs, and APIs, the unfashionable answer is to write a lot of tests. Go and look at the test repository of any reasonably designed harness and the volume of tests is significant. Harnesses are, when you strip it down, lines of code telling a model what to do, and you can manipulate that behaviour deterministically using classic software QA. Doing this well makes about 70% of the harness work easier, which frees up your time to spend on the parts that actually need experimentation. 5. Know where a harness will still matter As models get more capable, two trends run in parallel. For vertical-specific use cases, harnesses become more valuable, not less. Ramp built Ramp Inspect, their in-house coding agent, despite having access to Claude Code and Codex. It runs in sandboxed VMs, integrates with Ramp’s CI/CD, monitoring, feature flags, and internal services, and at last public report wrote 30% of merged pull requests across their frontend and backend repositories. Ramp built rather than bought because the specific problem they were solving needed integration that off-the-shelf coding agents could not provide. As the bar for vertical fit goes up, the in-house harness becomes the actual differentiator. For more general use cases, the opposite trend is playing out. The Titans paper from Google Research introduced a neural long-term memory module that scales context beyond two million tokens by learning to compress and recall information at inference time. If approaches like that mature, much of what current harnesses handle through external context management gets absorbed into the model itself. Things like tool calling, memory, and routing logic could move from harness code down into the model layer. The dividing line seems to be between stable use cases and nuanced ones. Stable use cases will get absorbed by better models and richer application layers over time. Nuanced ones, where context saturation or vertical-specific behaviour matters, will keep needing harnesses around them. A lawyer reviewing case documents still hits context saturation no matter how good the model gets, and the harness around that workflow is what makes the difference between something usable and something that frustrates the user. How to build the intuition The best way to learn is by getting hands on. Pick 3-5 different harnesses with different design philosophies and spend time in each of them. You will start to notice the differences yourself. CrewAI, for instance, is built around role-based agents, while Claude is built for long-running ones. OpenCode is model-agnostic and centered on tool calls and LangChain leans on inheriting components from a large standard library. Sitting with these design philosophies side by side teaches more than any single resource. A useful tactic when you find a harness you like is to hand the source code to Claude or Codex, have it walk you through the patterns, and then replicate the workflow yourself in code. This helps internalize the design philosophy by writing it out yourself, since harness engineering is mostly philosophy scaled via code. The other recommendation is to get your hands dirty. Try things, break things, and read the test suites of harnesses you like. The practitioners with strong views on harness engineering have almost all spent time in the source code of several of them. The vocabulary is going to keep moving. Whatever the field calls this layer six months from now, the underlying question stays the same. Can you give a model data, tools, and an environment, and can you trust it to get a task done autonomously? Whatever you call that thing is your harness. These sessions happen regularly, and they get into the details most tutorials skip. It’s invite-only for course alumni, but we’ll keep sharing highlights here every week. If you want in, subscribe to The Nuanced Perspective to get notified. The Nuanced Perspective is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Upgrade to paid 24 Likes ∙ 1 Restack 24 1 Share