Owning Your Intelligence Starts With the Harness

Sonya: Harnesses. I think this is a very important topic — a lot of you are thinking through building your own harnesses right now. I'm very excited to introduce Harrison. I first noticed Harrison on Twitter in 2022, back in the GPT-3 era, and Harrison was one of the first people thinking about: okay, we have these models — how can we build an entire harness around them so that they're not just doing autocomplete tasks, but start acting as virtual collaborators, or agents? The ecosystem has grown so much since 2022, and I've seen you grow a lot too in how you think about building agents, building harnesses, how to eval them. So I'm very excited to have you talk today. The talk is going to be about both harnesses and evals, and the format again will be 15 minutes or so of presentation content, 15 minutes of Q&A. Thanks for joining us, Harrison.

Harrison: Cool. My name is Harrison, co-founder and CEO of LangChain. I want to talk about evals and harnesses in the context of owning your own intelligence.

When we talk about intelligence, we're normally talking about agents. What exactly makes up an agent? At LangChain, we think there are three main parts: there's a harness that orchestrates a model and some context. And if you're talking about owning your intelligence in general, you probably want to own all three.

On owning the model, I'm not going to talk too much — Lin was here from Fireworks talking about open-weight models and owning that. A big part of this is also the ability to switch models. There used to be this concept of being cloud-agnostic, being able to switch clouds back in the day. The same thing exists for models: you want to be able to switch to avoid lock-in, but also to just use the best model when it's available. Context: you want to own all the context your agent uses, whether that's memory, semantic knowledge, or previous conversations. These help personalize and guide the agent as it goes along. And then the last bit is the harness, and that's what I really want to focus on.

How do you really own your harness? What does that even mean? The main job of a harness is to bring context to the model at the right point in time. It does all the orchestration around the fixed context and the dynamic context: it brings it into the context window of the model, shows it something, gets a response, and then does something with that. Agents need to do all these different things to accomplish their jobs — there's a ton of domain-specific stuff as well — but they need to interact with external systems, and those external systems, when you interact with them, emit more context that gets fed back into the agent and into the loop. The harness is the thing that orchestrates all of this together.

At their simplest, when everyone talks about agents, what they really mean is an LLM running in a loop calling tools. This is a really simple but really general architecture. Some request comes in, the LLM makes a generation, that generation may include a tool to call. If it does, you invoke those tools and pass the observation back to the LLM. This is the core architecture behind pretty much every agent out there today — but they're all different in slightly different ways.

On the left here is the base core loop, and there are a bunch of different things you can do in your particular harness at different stages. We build LangChain, which is a really base, minimal harness — that's LangChain over here. And then this is Deep Agents. Deep Agents is our model-agnostic, more general-purpose version of Claude Code. It does more things: it connects to file systems, it has skills, it has subagents. It's built on top of this really simple harness, but we customize it using these levers over here. You can run particular code snippets before the agent is invoked, before each model call; you can wrap the model calls; you can wrap the tool calls. You can customize this core simple loop in a lot of really powerful ways just by using small — what we call middleware — constructs. There are other ways to customize the harness as well: the concept of hooks and plugins has emerged in a lot of the coding agents, and that's essentially what they're doing — taking this base loop that's running and adding little hooks or plugins at various points to let you customize it.

So a lot of what you can customize is done through that concept of middleware, by modifying that core loop. The agent is still running in a loop, still doing that same simple architecture, but through it you can give it access to a sandbox, a file system, subagents, memory. You can have summarization — if we go back to this diagram, summarization would come in before the model is invoked: you check if the context is too long and then you summarize it. You add that into the core loop through middleware. Same with context offloading, which is a way of taking large tool calls and dumping them; that wraps the tool call. The point is, there's this really simple general architecture of an agent, and all of these more advanced agent harnesses are basically doing that loop while adding in a bunch of stuff. As you think about building or customizing your own harness, these are the different places you can insert things: your own summarization step, your own handling of particular tool calls. That's one way you can customize the agent and the harness to your particular domain.

The other way you can customize the harness the agent runs in is by having a more explicit cognitive architecture. This used to be the way a lot of people built agents in 2023 and 2024, because the models weren't good enough to run in a loop. To get them to do particular things, you'd have these very bespoke cognitive architectures. This one over here is for a deep research example, where it would generate some subquestions, fan them out, and then go execute them. And this one is for a code review bot — you can see these very bespoke steps. A lot of this has gone into the harness now, and by the harness I mean it's still this core loop; these might be added as particular modifications to it. But for a lot of really particular flows, we do still see people using cognitive architectures like these to guide things in particular ways.

One thing we recommend is to start with a general harness. It's the easiest to get started with and quickest to time-to-value. Then, as you narrow in on the use case you want to be excellent at, you can start to add more of these gates and checks around it to guide it in particular ways.

One question we get a lot is when to think about building your own harness versus using an off-the-shelf harness. A lot of the off-the-shelf harnesses work with particular models: Claude Code or the Claude Agent SDK, which work with Anthropic models; Codex, which works with OpenAI models. I think this is a big open question in the industry. My answer, generally, is that the more in-distribution you are relative to what the models are trained on, the better the off-the-shelf harness will be. As soon as you start to move further and further out of distribution, you'll probably want to tune your harness in some way.

There are different ways to tune the harness, too, because the models may be in distribution on particular things you're doing within an out-of-distribution task. What I mean by that: think about something like legal AI, which Gabe just talked about — I think he mentioned how they have their own harness. There are things in legal AI that are still in distribution for the main models. For example, editing files is something the main models have all been RL'd on, and they've actually all been RL'd in very particular ways: OpenAI and Claude models edit files in different ways in their harnesses, and as a result, their models are best at editing files in different ways. Now, the models themselves are out of distribution on the larger task of legal AI, but they're in distribution on the task of editing files. So if you're building a harness there, you'll probably want a custom harness, but you'll want it to use the edit-file tool that's in distribution for the model you're using.

One of the things we do in Deep Agents, for example — Deep Agents is our customizable harness — is this concept of model profiles: for things that are in distribution for models, like editing files, we switch between different edit-file implementations depending on which model is being used. That's an example of customizing the overall harness when it's out of distribution for a task, while keeping the smaller in-distribution parts as close to the model layer as possible.

The second big part of what I want to talk about is evals and observability. As you're experimenting with all parts of an agent — the model, the harness, the context — you're going to want to know what's going on inside the system, and you're going to want to be able to evaluate it. These are useful tools not just for custom harnesses but also for custom models.

There was a great Twitter article that Satya wrote two weeks ago where he talked about a lot of these concepts, and three quotes in particular stood out for me. One: create your private evals, because evals define what good looks like inside the organization. Two: retain ownership of your organization's memory, traces, feedback — the bold is mine — decisions and institutional context. And three: create your own continuous learning loop, a hill-climbing machine, that will allow your AI investments to compound the value of your firm. I think these speak to the importance of evals and observability, and the learning loop they power, in really owning your intelligence and compounding it.

So how exactly do they do that? Evals: Gabe was here talking about how they built benchmarks for the legal domain. I think every company building a mission-critical agent will build benchmarks for that agent. You can use them to define and catch regressions, or you can hill-climb on the benchmark, either by adjusting the harness or adjusting the model.

The thing we see becoming the industry standard for defining these benchmarks is Harbor. Harbor is an open source eval runner created by the makers of Terminal-Bench 2.0, one of the industry-standard benchmarks for coding agents, and it's become pretty popular across a variety of domains. What does it let you do? This is Frontier-Bench, which is another coding benchmark: you get this nice benchmark and you can compare different agent harnesses, different models, different reasoning efforts, and see how all of them do on your task. Having a benchmark for your task will become really, really important.

What exactly is Harbor? It's pretty simple at a high level. You run an agent against a dataset; a dataset has a bunch of different tasks. Generally they're run in sandboxes, because there are a lot of tasks and you might want to parallelize them, and, as I'll talk about in a bit, each task has its own environment. This is what a Harbor task looks like. On the right you can see it has an environment — this is where you define the environment the agent runs in. A lot of these longer-running, more stateful agents need to interact with their environment, so you spin up a sandbox, give it its own environment defined in a Dockerfile, and run it there. There's then a solution, which is a golden solution used to sanity-check the task — not that interesting. Tests is more interesting: this is the verifier for the agent run. The test scripts can do anything — run code, run unit tests, run another LLM-as-a-judge, run an agent as a judge. You define how the agent is scored in the tests. And then instruction.md is the prompt the agent is given. That's the core of Harbor: you define these tasks, which are bundled-up things that can be run in a sandbox, then you run a bunch of them against agents — and agents, again, consist of models and harnesses — and you score how they do.

When you do all of that, you get some nice results you can compare. This is the platform we build for evals and observability. You can see a bunch of different experiments here — we have a great integration with Harbor. You can see the feedback scores; in this case it's a single reward function. You can also track latency and tokens: when you're benchmarking agents, you probably don't just care about accuracy, you also care about latency and cost, so you'll want to track all of those. And for a particular experiment, these would be the different tasks in a Harbor dataset.

A bit about observability. Observability sounds basic, but I think it's really important and really underrated for agents. When agents mess up, they mess up because an LLM call goes wrong. Why might it go wrong? One of two reasons: one, the model isn't good enough; two, the context the LLM received isn't good enough. And I actually think it's the second one that more often causes issues. So having really good observability into what's going into the context window of the model, and how that context accumulated — what steps were run, what tools were run, how did that context get there — all of that is really important for debugging your agent when it goes wrong.

This is one view of observability we have, intended to be a more user-friendly view. It's similar to what you might see in Claude Code: we hide some of the tool calls — you can see seven tool calls up there — and we try to make it really easy to skim. Most agent paths these days come in the form of trajectories. You can think of trajectories as the list of messages you see when Claude Code is running: you type in a human message, it makes a bunch of tool calls — those are all messages under the hood — then it responds, then you type another human message. That message trajectory is becoming more and more central to these agents. But that's not enough to fully debug it, so we also have the full trace, where you can click into particular things and see exactly what goes on inside the model. That type of observability is pretty important for knowing what's going on.

Evals and observability really let you set up the data flywheel and compound the intelligence as your users start to use the agent and you start to get feedback. This is a slide one of our team members presented at the AI Engineer World's Fair, on a recipe for continuously improving agents. At a high level it's really simple: you build an agent, you start running it, you collect lots of traces, you curate the trace data, and then you run experiments on the data you've curated. Really simple — but of course there's a lot of complexity under the hood.

One thing that's really important for this is feedback: getting feedback either from the environment or from a synthetic source. From the environment — one thing I think is really underestimated in agent design is the UX design of how you present the agent to your users. If you present it in a really intelligent way, you can end up getting a lot of feedback from them. They may not click thumbs up or thumbs down explicitly — no one really does that — but if you design the UX in a clever way, you can get some of that feedback. The other thing you can do is get synthetic feedback: you can run what we call online evaluators over these traces to judge things. Gabe was talking about an experiment we did with Harvey where we significantly reduced the cost of some of these LLM-as-a-judge evaluations. If you imagine running one over every single trace that comes into your system, that racks up a big bill, so you want a really cheap and fast way of doing it. We've fine-tuned some SLMs for this, but you can of course use off-the-shelf models with custom prompting, or just use code if the things you want to test are simple enough.

So that's the full picture: curating the trace data — feedback is a big part there — and then using that data to update the system. You can update any part of the agent with this: update the harness by doing harness engineering, update the model by doing fine-tuning, update the context by doing memory. The part we think most about at LangChain is the harness engineering part, and I want to show a really quick demo of one of the things we added to help with that. I think Trajectory is talking next about some of the fine-tuning that can be done — it's a very similar process, where you run the agent, get traces, and use that data to improve the system. What is the system? It's these three pieces, and each can be updated in some way.

One of the things we think about is how to automate this as much as possible, because it's tricky and takes a lot of time. That's what we've been thinking about for the past few months. I want to do a quick demo of what we call [LangSmith] Engine, which is basically an agent that sits on top of your traces and does all this work: curating the traces, running experiments, and suggesting fixes to one of the three things. As I mentioned, we mostly focus on the harness engineering bit. Hopefully this will work — if not, it's not that big a deal.

Perfect. Okay, so this is [LangSmith]. This is a bunch of traces coming in, and we have this tab called Engine over here. This is an agent that runs in the background and creates what we call issue boards — this is the curating-data part. Under the hood, it's a coding agent that has access to our LangSmith CLI. With the CLI, you can filter traces for feedback and things like that. We give it a nice big prompt and some subagents that help it go out, explore the data, identify issues, and see what the common things are. Then it creates these issues right here. Here it's created an issue: it gives a description, it links to the traces so I can see supporting evidence, and down here — this is a very simple example — it's suggesting changes to the prompts. It's updating part of the context, it's updating some instructions, and you can see it's adding some code to go into the harness as well. This is something we launched in the past few months, and I think it speaks to the data flywheel — which, again, is a very simple thing: run agent, get traces, see patterns, fix. This is our attempt at automating it.

That's all I've got. Happy to take any questions on harnesses or evals.

Speaker 3: [For this automated harness optimization — what has the customer feedback been?]

Harrison: The customer feedback has been very positive. What we've found is that it generally finds smaller things — it's not going to make massive changes to your harness. A lot of what it finds are smaller sections of errors, not massive rewrites. One direction we've gotten feedback that people want more of is identifying larger swaths of area where — it's not that the agent is making errors, it's that questions are outside the scope of the agent. And I don't know how much of that we can automate, because that might require going to get a new data source or something like that. So we've noticed it's really good at finding these smaller, specific things that absolutely are errors, but it doesn't find the bigger directional things about the agent.

Speaker 4: This is great, by the way — I really appreciate the whole presentation. Engine itself is an agent, right, that's given a prompt and goes searching over things. Have you run Engine on Engine?

Harrison: We have it running, yeah. Engine also hooks up to Slack and sends us reports about itself — that's how we dogfood it. We also created what we call Issue Bench for Engine, which is a Harbor-formatted benchmark that we're constantly benchmarking different models and different harnesses on. I think Gabe talked about this a little, but one of the benefits of having a benchmark is you can run it against a bunch of different harnesses and see what they're good and bad at. A few weeks ago we ran our own Deep Agents, and then Codex, and then Claude Code on it, and we saw Codex doing a really interesting thing: it would write itself a bunch of small scripts to run against these traces, really aggressively, and that was actually allowing it to perform really well. So we did a sprint to do what we called the Codexification of Engine — basically take that learning and bring it into the core Engine harness. That's another benefit of having a benchmark: you can run a bunch of different things on it, see how they actually perform, and bring those learnings back into your core agent harness.

Speaker 5: Super cool talk. To what extent do you think harnesses will converge into one thing — users get educated on it, and the models are best for it — versus diversifying, where every company has their own way of doing things optimized for them?

Harrison: Yeah, really good question, and one we think a lot about. I chatted with Eno from Factory, who also thinks a lot about this. Honestly, I don't know is the answer. Some things I've seen: the general-purpose harnesses have gotten good enough to work for a lot of basic tasks, at least when you're getting started. So I'd recommend starting with an off-the-shelf harness, whether it's Deep Agents or Codex or Claude Code, because the models are now good enough, and the things we've learned make these models good — access to file systems, subagents, things like that — are good enough. But we often see that the more out of distribution you get, the more you're going to want to customize the harness. And it's a scale, right? At the extreme end, you might want to build a complete cognitive architecture that's really focused. Another reason you might want that, by the way, is predictability and control. We have a lot of customers in financial services who need predictability. We show them something like Deep Agents and they're like, whoa, whoa, whoa — that's way too scary an agent for us; we want more of a custom cognitive architecture where we can really control things. On the other end, you can just use an off-the-shelf harness, and there are things in the middle, like hooks or middleware. So it's a spectrum: the more out of distribution you get, the more custom a harness you're going to want.

And then there are other weird things. I think both OpenAI and Anthropic are getting really good at coding, but they've landed on different ways to edit files that are actually pretty different — I think there's some benchmark where one way was thought to be just strictly better than the other. So I think the model labs will kind of converge in that they all seem to be really good at coding, and the harnesses will converge toward being really good at coding if they keep going down that path. But at the same time, there are these really small differences, and I don't really know how to explain those either. Right now they show up most concretely in small things. But you could imagine: what if one lab really goes down bio, and their harnesses become really good at bio-agent things? Then the harnesses themselves start to diverge. So — I don't know is the answer. It's a fast-moving space. That's why evals and observability are important: to measure all of that. Cool. Awesome. Thank you guys.