Continual Learning: How AI Agents Get Better With Every Use

Sonya: Okay, last talk to bring us home. We have Arjun and Ronak, co-founders of Trajectory. They're doing a bunch of interesting research around what I think is one of the hottest topics: continual learning. Please welcome Arjun to the stage.

Arjun: Cool. What's up? How's it going, everyone? Thank you so much. I'm really excited. I know I'm close to the last one, so hopefully there's remaining attention span. Ronak is also back there, so we're together. But let's get started.

We're Trajectory, and we're building the platform for continual learning. Who are we? I think we're working on a really, really cool mission — that's the most fun part. The second most fun part is that I get to do this with my two best friends right over here, who worked at Windsurf and Transport One before this, and Michael, who worked on really cool robotics stuff at DeepMind.

I wanted to start with the worldview we have and how it shapes why we think this problem is important. And the worldview is — this is not a hot take — we are living in an incredible time in human history. Every single week, it seems like another model comes out that leapfrogs the last one. It's unavoidable, undeniable that models are getting better and better. But we think they're getting better on one axis, and that is IQ. These models are smarter and smarter, but it always feels like, when you're talking to them, it's their first day on the job. Having a Terence Tao in your pocket is great — but Terence Tao, day one at an accounting firm, is probably not the best accountant there. Give him a few years, or honestly maybe even a few days, and he'd probably be really good. So we have this orthogonal axis of experience that is much more important than we think, in conjunction with IQ. That's what we're calling the experience gap, and that is what we want to close as a company.

The question is: how do you close that? Where does experience come from? Well, it's already out there right now. There are so many — 100 trillion is a number — so many tokens out there being generated by these agents. You view them, or sometimes you don't even view them, and they get thrown away. But this is all real work they're doing. People act on it, and then it's thrown away. Our take as a company — as Gabe mentioned, our opinionated take — is that this is the signal we should be learning from. This is also how humans get better, and in classic AI fashion, if humans do it, it's probably a good thing to build a mental model off of.

Today's agents are still slow, expensive, error-prone. Over time, as you implement them, they probably don't get better. We want to imagine what learning agents look like: agents that, as they're used by people, get better and better over time. Tangibly, this has two benefits right away. It gets you much faster, better, cheaper models. But more excitingly, it gets you to this goal of systems that compound with use.

I want to start by talking about how we're approaching this problem, and then some interesting stuff for you at the end. Where it starts — and this dovetails with what Harrison was talking about before — is traceability. We need to first capture these interactions, capture this experience that's being thrown away. That's step one. Step two is where we start getting into our research. The way we view the company we're building is: we are building a product to allow for continual learning, and we're doing cutting-edge research under every single piece of it to make that possible. So you have all these interactions, and the next thing we're building is this idea of a model spec: the idea that you need a way to define what you want your agent to do, and have the agent learn against that. We're doing really cool research here on how to extract user interactions and turn them into reward — extract traces and turn them into exact specs of what you want your agent to do.

Okay, cool: you have what you want your agent to do. What do you do with that? There are two surfaces. One is the models. You want your models to learn from actual interactions, so we're doing really cool research on algorithms like SDPO and things along those lines, where we use RL to take these full, long traces and improve models over time. But models are not the only thing — you have harnesses as well. We're doing a lot of research on: when you have good feedback from people, does that go to the harness or does that go to the model? An example: if it's a fact, like this company has been delisted, you probably don't want to train that knowledge into the model. It's probably context that should be available to the harness. That's how we're splitting things, and that's the product we're building: allowing you to go from real interactions, to specs of what you want your agent to get better at, to better models and better harnesses, and then deploying those models right away in your product. That's the world we want to get to.

If you've built agents, which I'm sure most of you have, you know there are so many things that go wrong with every single part of this, and every single one is an unsolved problem. So the way I want to frame the rest of this: I'm going to imagine I'm straight out of Aladdin and I have four wishes. What are the wishes I'd ask for to make the agent ecosystem better? We have agents today, there's a gap, and there's a level agents should get to — and after that, we can really see the exponential effects of continual learning. I'll walk through the wishes step by step, in the same four categories: what we'd like to see on the traceability side, the eval side, the harness side, and the model side.

Wish number one: traceability, and how companies can get there faster. Two sub-wishes here. Number one is tracing the entire tree of what happens, including subagents. What a lot of companies do is trace the main action that's happening, but throw away the tool calls and the subagents that get made — and that makes it very hard to learn from the whole thing. Number two is probably the most important in here: you should build your product around how you can both capture this interaction data and elicit the right amount of feedback so you can capture it. A really interesting thing here: the level-one way of thinking about this is, okay, let's have a thumbs up, thumbs down, and capture that. Sounds amazing in theory, but it's incredibly noisy. If you've used any coding agent, you know you kind of just accept everything the agent does, and it's only five commits later that you're like, oh crap, this broke everything, let me go undo that. So it's the corrective behavior — the edits, the undos, the retries — that needs to be both elicited from the user and captured. That's my wish on traceability.

The second wish is around evals, and the mental model to use here is: the product your user uses should be as close as possible to where the eval is done, which, infra-wise, should be as close as possible to where the training is done. In an ideal world, these are all the same — there's no difference between them. This means evals are drawn from traffic: how people are actually using your product, both how they use it now and the things they're requesting on the frontier that might not be possible yet. All super helpful. Second is making every task rollout-able. This is a pretty big infra challenge, but it's the idea that if a user has done something, can I somehow replay what the user has done? That's an incredibly helpful boon in this era. And lastly, this goes with number one: grading through the real harness. Not a variation of the harness — the real harness people use in production. That's wish number two.

Wish number three gets into the harness. Pretty important. Number one — and I think the mental model here is that a lot of products built their harnesses around models that existed a year or a year and a half ago, when the primary function of the harness was to prevent the agent from doing bad things. That was really good when agents would randomly break or produce misformatted outputs. But now we're very much in a let-the-agents-cook world. So I'd view building the harness not as enforcing specific flows, but as: what are the primitives your product has, whether that's search tools or private information, and then viewing the agent as orchestrating those primitives — and letting it. That's sub-wish one. Sub-wish two — and this is how we're building our product as well — is making the agent interface as close to the user interface as possible. In an ideal world, every single thing I can do in your UI, your agent can do via a tool call as well. That's a world that makes training and continual learning so, so much easier. And lastly, a small one: make tool responses informative. It's easy to think, I just want to know the state — say I'm calling a search tool, I want to see if it worked; if it's writing to a DB, it just writes to it. What we often see is the response from that tool call is "Done. Finished." Sounds fine in theory, but from an agent's perspective that's incredibly confusing, and if you're trying to train or learn off of it, there's no signal. You have no idea what was actually written or read. So that's the last sub-wish under the harness: make your tool responses really informative. That's number three.

Canonically I only get three wishes, but imagine I have four, because the fourth is around the models — and we've talked about that a lot today. I wish it were as easy as just switching to an open-weight model, but if you've tried, you've probably seen there are 50 other considerations, from security to safety to different access provisioning. So start getting comfortable running on open weights, because that's what unlocks the door to owning your weights and then continually improving on top of them. And then — Gabe also talked about this — model routers are probably going to play a really large role in routing intelligence to the exact capability of the task at hand. So experiment with routers: that's another wish I'd cast into the world.

So that's the high level. When we work with companies, the first thing we often do is an audit of these areas and where things stand, and for the most part there's a lot of work to be done. So we work with them to build it out. It would be my dream to go to a company and find a lot of this already built. Granted, a lot of the work we're doing assumes we can't expect that: we're doing a lot of cool research on, assuming nobody has evals, how can we overcome that? Or assuming the traceability is off, how can we learn from the signals that are there? Those are my goals for the universe here.

I want to quickly go over — I think this is a very, very core research and product problem, and we're really excited about the team we've put together. But more importantly, and two of the folks are here today, we're working with extremely frontier customers: the folks who are really pushing the boundaries of what agents can do in production, both today and tomorrow. Really excited to be working with them.

Another thing I want to show — oh, that's my email — we have a beta. A really core belief we have is that this capability, this ability to own your own intelligence, should not be something you need to consult away. A lot of the time, it should be expertise you build on your own, because of how important it is to your product. What that means for us is we want to build a product that empowers any company to own their own models, and mainly, to get to continual learning from real interactions. Here are some screenshots from the product. [I don't know if Niko is here, but I'm supposed to show this to Niko later today — so this is a sneak peek.] Here's Harvey; here's the Legal Agent Bench that Gabe was talking about. Really easy to import. Training a model is incredibly easy — we're extremely proud of the interface we've put together here, because if you've post-trained, you know there are 50 million things that go wrong, 50 million knobs to turn, and it's all "researcher intuition," trademark, asterisk, whatever that means. What we've tried to do is take all of how our researchers post-train their models, use an agent behind the scenes to take care of most of those knobs, and expose only the ones you need to know, to make post-training as easy as possible. So: really easy to train a model, see how it's doing, eval it, compare it, see if it's better than the model you were using before, and then deploy it. In terms of actual work you put in — not counting waiting for the model to train — probably 15 minutes.

That's the world we're trying to get to: to empower all these companies, and hopefully more folks in this room. That's our goal. We want every company to own its own experience layer, and we're excited to build that alongside some of you all here, hopefully.

Speaker 3: Thank you, great talk. When you talk about continuous training, what do you think of as the trainable object — between model weights, harness, tools, application layers? How do you think about them, and how do you prioritize?

Arjun: A fun phrase is that if you ask six researchers what continual learning is, you'll probably get seven answers. You can be very pure about this and say it needs to be like a human: it's just the weights that need to be adapting in real time, one-shotting. But the way we view it is that the intelligence your product runs on is a system — a system with many components — and true continual learning is something that optimizes across that system, based on which parts of the system need to be updated for the information you're learning. A mental analogy I have: when you're saving things, we no longer think about where in your RAM or where on your hard disk to save. That's abstracted away, based on what makes the most sense. We think about models versus harnesses versus context the same way. It feels wrong that we have to decide, off of very little prior, what to update based on what. This is a scientific problem that can be solved. Let's solve it, and then let's abstract it away.

Sonya: Some of the speakers today talked about the importance of not training on your customer data, and I imagine a big part of continual learning is that you're learning from all these interactions. How are you thinking about things like differential privacy? How do you actually make your system better when, I imagine, most of these application companies have customer-specific data arrangements that make that very hard?

Arjun: A hundred percent — I think that's an incredibly important problem. Actually, before this I was at Apple, and this is a problem we worked on quite a bit as well. There are a lot of clever things you can do where you're not actually training on customer data: you're instead sampling distributions from your customer data, synthetically generating your own, and then — a mental model, maybe like a cryptographic thing — comparing those distributions to see whether your data is actually on-distribution, without training directly on the customer data. So that's totally right, there's a lot of interesting work there, and it's very central. Already with our customers, we're doing some interesting and fun things to get over that.

Speaker 4: I think episodic memory plays a role in continual learning to some extent. For example, if a user does something on our platform that corrects what an agent did before, we might want to update the agent's behavior. Does Trajectory have an opinionated take on that in the platform?

Arjun: Just to repeat, on episodic memory — if I'm thinking about this correctly — how different types of feedback impact what is learned?

Speaker 4: Yes.

Arjun: Okay, so many thoughts. One way to look at it is that there are two different types of signals. There are signals that just say something went wrong: for example, somebody flames the agent and says "you're really bad" but doesn't follow it up with anything, or just thumbs-downs, or drops off the session. Those are cases where you know something went wrong, but not what right looks like. Then you have things like: the agent retried and got to the correct solution, or you corrected it by showing the difference. That's a distinction we make. In the latter case, we're very confident about the reward associated with the output. With the former, we know to penalize the behavior, but we don't necessarily know the correct answer.

Another way of looking at it: there's a hierarchy of how pertinent the information is. There's some information that's probably globally true — like a tool call repeatedly failing when the agent tries a certain tool. That's relevant to everybody, so it should be trained into the model, and the model should get better at using that tool. Then: a certain user says, I never want to use the subagent, please, please, please don't do it. That's probably not something you should train into the model — leave it to the context. And what we're really excited about is the stuff Harvey was talking about, where this will probably happen on a per-org basis, or recurse even further to a per-customer basis. So that's another way to think about it: the hierarchy of what the feedback is pertinent to.

Speaker 4: Got it, thank you.

Speaker 6: One of the themes of today has been this idea that there are some tasks or parts of your product where you might want to experiment more, go cheaper, add in more open source — and other areas where being at the frontier, using a closed model or framework, is more useful. Extending that to continual learning: what's the shape of problem where you've seen this really matter a lot, versus where a statically trained frontier model — or even a statically trained open source model, with the harness injecting relevant information into context — is sufficient to achieve what the user wants?

Arjun: This is a great question. The level-one answer is that most tasks work for continual learning. But the ones I'm most excited about are tasks on the frontier. The mental model I have for AI progress is that people ask products for things based on their expectation of what the product can do. Maybe they ask at the edge of what it can't do, they see it fail or do something wrong, and they retreat: oh, it wasn't good enough, I can't use it for this. A great example: would I even have dreamed of typing into Cursor two years ago some of the whack queries I give it today? Definitely not — I was way below that. But over time I started querying it more, the models got better, and I thought, okay, maybe we can do bigger and bigger things. What we're most excited about: so far, that has happened because models generally got better and were pushed in certain targeted directions. But what we're seeing with some of our customers are cases where users ask for things the model can barely do — and then, through training, it learns how to do them, and the user can now do the thing they couldn't do before. That chain is what's really exciting about continual learning: really pushing the frontier of what's possible based on what the user tries and cannot yet do. Awesome.

Sonya: Thank you, Arjun — great talk, and I really appreciate you doing this.