Building Frontier AI at the Application Layer

Sonya: Next up, we are honored to have Gabe with us. Gabe is co-founder and president of Harvey. You were a research scientist at DeepMind ages ago, and then at Meta, before you showed your college roommate what GPT-3 could do — and this duo then became Harvey. We're really excited to have you here. Everybody in the audience is an application company thinking about how to start doing their own research, post-train their own models and create their own labs. Harvey has really set the example here, and we're delighted to have you give a talk on how you built Harvey Labs.

Gabe: Awesome. So the alternative title to this talk is Building a Research Lab on a Budget. It's an unfair game competing with the frontier labs if you're an application-layer company: there are rich teams, there are poor teams, and then there's us in the application layer. The frontier labs have more money, more talent, more compute, infrastructure and data. So how do you compete? By using the frontier ecosystem.

When we started Harvey four years ago, most of these companies either didn't exist or were just getting started. So we either had to build everything ourselves or, in most cases, focus on something different, like building our GTM org and a great product. But today, using the frontier ecosystem, I think you can compete with the frontier labs and build frontier intelligence. This talk is our high-level playbook for doing that. I'm going to talk about how we build benchmarks and training data, how we work with the neolabs to do post-training, and how we serve these models in production.

To start, you want to build a benchmark. If you don't have a good benchmark, you can't train models. And if you can't train models, you don't need to serve them in production. This year we released three of these datasets. We started by building Legal Agent Bench, which is a taxonomy of tasks that associates would do at a large law firm. These cover multiple practice areas, and they're complex tasks: drafting complex fund formation documents, doing case law research, things like that. We followed this up by building a contracting dataset, which allows us to teach agents to do negotiation like you would in an in-house department. And then the one I'm most excited about, which we recently released, is a large diligence dataset. This is, I think, one of the largest RL environments that's been released. The largest data rooms here are 80 million tokens, and it lets us do research on long-context, very complex tasks.

I think the most interesting thing about these datasets is how we built them. One challenge we've always had at Harvey for training models is that we can't train on our customers' data. We work with the largest law firms and enterprises, and their legal data is incredibly sensitive. It's privileged. You can't put it in generic models; we can't even put it in our models. So how do you train models given that?

The thing that started working really well this year is using domain experts to guide synthetic data generation. Brendan from Mercor had a good analogy: the same way that engineers now don't write code — they vibe code and guide these coding models — we're starting to do the same thing with data. My younger brother is actually a lawyer at Harvey, and he's gotten very good at using the coding models. He's trained our other lawyers to do it, and they can generate incredibly realistic datasets that we can use for training and also for evaluating our product. Once you've done that, synthetic data alone isn't good enough, but it's a way to get started. So we work with companies like Mercor and Snorkel, who let you scale up this process and build larger sets, particularly for training.

Once you've done that, you need to turn these datasets into efficient RL environments. As these datasets get larger, evaluation gets very expensive. For example, in our diligence dataset, we have over a thousand unit tests grading model outputs using LLM-as-a-judge. If you use the largest models and you want to do RL rollouts, it gets very expensive. So there's a lot of work — here's some we did with LangChain — on making these very efficient.

And then the last thing we did, which I think was a little controversial at the time, was open sourcing some of these datasets. The motivation was that it's very hard to know your dataset is good unless a lot of people train on it. When I used to do research at Google Brain and DeepMind, the best datasets were open — ImageNet, CIFAR, MNIST — and everyone used them, and you were able to find all the issues. We get a ton of pull requests, we get suggestions, and increasingly the labs benchmark on our dataset when they report new models. And most importantly, we had Elon retweet it.

So once you've built the benchmark, you have something to train models against. And the exciting thing now is that open source models are getting competitive. In the past, it wasn't worth doing post-training, because the models were improving so quickly from pre-training that any post-training you did quickly got absorbed by the next pre-trained model. But now, with models like Kimi K3, GLM 5.2, Nemotron, Inkling and others, it's possible to take these very strong open source base models and post-train them to levels of frontier intelligence. Maybe not general frontier intelligence, but if you have a specific task like us, they are competitive.

The way we recommend getting started is working with the neolabs. They have a bunch of expertise and infrastructure already in place to help you make sure your training datasets are good. They have recipes. And usually, if you work with them and you're not able to get better results, there's probably something you're doing wrong with your dataset. This is a very good way to bootstrap.

Some of the interesting work we've done with these different providers: with Fireworks, we got some very interesting results training GLM 5.1 to use Fable, or maybe Opus 4.8, as an advisor model. With Baseten, we did some interesting work on KV cache compaction. With Engram, who I think is here, we're doing interesting work on enterprise search and firm knowledge. With Trajectory, we worked to train Nemotron models. And with Applied Compute, we're doing some interesting work on our Vault product.

One question we got is: why work with multiple neolabs? Why not just pick one? For us, as we're scaling the research lab, we have more research projects than we have bandwidth to do internally or with a single neolab. And every neolab is taking a different bet. They have different ways of thinking about research; we have different open source models we want to train. The more we work with, the more we learn.

It's also getting easier than ever to do post-training. So one, working with the neolabs, we're learning a lot in partnership with them. And two, we're doing more and more post-training ourselves internally, with APIs like Tinker and the infrastructure Fireworks and Baseten have built. It's never been easier to post-train these models and then serve them. And there's increasingly more post-training talent available that we're hiring. Inspired by Cursor, the goal of these efforts is to build our version of Composer 1: to package all the work we've done with synthetic data, scaling it with Mercor, and the work with the neolabs into a model we can serve alongside the closed source models.

Now, once you've trained a model, you need to be able to serve it in production, and this is non-trivial. I want to start with our model serving infrastructure, because I think people sometimes still think of application-layer companies as calling a single model endpoint for a chat product. A big problem we've solved over the past four years is that we operate in 60 countries, we have multiple product surface areas, and customers have different model preferences. Even with just the closed source models, how do you serve these at scale? This matrix gives you a sense of everything we need to handle. As a simple example, for each of these model families, we need to serve multiple models, and we need fallbacks across providers to hit our SLAs. And now, with companies like Fireworks and Baseten, we can add open source models into this mix. You need this infrastructure in place before you even think about post-training.

This is how we think about it when a new model is released, whether it's open source, closed source or a model we've post-trained: how we decide whether to put it in production, and once it's in production, whether to keep it there. Pre-production, we have a set of generic evals. On the automated side, we have the Legal Agent Bench benchmark I talked about: whenever a new model comes out, it gives us a very quick sense of whether it's a frontier model, how strong it is, and what areas of legal it's good at. We have human testing in the generic case, where we run side-by-sides of the model against other models. Then, for every product surface, we have critical user journeys and automated product tests, because a model could be very good generically but not a good fit for a specific product surface. And then we also have human product testing. Together, these signals, along with heuristics around cost, latency and region availability, are how we decide whether to put a model into production.

The important point here is that this is the case for post-trained models and non-post-trained models alike. You can reuse this infrastructure, and you should have it in place before you think about post-training. Once a model is in production, same thing: it doesn't matter if the model is post-trained or not. We do A/B testing if we're rolling out a large change. We look at engagement to track whether the model is performing how we expect. We look at things like uptime and token efficiency. And then there's what we officially call product feedback — I call it angry customer emails. All of these signals tell you whether it's working as expected. You need that infrastructure in place before you think about serving models.

And then the thing to do even before serving your own models is what I call the simple open source switches. First: look at all the places you're serving models and ask, are there places in my product where I can just naively swap in open source models? For example, we have parts of our product that generate citations that don't need the largest models, and there are opportunities to swap in GLM 5.2 and get cost or performance benefits. Doing this builds the muscle of serving open source models alongside closed source models. Second, which has gotten very popular now, is model routing. There could be places where you can't naively swap in an open source model, but on certain queries, you can route to open source models.

Once you've done all this, you have everything in place to start building the post-training flywheel: serving models in production and collecting feedback. One caveat: you need to be very careful about what collecting feedback means. In our case, it does not mean training on customer data. But we do get feedback signals from our user testing and other sources that can inform how we build future datasets and improve these models.

And that is our playbook for building a research lab. We think that in the future, every company is going to need to become an AI company and figure out some version of this playbook. And despite that, most people are still betting against this playbook, against application-layer companies and the frontier ecosystem.

[A clip from Moneyball plays: "If we win, on our budget, with this team... we'll have changed the game."]

Gabe: This is a scene from Moneyball, which hopefully you've seen. They're talking about how they just won 20 games in a row, and Billy Beane says it doesn't matter if we don't win the championship — no one will appreciate what we've done here. And the quote is: but if we win, on this budget, with this team, we'll have changed the game. I think now, with the frontier ecosystem, all of you have the opportunity to do the same. Go change the game. Thank you.

Speaker 4: How do you deal with the challenges of having legal customers with very sensitive datasets? And can you talk a little more about your brother the lawyer, and other in-house experts doing their version of vibe coding? What does that actually look like?

Gabe: Great question. The biggest challenge with legal work is that at a large law firm, the type of work you're doing is, for example, representing a company doing an M&A. You get a data room, which is all the contracts of the company you're trying to acquire, plus all these emails and meetings about the negotiation. None of that data is public. So you don't have the analogy of open source GitHub repos. The biggest challenge you run into when training is that you can maybe find some of the final work product publicly, like a public purchase agreement, but you don't have any of the input data. And historically, it was very awkward to get a bunch of lawyers and say, hey, make a fake data room, because these data rooms can be 10,000 contracts that all need to fit together.

What Julio figured out is a very clever way to generate these datasets. He started from the rubric: he planted all these issues and said, here are all the problems in the data room, here's what I'm going to check for, and a scenario. Then he used that to generate the data room. You can plant all of these issues — these contracts don't tie together, this contract is missing — and generate all the data. Then we use Mercor and Snorkel to make all the contracts look realistic. Now you have this input dataset, you can have the model generate a diligence memo, and you have a way of checking it: did you catch all the issues we know are in there, because we planted them?

Obviously, depending on the work you're doing, you need to be clever about how you create it. But that to me is the really big unlock, because now we can start doing this training. Before, you had this chicken-and-egg problem: we'd go to law firms and say, hey, we can train a model on this client data, and they'd say, okay, prove it. And we'd say, show us the data. And they'd say, no. Now you can prove it here, and we have a lot of interest from law firms saying, oh, this is really interesting, can we do it with our data? Good question.

Speaker 5: We're also building an applied AI layer. From a hiring point of view, it's difficult to compete with researchers from Anthropic and the labs. Do you try to play that game, or do you try to hire domain logic and lawyers and experience? Where has that worked, and where has it not worked?

Gabe: I think this was one of the big mistakes I made when we first started the company. Because I had worked in these labs, I knew a lot of the folks, and I would say, come work with us. And they were getting these $100 million-plus pay packages, and we weren't at the scale where that made sense. I would say we're now just getting to the company size where — we're still not competing with the frontier talent, but there are increasingly folks doing PhDs, folks who don't want to work at large labs. So I think that is changing.

The second thing is that a lot of why you needed that talent early on was that it wasn't just doing the post-training: you needed to build the training infrastructure, the serving infrastructure, and all of those things combined. The talent to do that was very unique. My old roommate was one of the folks who ran post-training at OpenAI, and he was one of the best researchers I've ever worked with. But now you can use things like Fireworks and Tinker-like APIs, so you don't need to build the training and serving infrastructure. I think it opens up the pool of talent, and it feels very feasible to get some of this talent now.

Speaker 6: Another question on benchmark creation. When you're creating these rubrics, you have to create them in such a way that they separate the frontier models, and you have to do that with the experts themselves. How do you navigate that challenge, especially when your in-house legal experts might not know how to design a rubric for a model?

Gabe: That's a good question. We think of it more as: how do we make them realistic representations of the work our customers are doing? Part of why we picked the legal domain is that if you just build a realistic client matter that these top firms are working on, the frontier models still can't do it. And the thing we've built over the past four years — my brother was one of our first hires, so he's been working with the models for four years, and his intuition for how the models work and how to generate these datasets is incredibly good. He's trained a bunch of lawyers to do this. But I'd say the focus is still more on how to build a really realistic data room, and then rubrics for the diligence. When we run the models, we find, okay, there are gaps and there's room for improvement. But I think it depends on the domain.

Speaker 7: Great talk, by the way. If you think of the end-to-end motion, there's data generation and the environment piece, then there's the algorithm piece — how do we train, the research piece — and then there's the infra piece. If something is not working, anecdotally, what have you seen? Operationally, dollar-wise, or human-hours-wise, is it usually in the data layer, the research layer or the infra layer? And who is orchestrating or debugging this whole end-to-end pipeline?

Gabe: This is a great question — this is a whole other talk. I don't think there's a single thing. What makes it easier for us is that we've found product-market fit. We have a product being used in production, and we did this largely with closed source models to start. That let us build a lot of confidence that the thing is working end to end: we have a product, people are using it, people have been using it for many years, the models work in this way, and we can monitor whether they're working in production. That was the point of the serving section: you need a lot of this in place. And then we've built the muscle of, when new models come out, how do we put them into the product, whether closed source or open source? So you can think about the full cycle. And obviously we've done a bunch of work on harness engineering and context management and all of these things.

So now you can think of post-training as one small input into this broader system. Our team post-trains a model, and then we feed it into the broader system and treat it as just another new model. Then you have all these signals for what's going wrong. Across these steps, there are easy ways to gate it. If you start with the benchmarks, and you train a model and it doesn't work well on those, it's unlikely to go farther. If it works well on the benchmarks and you put it in the product, someone might use it and say things feel weird — because there are definitely things our benchmarks don't catch. The best example: you can have a model that does very well on our benchmarks but has overfit to some degree, and then you put it in a more generic assistant-like product and it falls apart when you go out of distribution. You need all of the stages and gates in place, and then it becomes obvious where things are breaking. Good question.

Speaker 8: On the benchmark side: you had BigLaw Bench before, and now Legal Agent Bench. What's your philosophy on open sourcing the benchmark? If you open source it, you get traction with people trying the benchmark, but the labs can hill-climb on it as well. And if you close source it, there's a question of whether it's a real benchmark. What's your philosophy?

Gabe: That's a good tension we think about. I'd say that even before we open sourced this benchmark, we already worked closely with the labs: we share data with them to help them improve their models, which improves our product. The way we think about it is that there's a huge advantage in your industry — particularly in legal — in helping your customers understand how good different models are at different things. When I started Harvey, I had the intuition that we'd just build the best model and customers would be happy because it's the best. It's very clear now that every customer has different preferences, and different models are good at different things.

The second thing I didn't anticipate is how big the frontier ecosystem would become. Now we have so many companies reaching out saying, hey, we have this new technique, can we try it? Before, we just didn't have the bandwidth. Now, with this open source dataset, we can say: go try it, and if it works, great, this is something interesting to invest in. Strategically, the way we think about it is that the valuable data for us is going to be helping law firms train their own systems on how they work on private data. And then we want to help everyone improve these systems with synthetic data, some of which we open source. But obviously it's a balance, for the reasons you mentioned.

Speaker 9: What are the remaining open questions where you want help?

Gabe: That's a great question. One of the biggest challenges: we can generate these very realistic synthetic datasets, and we can augment them with humans, but the distribution of that data still doesn't match our production distribution. These datasets we're generating are somewhat future-facing. What I mean is, we can generate a really realistic data room, but right now our product isn't used purely to do diligence. So if a model does well on that, but then someone uses it to draft an email, maybe it doesn't work as well. There's still that gap, because we can't look at customer data. How do you bridge that gap? To me, that's the biggest question on the dataset side, and it's challenging.

Then there are a bunch of questions around, for example, the diligence data: the largest data room is 80 million tokens of contracts, and the models don't do a good job of managing this context. How do you train models to operate in these very complex environments? There's still a big performance gap.

And the third: we're making good progress on post-training models ourselves, but I think figuring out some form of continual learning is the endgame here. It's not for us to build the best legal model; it's for us to help every law firm or enterprise customer customize it to the type of work they're doing. Thinking about how to operationalize that — where a large law firm's AI system gets better every time they work on a client matter, while you're also protecting the client data — that's one huge technical, operational and AI challenge that is super interesting.

Speaker 10: From a product perspective, how do you think about competing with the labs' general products — Codex, Claude Code — given the open source models and infrastructure you can leverage? How would you prioritize: workflow, model quality, efficiency?

Gabe: The big shift we're thinking about is that our original product, and things like Codex and Cowork, are very individual-focused products: they're about individual productivity. Increasingly, the product we're building is about organizational productivity. If you think of a large law firm, the problem they're trying to solve is not how do I make my individual lawyers more productive. The problem is: I have 10,000 clients, I'm working on client projects for all of them, I need to make sure all those projects go very well, and I need to do them in a way that's profitable. A lot of the infrastructure we're building for law firms is how you operate that machine.

When you start thinking about an individual client project, most of the challenge is not how do I draft this one section. It's: I'm working on this project for six months, I have a team of 20 to 30 people across the firm, I need to coordinate all of them, and I need to coordinate all the outside parties. A lot of it starts to look like project management: orchestrating these humans and these agents. That's at the team level. At the organization level, you have a thousand of these projects, and you need to start thinking about resource allocation: what am I billing, what am I pitching? And with enterprises it's even more complicated: a Fortune 500 is working with a thousand law firms and has a thousand people internally. What are all the systems to start orchestrating that work? So the simple answer — and this is historically what enterprise software has done — is: go hyper-vertical into your domain in a way that the horizontal products won't. Good question.

Sonya: Thank you, Gabe. That was an iconic talk. Thank you for joining us.