How RL Environments Are Built, and Why They're Your AI Moat

Sonya: I think you grew from a one to a two billion dollar revenue run rate in the last four months or so, so this company is off to the races. And you have been so front and center to how companies are thinking about post-training their own models and building their own intelligence, so thank you for joining us for this conversation. Format-wise, we have 15 minutes or so of content from Brendan — he's going to talk about RL environments in particular, which is a new frontier topic that'll be fun to explore — and then we'll leave 15 minutes or so at the end for Q&A. So please keep questions in your back pocket. I'll turn it over to you, Brendan.

Brendan: Sweet. I'll be talking about RL environments. Starting out, I figured it's helpful to give a little background on the history of the data market and how that history ties into Mercor's origin story.

Things really started in 2020 and the era of crowdsourcing data for behavior cloning. This was mainly supervised fine-tuning data — inputs and outputs — and RLHF data, where an annotator would select which of a couple of model responses they preferred. We were able to make all this progress in fine-tuning GPT-3, making progress toward ChatGPT and GPT-4, in the crowdsourcing era of data.

What we saw changing in the market, especially heading into 2024, was a giant transition away from the low-skilled crowdsourcing era of behavior cloning data and toward the agentic era of data: how do we find the highest-skilled experts in the world — the software engineers, lawyers, doctors, bankers — who can work collaboratively in teams to build frontier evals and RL environments for the next generation of models, measure the frontier of intelligence, and use that to improve model capabilities? Mercor grew up with our first big project being deep research — I guess the first prominent RL agent — scaling up dramatically with all of the frontier labs to become the primary agentic data vendor to the leading labs, and also to the leading application-layer companies, ranging from Harvey, Sierra and Cognition to Ramp.

What's been really exciting over the last 12 months especially is how RLVR, within the agentic data paradigm, has evolved to also include RL environments: rich apps and worlds that teach agents how to use all the tools on our laptops that we use every day. I'll be talking about that, and about how this technology that started in the frontier labs is now getting disseminated to the application layer and all the products you're building.

At a high level, an RL environment includes three parts. The first is the worlds: all the messages, slides, docs, sheets and so on that correspond to everything you would have in a real project or company you're working on. The second is the apps: high-fidelity clones of popular applications — Salesforce, ServiceNow, Microsoft 365 — that agents can interact with via MCP, CLI or CUA. And the third is the tasks: prompts and verifiers. Verifiers could be rubrics or unit tests that can be used either for evals or for training.

The barrier for the frontier labs to automate everything you can do on your laptop using Claude is: how do they cover the full distribution of all the worlds, all the apps and all the tasks in the economy? There's been an enormous scale-out to do that, and humans have been really central to how we build these environments — obviously with models in the loop, meaningfully. I've put a graph here of the throughput from our talent network over the last 24 months, and it's a pretty crazy trajectory: 2.5 million expert hours in Q2 alone, with growth accelerating in the amount of expert time used to build out these environments.

The reason, as I mentioned, is that we need to scale out the environment distribution across every category in the economy. Many of you might know GDPval, where there are 205 domains in the Bureau of Labor Statistics across all the different jobs. But then you have to think through how to build all the apps corresponding to all those jobs, all the different scenarios, all the tasks. It's an enormous buildout.

And only humans can measure the frontier in most domains. Not every domain — there are rare exceptions like math, where you have a really clean simulation environment and the model can learn from whether it got the right answer. But in most domains, like building a slide deck, the model has an incredibly hard time reliably identifying where it made its own mistake. It's as if you asked a human to grade their own homework. That's why it's really valuable to have a human create a rubric, similar to how a professor would create a rubric to grade an essay, or a TA would grade that slide deck — similar to the way a lot of us learn, which is in large part from the feedback of those around us rather than purely plugging things into a calculator or clean simulation.

Building these verifiers is hard, because any time you're building that slide deck, you need to understand the full problem space: what are the ten different slide decks that could be a good path to go down, what are the dozens of mistakes you could possibly make, and how do you build a comprehensive verifier that captures the full [solution space] of what's possible?

So I'll walk through a sample RL environment to break this down. Part of the reason this is so cool, which I'll get to in a moment, is that we developed a lot of this technology in collaboration with the labs — these are, of course, ones we've open sourced and published to the world — and now it's all starting to get disseminated to the application-layer companies that are building and owning their own intelligence, as they realize that the three core pillars of their AI strategy are their compute, their algorithms (their researchers), and the datasets they build. And data is often the most differentiating factor.

This is one we published as a legal environment. We have lawyers from top law firms like Latham & Watkins write out a scenario from a real project they worked on in their Big Law job, and then create a full outline for a data room corresponding to all the different messages, emails, files, even file sizes. I cut off the full data room because it's very extensive. And of course there's a lot of model-in-the-loop in how they populate this — similar to how a software engineer now shouldn't be coding entirely by hand; they should probably be orchestrating agents to do this very productively. Then we render that data room into the apps — the clones of Google Workspace you can see in this scenario — and have prompts to roll out model trajectories against it. In this one, it's evaluating the maximum total liability for Sterling Oil Tankers International Limited compared to Cooper Jefferies Energy Corporation under the Oil and Petroleum Act, considering all the context from this scenario in the data room.

Then, as I mentioned, similar to how a professor would create a rubric to grade an essay, they have rubric criteria corresponding to the characteristics of an accurate model response, making sure those criteria avoid reward hacking and effectively align with the goal. When you roll out 100 trajectories, making sure all of those scores are accurate is incredibly technically challenging. There's an enormous amount of research, agentic quality control, training on the data and so on that goes into solving that problem, and then ultimately producing these high-quality verifiers and leaderboards that give you an aggregate model score for how well the different models are doing in a particular domain.

As we can see, one of the big changes over the last few months is that GLM 5.2 and K3 are on the leaderboard. That is a huge opportunity for all of you, because it gives us the foundation to actually achieve frontier intelligence in the specific applications and verticals you're focusing on. That's not too far away.

To give a little context on what that looks like, I'll share an example of post-training on APEX Agents, which is the dataset from the sample I just showed. We have 1,800 tasks in this example. This was a post-training run of GLM 4.7, but we're redoing a lot of them for Kimi K3, so we'll have updated results for you soon. You can see the jumps just on 1,800 tasks, with about $500K in compute, are pretty dramatic: corporate law going from 4.7% to 26.6%. And notice that this is just the APEX Agents dataset we gave it — it actually generalized incredibly well to [GDPval] and APEX v1, which doesn't have these data rooms, and even saw nominal gains on some other benchmarks as well.

We're doing a lot of this work with customers like Harvey — who I know will present later — to help build out the environments corresponding to their specific domain so they can build frontier intelligence within it. I think [Andrew] talked about how Cursor was a great first example of an application-layer company building an industry-leading model that created enormous value for their customers. I believe that over the next 12 months there are going to be dozens of examples just like that, where companies own their own intelligence — and that is the key source of the moats they're building.

Josh and I talked about this the other day as well: a couple of ways to curate high-quality datasets. The general three we see most, and I'm happy to talk about these and send people links: First, by task. This is the most common, where people say, I really like this data shape of environments in law, and we'll pay $2,000 per task to scale it up. As an example, certain frontier labs might buy 50,000 tasks a month from us, so it tends to be pretty dramatic scale. These tasks are generally very complex — some take humans up to a month to complete, some just a few hours — and it's custom per-task pricing. Second is off-the-shelf data, where we've invested hundreds of millions of dollars building our own datasets that we sell to multiple customers. The neolabs generally err toward off-the-shelf data, because it doesn't make sense for ten different labs to all build their own datasets, and there's a lot of value in building something once that can be applied to everyone. And the final one, which we see a little of but is less of our focus now, is just providing the experts, so customers can organize them on their own in an hourly model. We do a little of that — that's actually how Harvey got started with us, hiring some lawyers — but it generally moves toward the more scaled data offerings over time.

So that's a little background on RL environments: what they are and how to build them. I'm really excited that all this technology, which was previously limited to the frontier labs, is making its way to all of you. Happy to answer any questions. Sweet — go ahead.

Speaker 3: My question is open-ended but simple: how do you price data? How do you value data?

Brendan: There are so many different ways. The most natural: our customers care about model improvement, right? Our customers have a goal — they want to be at the frontier on a given leaderboard — and we're able to work backwards from how much that's worth to them, how much we should charge per task, and how many tasks we think would get them to that goal. When we think about a company like Nvidia, they're probably willing to pay a billion dollars to have a frontier open source model, and so there's a lot of complexity in how we price all the different ingredients that go into making that happen. The other lens we look through is our cost structure: when we have a task that takes ten hours of human time and we're paying the human $150 an hour, there might be a $1,500 cost basis, and then it becomes a question of what margin we want to run on top of that, based on how differentiated and frontier that specific task is. But it's a super wide range: we have tasks that range from $50 to $10,000.

Speaker 4: How do you think about data quality? You mentioned utilizing human experts to label data — how do you compare human-labeled data and the frontier LLMs' judgment data?

Brendan: So the first question was how we think about quality; the second was how we compare the judgment of preference labels to the autograders. When people say data quality, they're generally referring to two things: first, realism; second, accuracy of verifiers.

On realism: the labs want to automate everything in the economy that corresponds to, in this case, corporate law. So how do we make sure this actually reflects the real distribution of what we'd see in a real lawyer's environment? That's one reason experts create outlines and guide the data curation process. Realism of the environment, the apps, the tasks — everything is incredibly important, along with granularly understanding the taxonomy that drives that realism across the entire distribution you're looking for.

The second part relates to the accuracy of the verifiers. The way you'd train one of these models is you might roll out 100 trajectories of K3 and use the rubric to score all of them. As you can imagine, there are so many different paths a model can go down, so you want to make sure the rubric is scoring them the same way as if humans had stack-ranked those 100 trajectories. What we do for that is a process called trajectory analysis: we roll out ten trajectories of the model we're focused on improving, score all of them, and have some combination of agentic quality control systems and human review go through to make sure the scores align with the goals. Sometimes you can also use human feedback evals, or preference labels, as an eval for your autograder — that's the other way to solve for it. Go ahead.

Speaker 5: How much do you think synthetic data generation plays into all of this, especially creating large data rooms?

Brendan: The fascinating thing is, I think there's been a lot of misinterpretation of what people mean by synthetic data — because RLVR is a bet on synthetic data. It's basically: let's roll out a bunch of synthetic model trajectories rather than having humans write the SFT, then score all of them and let the models learn from those synthetic trajectories. That's the first way synthetics get used. The second way is that models play a giant role in how we populate environments and create tasks — the same way a lawyer writing a legal memo should definitely be using [Claude] or ChatGPT, the experts building out these data rooms should definitely be using [Claude], ChatGPT or whatever model to help them. There are a lot of ways the model makes them more efficient.

But the reason humans are still an essential, incredibly differentiated component of the process is that you need humans, almost definitionally, to measure what is beyond the frontier of the model's capabilities. You can't just tell the model: come up with the legal environment, and then tell me which of your legal memos are good and bad. It's super noisy, and there's no clear signal in that. You need something with capabilities beyond the frontier of that model to do it reliably.

Sonya: Thank you for doing this. My question is: RL environments seem like they're all the rage now, and maybe have been for about a year — I hadn't really been hearing about them prior to that, and it was all human expert labeling. Why is it all about RL environments now? Is that the most relevant thing for application companies to be thinking about? And is there something after RL environments?

Brendan: I'll start with why it's become the rage, and maybe some differences between the deep research paradigm and the environments paradigm we saw in 2025, and then talk about what we see evolving in the data landscape looking forward.

I think the reason the deep research environments came first was that deep research had tool use with search. Search was the tool in the environment the model worked in, but the experts wouldn't necessarily be populating apps — it was a lighter version of an RL environment, where they'd just create rubrics corresponding to it. And I only talk about this stuff because it's a couple of years old at this point, so it's no longer super confidential. Then, for the trend of apps in 2025: I think that became giant because people realized the primary bottleneck to making models useful was how they used all the context in the code base and all the tools on our laptops. If we want that in the user distribution of usage, we need to get it into the data distribution the models are learning from. So there's going to continue to be a giant scale-up of diversity across all three of these categories going forward.

But there are going to be some changes. To name the two we're thinking about most: the first is ultra-long horizon. Right now, agents mostly aren't trained to do things that take over ten hours, and we need to start building tasks for things that might take a human 100 hours, or even 1,000 hours. That's going to be a giant shift. The other large shift is introducing virtual coworkers. One of my favorite questions to ask people when they're thinking about their data distribution is: what percentage of the tasks you do in your job require interacting with other people? Most people say 60 or 70% — some more, some less. But if you map that onto what percentage of evals measure how well models interact with other people, it's like 1%. Maybe tau-bench has a little of this. So there's a giant real-to-sim gap in how you measure how well agents engage in social interaction with all the people and other agents they need to work with in their jobs. Go ahead.

Speaker 6: You talked about rubric generation, which gives you a bespoke axis of verifiability per task, and from what I understood, that's bottlenecked by experts. Have you found any success scaling that up with models giving you some heuristic, or even better, post-trained models?

Brendan: We have found that you can make it a lot more efficient if you have an AI copilot that works with the expert in creating the task and the verifier, so the expert can talk to the trajectory and understand exactly what's happening and where it's going wrong. The challenge is just that if you're trying to improve Fable, Fable cannot reliably write out the rubric criteria for where it's making mistakes. It might get half of them right and half wrong, and that amount of noise is unworkable from a training standpoint. That's why task creation is the part of the process that requires humans the most. For a lot of the environments, we can use a lot of synthetic — it's helpful to have humans write the outlines, since they're familiar with the environment and grounded in a realistic distribution — but the tasks really tend to require humans. With rare exceptions in code, or if you're distilling: if you have a model that's worse than K3, it can definitely learn from tasks K3 is creating. So there are some exceptions if you're doing it that way.

Speaker 8: When we think about RL environments for certain provable domains like cyber defense or incident response, where the model or agent is trying to find a flaw in an existing system: do you use humans just for authoring or setting up that environment, or also for grading? Or is there a way to scale that up?

Brendan: I actually think cyber is one where you don't necessarily always need humans for the verifiers, because you can have an attacker and defender agent. And I think you're right that for cyber, you can use humans more for what a realistic environment looks like, and for setting up the environment, because you do need a lot of diversity. But it's less human-intensive with respect to building verifiers.

Speaker 9: How can you tell when you're limited by the base model?

Brendan: What do you mean by that?

Speaker 9: Well, ostensibly you're using the same dataset for all these models here, and they somewhat land around the same final performance on this list. But maybe if you tried a smaller model, which might be a good place to start, it would land much lower. What's the cause there — is it just parameter count?

Brendan: Parameter count will definitely play a role in how effectively the model does, insofar as how trainable it is. But the main thing to look at is generally the gap between pass@16 and pass@1. If you have a model where you roll out 16 trajectories and it gets all of them totally wrong, it's sort of hopeless that the model is going to learn from that — maybe you roll out another 100 trajectories and it gets one right. The ideal case is that at pass@1 it fails, but at pass@16 — when you roll out 16 trajectories — it gets it right once or twice. Then the model is able to learn very effectively. That's generally the heuristic we use for how strong the base model needs to be to learn effectively from a given dataset. Cool — final question, really quick.

Speaker 10: What advice do you have for companies on data they should build themselves as part of their RL post-training, versus relying on a partner? What's complementary?

Brendan: I think this is why a giant portion of our business is custom data, where we have teams that are siloed and fully exclusive to critical customers, to make sure we build the best datasets in the world that they own. That allows them to maintain their competitive advantage while also benefiting from all the infrastructure we've built. There are some companies that try to build out the entire talent network and infrastructure in-house. But if you look at what the frontier labs and the best models do, it's a pretty good indication that there are so many economies of scale from working with a partner that has them: the platform, the talent network, and so on. So anyways — thanks for having me.