What is the value proposition of "RL environment" companies? [D]

[u/arnavneedshelp]

Recently I've been getting bombarded with reach outs from companies that build RL environments and sell to frontier labs like Anthropic, OAI, etc. I'm not really understanding how and why these companies exist.

Lets say you somehow create an environment that no other model is able to perform well on. Your second criteria would be that this environment is suitable for RLVR, perhaps you have some metric of gradient information per example in the dataset. So you have to prove to the lab you are selling this on, that 1) their model cannot do this task currently and 2) this task is solvable via RL.

At that point, why couldn't you just approach a VC (most of these companies are startups with a good amount of funding anyways), show them scaling laws + your current RL env and just raise money to train your own model? We already see open source models tend to lag behind frontier ones by ~3 months or so, it sort of implies the frontier is mostly new RL environments and stuff (or distillation methods are really good). I'm kinda operating under the impression that pre-training/architecture/infra for large models is largely much easier nowadays, but maybe this is just not the case?

I guess an answer to my question would be either 1) I am totally wrong regarding my assumptions on the difficulty of pre-training + infra (which from my experience in industry does seem to be much easier) or 2) actually collecting all of these other, past, RL environments is still hard and only certain labs have access to all of them or 3) all of these companies are or eventually will train their own models?

[u/Bardy_Bard]

I believe you need a lot of different RL models to get that generalization going and it’s hard to do them.
So unless you have access to the already built RL models you just have a RL model + open source to draw upon

[u/Disastrous-Shoe3450 → u/Bardy_Bard]

The "just raise VC money and train your own" route sounds cleaner on paper than it is in practice. The compute cost alone for a frontier-level training run is eye-watering, and securing that kind of cash means convincing investors you can compete with labs that already have billions and established talent pipelines. Selling environments to those same labs is a much leaner business, you're monetizing the one tricky piece you're good at without betting the farm on becoming a full model provider.

Also think about the data flywheel the existing labs have. They've got years of proprietary RL environments, human feedback, and internal tooling that a startup can't replicate overnight. Being 3 months behind on open source benchmarks is one thing, closing the gap on a sprawling suite of internal evals is a different beast entirely. These companies probably exist because the labs would rather buy a novel, well-crafted environment than pull engineers off their main research to build yet another sandbox from scratch.

[u/Pale-Training-6944]

My suspicion is that many of these companies are transitional businesses built around the current bottlenecks in model development. Selling environments to labs makes sense today, but I’m skeptical about how much lasting bargaining power the suppliers will have.

The more interesting path, IMO, is to build an AI product around the workflow they understand. That doesn’t require training a frontier model from scratch. If you can build a useful product with existing models, you get a continuous stream of real tasks, failure cases, and customer feedback. You also capture value from actually solving the customer’s problem, beyond whatever a lab will pay for training inputs.

Of course, production data doesn’t automatically become a resettable environment with reliable rewards. There’s real engineering value in that conversion. But without exclusive access to workflows, domain expertise, or customers, what prevents that work from becoming commoditized or moving in-house?

I also suspect part of the appeal for teams coming straight out of PhDs or research labs is that selling environments offers a quicker way to monetize their existing expertise. Building and distributing a product requires a different skill set. My concern is that the immediate revenue opportunity lets teams postpone developing those capabilities, leaving them dependent on a few labs’ changing training priorities. That can be a good business for a while; I’m less convinced it’s a durable company.

[u/GuessEnvironmental]

Pre training is expensive and post training is expensive if you want the model to be as general as possible. Also designing the rl environments is also really hard and expensive depending on what it is built for it is a lot of trial and error and designing things that actually improve that model capability. Why don't these rl environments companies build their own model is some ways I agree but not to necessarily scale their own based model I suppose but instead make a model that specifically does that task however you have to prove that task is something valuable. It is selling the shovels Vs digging for gold. I was at a company selling these rl environments and it prints money.

[u/InterestingCarrot997]

my guess is it's mostly the reward design and the grading, not the sandbox itself. building an env where the model cant just game the checker is way harder than it sounds