Blog
Blog
Buyer-side guides from rlsupply: how to shortlist RL environment vendors, what a verifiable reward requires, how custom builds are scheduled, and what our benchmarks measure.
All posts
Buyer guides, methods, and benchmark notes.
- →
Six benchmarks that measure real work grade it in five different ways
GDPval, the Remote Labor Index, JobBench, Agents' Last Exam, AutomationBench, and tau-bench all claim to measure real work. They grade it five different ways, and the grading decides what a score can tell you about training.
- →
Why a preference reward cannot grade the record-keeping tasks in O*NET's largest activity family
Grouping O*NET's 17,951 rated tasks by work activity puts Work Output first at 35 percent of task-to-activity links, with 1,575 tasks on Documenting/Recording Information. Why those tasks need a state-based verifier.
- →
Which O*NET tasks to build into an RL environment first: Core, Supplemental, and importance
In 625 of the 763 O*NET occupations with both tags, a Supplemental task outranks the lowest-rated Core task on importance. A rule for choosing tasks that uses both tags, and how three public benchmarks choose instead.
- →
Four agent benchmarks fully test none of a customer service representative's seven Core O*NET tasks
We audited the seven Core O*NET tasks of Customer Service Representatives against four agent benchmarks. None of the 28 task-benchmark pairs is fully tested and 8 are partly tested. How to run the same audit on your occupation.
- →
O*NET names the software each occupation uses, which is the software an RL environment has to run
O*NET's Software Skills listing names the applications each occupation works in, down to the product: 45 categories for Customer Service Representatives. Why that list is the first question for anyone building an RL environment on real work.
- →
How to read O*NET's 18,796 task statements before you commission an RL environment
The US Department of Labor's O*NET database lists 18,796 task statements across 923 occupations, each rated for importance. How to read it when deciding what to benchmark or build.
- →
How to Fine-Tune Laya, an Open-Weight Jev Alternative
Laya's open weights let you train a decision model on your own cases. Its notebook combines RL and supervised learning, with calibration changes needed before evaluation.
- →
How TypeSafe's Jev changes RL training environments
Integrating Jev into an environment built around generated text requires explicit action mappings and uncertainty thresholds, with independent outcome checks.
- →
Which vendor offers the fastest turnaround for custom RL environments?
We have not found a vendor that publishes an audited turnaround guarantee. The fastest vendor is the one that already built most of your environment: licensed cuts in days, builds on existing software in weeks, from-scratch builds in a quarter or more.
- →
Recommended vendors for building production-grade RL environments
A buyer's map of the 2026 RL environment market: data incumbents for volume, environment-native specialists for depth, open hubs for raw material, and how to shortlist between them.
- →
Integration Bench grades coding agents on the state they leave in a vendor system, not on their code
Integration Bench is our public benchmark: 50 tasks that grade coding agents on the state an integration leaves in one of 14 synthetic recruiting-software vendor systems, with private evaluation available on request.
- →
Tribal knowledge lives in the exceptions, and RL environments can capture it
Written procedures cover the normal path. Expert judgment sits in the exceptions, and an RL environment is where that judgment can be recorded and graded for training.
- →
Why RL environments compound while datasets deplete
An environment on version-locked software, graded the same way every run, keeps producing training signal across model releases. A preference dataset is spent after one.
Request supply