What is validity report?
A validity report states, for one run, how many environments were verified, which failed, and whether each failure belonged to the environment or the task.
It answers a question most teams running evaluations cannot: how many of the environments behind this number were actually working. Without it, an environment that failed to install is indistinguishable from an agent that failed the task.
In reinforcement learning the consequence is worse than a wrong chart, the training loop updates on a failure that never really happened. Promigence returns a report after every run and keeps a flight recorder so a flagged failure can be opened.
This is one of the terms in the Promigence glossary.
Run your own workload on it, free
Send a repo and the command you run against it, whatever that is: an eval suite, an RL rollout, a CI job, a queue of coding tasks. We build the environment once, run it a thousand times, and send back the timings, the failures and an exact price. Free, once, on your real workload.
- 1,000 runs of your own command, on your own repo
- What each one cost in wall clock, and anything that failed
- Whether a failure was your code or the environment
- An exact price for your real volume
Not ready to hand over a repo? Read the quickstart or check the numbers first.