Scorekey

How we build an environment

A Scorekey environment is a complete analytical assignment, built so that a model can do the work and be scored on it. An experienced analyst owns the reference analysis and every decision inside it.

What goes into an environment

The assignment. Each environment starts from an assignment of the kind a company or a client gives an analyst: a question, the material the answer has to come from, and the form the result has to take. The material is fixed as a set of files shipped with the environment; nothing is fetched live.

The connected decisions. Carrying an assignment from its material to a result means a chain of decisions, each resting on the ones before it. In a market build, for example: what exactly is being measured, which sources to build from, which figures to read in them, what to leave out, how to fill what the sources do not cover, how to combine the parts. The stages and the decisions differ by assignment; what every environment captures is the whole chain, so that an error early in the work shows up in the result the way it would in practice.

The reference analysis. Before any model runs, the analyst carries the assignment through and produces the result. That becomes the answer key: the result, every intermediate answer that leads to it, and the assumptions behind them. Where the brief or the material leaves a choice the analyst has to make, the key records the decision, the alternative and the reason. The record ships with the environment, so a buyer can see what is scored and why, and challenge any decision in it.

The grader. Every answer has a fixed form, so the grader is code: an answer is credited only if it matches the key, a figure within one part in ten thousand. There is no rubric, and no model judges another model's work.

How we build it

One analyst owns each environment from start to finish: someone who has done this kind of work in practice, inside companies or for clients. The same person defines what is measured, chooses the sources, makes every recorded decision and builds the reference result. Because the same person makes all of them, the reasoning behind each decision is present when the next one is made, and one person answers for the analysis as a whole. Whether it holds together is then tested before release. Where a decision is open to question, the analyst puts it to another experienced analyst who has not worked on the environment before fixing it.

We use AI agents, in separate roles, to build the environment itself: the stage tasks, the key files, the grader and the runner that delivers the tasks to a model. Nothing an agent produces enters the key unchecked. Every figure in the key is computed from the shipped files by the release's own code, and the release gate re-runs that computation on every release, so a figure that cannot be reproduced from the material does not ship. Before release, a separate agent that did not build the environment worked the stage tasks as a candidate, from the shipped material alone; wherever its answers differed from the key, the analyst settled which was right. The same audit tested the opposite, whether the tasks could be answered without opening the material; where they could, the tasks were rebuilt before release. The released files carry checksums, and the decision record cannot change without the release changing.

What researchers can learn from it

An environment is run in two ways. As a complete assignment, the model is given a starting point, from the brief alone to a point where only the last steps remain, and asked for the final result. In the published environments it is scored on whether it used the right sources, whether its own calculations hold together and whether its final figures match the key, as separate scores; the measures follow the assignment. Stage by stage, each step is run on its own with the earlier steps' verified answers supplied, so a score points to where in the work a model fails. A model can score well on every stage and still fail the assignment, which is why both are run.

A score is the share of answers credited for agreeing with the reference analysis, under the rules in the technical notes. Reading why an answer missed is a separate step, and the record makes it possible. After a study, a reviewer reads every transcript against the record and sorts each miss: the model had what it needed and got it wrong, or it decided a recorded decision the other way, or the material we served left it no way to know, or a technical event stopped the run. Where the record cannot settle a miss, it is counted as not established. The sort never changes a score; it changes how a result is reported. Where a model decided a recorded decision the other way, the record shows what the key decided and why, so a reader can judge the decision for themselves. Where the material we served was at fault, the score stands, the defect is recorded against that release, and the fix goes into the next.

One example

In the published market sizing environments, the card payment statistics count every card payment, in stores and online, and carry a column that says which. The brief asks for goods bought online, so the right series has that column set to remote, and the key records it with every other dimension of the series. A model that leaves the column at "either" reads a real series, but a different one: it counts in-store purchases as online, and every figure built on it is off by that amount. The pin is scored at the stage where the series are chosen and again in the full build. In the study, one model left it at "either" on both card files in one run. The full study of five models is on the Research page.

Technical notes

Scoring rules in the published environments. A stage score is the mean of its questions, each weighted equally. Wrong answers count against right ones: on a question that asks for figures, a wrong figure, a wrong gap or an invented figure cancels a right one; on a question that asks for a set, a wrong name cancels a right one, and naming everything or nothing scores zero; on a question with one answer to pick, what guessing would earn is subtracted first. No question scores below zero. A figure matched to the key is credited within one part in ten thousand; the check on whether a model's own calculations hold together uses one part in a thousand, since it tests the model's arithmetic against its own inputs. A complete assignment returns separate scores for sources, calculations and final figures, each applied only where the starting point leaves that work to the model. Where the key records that a figure can be recovered from other figures in the material, it expects the recovery and a declared gap is wrong; where it records that it cannot, the gap is the right answer. A commissioned environment is scored by rules agreed in its specification.

Releases. A study's scores are the record of that study on that release and are never re-scored. A problem found afterwards is recorded against the release and corrected in a successor with a new id. One case from before publication: the first run of the study on the second environment gave each run one working copy of the files for all its tasks, so a task could open the tasks meant for later stages and read in them the verified answers to its own stage. The runner's transcripts showed it, every result of that run was marked invalid and never quoted, the runner was changed to one sandbox per task with a check that refuses to start if anything else is present, and the study was re-run on the next release.

Working with us

We license existing environments and build new ones to a specification: the assignment type, the material, the decisions to be scored, and the form of the result. A new environment is built by the same process, by an analyst who has done the work, and goes through the same release check before delivery.

contact@scorekey.ai