Demo
Run your model on a Scorekey environment
Three tasks from the online goods market environment: one stage where every model scores 0.90 or better, and the full build from two starting points, the brief alone and the plan, which none of them gets right. The packet is a public repository: the three tasks with the lists of values an answer may use, the 55 data files the model may read (about 120 MB), and the runner that gives your model exactly what the models on the Research page were given. Your model's code runs in a sandbox with no web access; the runner itself only calls your model's API and our scorer.
Get the packet on GitHubRun it in two commands
python3 run_demo.py --packet . --model <name> --out runs/<name>
python3 submit_demo.py --answers runs/<name> --url https://scorekey-env0.fly.devLinux, run as root (a VM or a privileged container), Python 3 standard library only. Your model API key and your scoring token go in two environment variables; the README names them.
The packet is open; scoring needs a token. We issue it by email to your work address. It allows five submissions per task, and a rejected submission counts as one.
The scorer returns one score for the stage and, for each build, sources right, calculations hold and final figures right, net. From the brief, the scorer returns final figures right, net as one combined figure for the final figures and the datasets correctly left out; the table below shows the two parts separately from our own records. Scores are per task, not per figure.
How the models did on the same tasks
| Task | gpt-6-astra | claude-fable-5-1 | gemini-3.8-flash | Mistral Medium 3.5 | grok-4.7 |
|---|---|---|---|---|---|
| Extract the figures | gpt-6-astra0.97 | claude-fable-5-10.97 | gemini-3.8-flash0.90 | Mistral Medium 3.50.92 | grok-4.71.00 |
| Full build from the brief, sources right | gpt-6-astra0.87 | claude-fable-5-10.87 | gemini-3.8-flash0.30 | Mistral Medium 3.50.00 | grok-4.70.21 |
| Full build from the brief, final figures right, net | gpt-6-astra0.00 | claude-fable-5-10.00 | gemini-3.8-flash0.00 | Mistral Medium 3.50.00 | grok-4.70.00 |
| Full build from the brief, datasets correctly left out | gpt-6-astra1.00 | claude-fable-5-10.86 | gemini-3.8-flash0.33 | Mistral Medium 3.50.00 | grok-4.70.76 |
| Full build from the plan, sources right | gpt-6-astra1.00 | claude-fable-5-10.93 | gemini-3.8-flash0.09 | Mistral Medium 3.50.02 | grok-4.70.66 |
| Full build from the plan, calculations hold | gpt-6-astra0.60 | claude-fable-5-10.73 | gemini-3.8-flash0.40 | Mistral Medium 3.50.57 | grok-4.70.60 |
| Full build from the plan, final figures right, net | gpt-6-astra0.00 | claude-fable-5-10.00 | gemini-3.8-flash0.00 | Mistral Medium 3.50.00 | grok-4.70.00 |
Sources right is the share of the 21 series the build reads that the model named exactly, every dimension fixed to the one value the build uses. Calculations hold is the share of the 10 relations between the figures the build produces that are consistent in the model's own answer. Final figures right, net is right final figures minus wrong ones, floored at zero. Each figure is the mean of three runs; 1.00 means every answer right. gemini-3.8-flash made no submission on two of its three builds from the brief and on one of its three from the plan; those runs count 0 on every measure of that build. From the brief, the five final market figures are scored separately from the datasets the build correctly leaves out; none of the five figures matched the key on any run of any model. From the plan, the model is handed the verified structure and sources and scored on what it computes from them; no run of any model produced a right final figure. Calculations are not scored from the brief, where only the final figures are submitted. The Research page explains each measure.