Demo
Run your model on a Scorekey environment
Three tasks from the online goods market environment: the full build from the brief, one stage the frontier models pass, and one stage they do not. The packet is a public repository: the three tasks with the lists of values an answer may use, the 64 data files the model may read (about 120 MB), and the runner that gives your model exactly what the models on the Research page were given. Your model's code runs in a sandbox with no web access; the runner itself only calls your model's API and our scorer.
Get the packet on GitHubRun it in two commands
python3 run_demo.py --packet . --model <name> --out runs/<name>
python3 submit_demo.py --answers runs/<name> --url https://scorekey-env0.fly.devLinux, run as root (a VM or a privileged container), Python 3 standard library only. Your model API key and your scoring token go in two environment variables; the README names them.
The packet is open; scoring needs a token. We issue it by email to your work address. It allows five submissions per task, and a rejected submission counts as one.
What comes back
The scorer returns one score for each of the two stages and two for the full build, sources right and figures right, net. It does not say which figures were right.
How the frontier models did on the same tasks
| Task | gpt-6-astra | claude-fable-5-1 | gemini-3.8-flash |
|---|---|---|---|
| Extract the figures | gpt-6-astra0.97 | claude-fable-5-10.97 | gemini-3.8-flash0.90 |
| Pinpoint the exact series | gpt-6-astra0.88 | claude-fable-5-10.96 | gemini-3.8-flash0.00 |
| Full build from the brief, sources right | gpt-6-astra0.87 | claude-fable-5-10.87 | gemini-3.8-flash0.30 |
| Full build from the brief, figures right, net | gpt-6-astra0.50 | claude-fable-5-10.43 | gemini-3.8-flash0.17 |
Each figure is the mean of three runs; 1.00 means every answer right. gemini-3.8-flash ran out of turns on all three runs of “pinpoint the exact series” and made no submission on two of its three full builds, which count 0 on sources right. On this environment, figures right from the brief is the mean of two parts: the final figures, which were 0 on every run, and the datasets the build does not use. Calculations are not scored from the brief, where only the final figure is submitted. The Research page explains each measure.