Scorekey Market Sizing Environments
Evaluating frontier models on complete market sizing builds, every answer scored against a key
A Scorekey environment is a complete market sizing build from public data, with a key that holds the right answer for every cell. The build is tested in eleven pieces: the eight stages of the work, each asked on its own with the earlier stages' answers supplied, and three full builds from different starting points. An answer is credited only if it matches the key within one part in ten thousand. Nothing is scored by a rubric or by another model. Environment 1 is the online goods market in the public demo; Environment 2 is a second market question built the same way.
We ran gpt-6-astra, claude-fable-5-1 and gemini-3.8-flash on both environments, three runs of each of the eleven pieces. The models pass the execution stages, where the inputs are fixed and the answer follows. On the two stages that decide what the market is, they score 0.43 to 0.61. Asked to build the whole thing from the brief, no run on either environment produced a right final figure, and the typical figure was 33% to 45% off.
Below: what the environments cover, one stage walked through, how scoring works, the results, and what comes next.
One run, the whole build, three checks. Three runs per model.
Build from the brief only
Build from the plan and datasets
Build from the extracted figures
Where frontier models go wrong
- Finds no second route: When the cell it needs is suppressed or missing, the model reports "not available" instead of finding the figure through what the sources do publish. Nine of nine runs on one such case.
- Builds the market from one table: It sizes the part of the market that sits in the largest table and leaves out the parts that sit in other files. Five of the eight runs that submitted built Environment 2's market from a single source that covers one part of it; gpt-6-astra named all five sources on every run.
- Picks the wrong of two similar series: Statistical sources publish the same activity measured from different vantage points, because each answers a different question, and a market as defined needs exactly one of them. The model picks the other. gpt-6-astra did so on every Environment 2 run of stage 3, pinpoint the exact series, and most full builds from the brief did the same.
- Skips exclusions and counts twice: It does not exclude what the brief puts out of scope, and it counts things twice: a payment code that belongs inside the base count classified as outside it, a cross-check dataset declared as a source, exclusions it was handed not applied.
- Reports a partial figure as the total: Where one input needed to complete a figure is not published, the model reports what it could build as if it were the whole, instead of declaring that the total cannot be completed. On Environment 2 from the brief, five of the eight runs that submitted did this for a country whose total cannot be finished from the public record; gpt-6-astra declared the gap on all three of its runs.
1. Coverage
The eight stages are the stages of sizing a market in a commercial due diligence, in the order the work is done, from the brief to the figure that goes to the investment committee. Each environment takes one market question through all eight, about 400 decisions on Environment 1 and about 250 on Environment 2.
Get the data
- Structure the model. Fix what is being sized (product, buyer, channel, geography, period) and the formula that produces it: volume times value, spend read directly, or supply side. Every later stage supplies an input to this formula.
- Select the sources. Decide which of the candidate datasets the build stands on. Every model has more candidate sources than it uses, and the wrong choice is the most expensive error in the build.
- Pinpoint the exact series. Statistics are published in their own categories, not the market's. Pin every series to read, dimension by dimension, and where the sources publish codes, decide for each whether it is inside the definition, outside it, or already counted elsewhere.
- Extract the figures. Pull the raw numbers at the pinned addresses and deal with suppressed or missing cells.
Refine it to the market
- Fill the coverage gaps. Every dataset covers the definition only partly: one payment method, some countries, a sample, a year. Derive the factors that cover the rest, from related sources, by a stated method.
- Set the market perimeter. Every source measures a population, and the market's perimeter rarely matches it exactly. Decide how the perimeter maps onto the measured population, and the share of the measured activity that lies inside it.
- Finalize the inputs. Turn the extracted figures into the inputs the formula needs: establish what falls outside the definition and take it out, and bring what is only partly covered up to the whole, with factors from stage 5 or established here.
Result
- Compute the result. Bring the inputs together, apply the exclusions and the stage 6 share, run the stage 1 formula, and declare where an input does not exist.
How much each stage weighs depends on the market; none can be skipped without saying why.
How it is tested. The build is tested stage by stage and as a whole. Stage by stage, each of the eight is asked on its own, with the earlier stages' verified answers supplied, and every answer inside it is scored: 324 assessed answers across the eight on Environment 1, from 5 on the market perimeter to 105 on the exact series. As a whole, the model is asked for the complete build three times, from the brief alone, from the brief with the build plan and the datasets, and from all of that with the extracted figures; each is scored on its sources, its calculations and its final figures, 40, 56 and 35 assessed answers. One run of all eleven is about 1,240 decisions and 455 assessed answers on Environment 1, about 760 and 319 on Environment 2. Decision counts are the builder's count, rounded; assessed answers are the grader's exact count.
2. Sample stage: pinpoint the exact series
This is stage 3 of 8 in Environment 1, the online goods market.
What the model is given. The brief: a client wants the market for physical goods bought online in five European countries, one figure per country, reference year 2024. The data landscape, the files the model may read: thirteen extracts of the ECB's payment statistics, three waves of a consumer payment survey with their codebooks, and the codelists, about 120 MB, read-only in a Python sandbox with no web access and at most 120 turns. The ECB's payment statistics count every card payment, transfer and other payment made in the euro area, and publish the counts broken down by many dimensions at once: who paid and who received, where each of them was, how the payment was started, the payment scheme, the type of card, and the reporting period. A single figure the build needs is the count at one combination of those dimensions, and the whole build's precision comes from choosing the right combination for every series it reads. The menu of every dataset, column and value an answer may name. And the verified answers of the two earlier stages, the build's structure and which datasets it uses per country, so an error in an earlier stage does not count against this one.
What it has to decide and submit. Two things, 105 answers in all. First, for each of the five datasets the build uses, the exact series to read: every dimension the file carries fixed to one value (frequency, counterparty area, terminal location, transaction type, channel, remote or not, scheme, card function, unit), 21 values. Second, what counts as the market, code by code: each of the main source's 58 merchant categories classed as goods, digital, fuel, services or money; 14 of the survey's 15 purpose codes as physical goods or not (one is not scored); each of its 12 payment-method codes as already inside the base count or not. One JSON object, one answer per question. Two answers to one question count as a format error.
What the grader checks. Each of the 105 answers against the key, exact match. The score is the share right, with no credit for what guessing would earn, so one wrong classification costs more than its one cell. There is no partial credit on a series: one dimension wrong is a wrong series, and naming a column the dataset does not carry is charged as wrong.
What a wrong answer looks like. A series one dimension off, usually the frequency or the counterparty area (where the other side of the payment is located: same country, or abroad), which selects a real series that is not the one the build needs. A merchant category put on the wrong side of the goods line, or a survey purpose code counted as goods when it is not. A payment-method code classified as outside the base when it is already inside it, so the purchases it covers are added a second time.
What the three models scored. On this stage, gpt-6-astra scored 0.88, claude-fable-5-1 0.96 and gemini-3.8-flash 0.00, out of 1.00. The zero is three runs that ran out of turns without submitting. gpt-6-astra's three runs got two, four and five of the 105 answers wrong; claude-fable-5-1's got one, the same one on all three runs. The scores are lower than those counts suggest because a classification earns credit only above what guessing would give, so one wrong code costs more than one cell.
This stage, extract the figures and the full build from the brief are in the demo, open on GitHub and scored with a token.
3. How scoring works
Five design choices carry the scoring: no rubric and no model grading a model; one key value per cell, fixed before any model runs; every decision on record with its alternative; one author for the whole build; every score re-derivable from the stored submission.
Public data on a dated snapshot. Every figure in a key traces to a published table in the data landscape, a read-only snapshot of the public sources pinned to a date. Anyone with the packet can follow the trace, and any run can be repeated to the same score. The menu lists every dataset, column and value an answer may name, which is what makes an answer checkable by machine. The model works in a Python sandbox over the landscape files, writes an answer file, and the grader scores it afterwards.
Exact match, no rubric. An answer is credited only if it matches the key within one part in ten thousand. A stage score is the share of answers credited, with no credit for what guessing would earn, so one wrong classification costs more than its one cell. A full build returns three separate numbers that are never combined: sources right, the share of the right sources and series declared; calculations hold, the share of the model's own calculations that hold together; and figures right, net, right figures minus wrong ones, floored at zero. A wrong figure is penalized, and so is a figure given where the source publishes none. A gap declared where the source publishes none counts as right. The grader is plain Python over the packet.
Every decision on record. Where the brief leaves a choice a senior analyst has to make, the key takes the position a senior analyst would take and records it in the key's register (27 entries on Environment 1, 11 on Environment 2) with the alternative and the reason. A model on the other side is scored as wrong on that cell, and the record says which decision it was and shows the alternative. That is how a decision becomes scorable: it is fixed and written down before any model runs, and the model is measured against it rather than against a grader's reading. Anyone with the sources can challenge a position.
One author for the whole build. Each environment is made end to end by senior consultants who led this work on transactions from the hundreds of millions to the billions and answered for the figures that went to the investment committee. The same people choose the sources, make every ruling, build the figure and write the key. It is not a set of tasks written separately by many contributors and assembled afterwards.
Checked before release. Every score re-derives from the stored submission and the release grader, 334 of 334 on 20 September 2026. The packet matches its checksums. Every recorded position is applied in the grader. Reproduced without the key: gpt-6-astra reached every figure on the build from the extracted figures twice on Environment 1, 25 of 25 cells within one part in ten thousand, and the figures of the builds taken apart in 4.9 to 4.11 were reproduced to six decimals from the sources and the models' own routes, except where the record says it could not be done.
4. Results, stage by stage and build by build
The brief is a client engagement: a PE client wants the consumer market for physical goods bought online in five countries, to support a growth and expansion strategy. The environment covers the first module of that work, the data module: establish what the market can be measured from in the public statistics, and assemble the inputs the market build needs, for the base year. The growth view, the segments and the strategy all use these inputs, so they have to be right.
Each of the eleven parts below covers one stage or one full build under the same five headings: the task, the score, where the misses are, an example, and a conclusion. Where a stage had no market error there is no example. A stage is one step of the work tested on its own, with the earlier steps' answers supplied; a full build is the whole module in one run, handed only its starting point, the brief, the plan and datasets, or the extracted figures.
The counts cover the three models tested, three runs each, nine runs per environment. Each run has up to 120 turns, one model action per turn. A run that ran out of turns, or got an empty reply from the model's provider partway through, is counted as not submitted, not as a wrong answer. A cell is one figure in the answer table. Every cell is scored against one key. Where the brief leaves a choice a senior analyst has to make, the key takes the position a senior analyst would take, states it in the answer record, and scores against it.
4.1 Structure the model
The task: the brief says what the client wants and what the module has to deliver, the base-year market per country. Before opening any data, the model gives twelve scored answers to four questions on the plan by picking from listed options: which purchases are in scope, what each of three ways of building the figure would need (purchases times an average purchase value, spending figures read directly, or seller-side data), how to handle the five countries, and which of the three ways is the most granular. On Environment 2 there are four questions. The skill is reading a brief for its exact scope and planning inside it.
Score: 0.89 on Environment 1, 1.00 on Environment 2; 0.92 across both, the same for all three models.
Where the misses are: there was a single error on Environment 1, and every model made it on every run. The brief asks for physical goods, but when the models described what two of the three build approaches would need, they described a build of all online purchases, goods or not. That is nine of nine runs and eighteen of the 108 answers, and it accounts for the whole gap between 0.89 and 1.00. Every other answer was right on every run, and on Environment 2 nine of nine runs were right on all four questions.
Example: gpt-6-astra on Environment 1 got the scope, the country handling and the most granular approach right on all three runs, and in all three its plan sized all online buying rather than goods.
Conclusion: everything in the plans was right except the market itself. The brief's first sentence says physical goods; all three models planned for all online buying.
4.2 Select the sources
The task: handed the brief, the plan and the files available, fifteen datasets on Environment 1 with their documentation, the model decides for each country which files the market is built from and which are not sources. That is eleven answers. A file the build only reads to cross-check is not a source, and Germany's main card table prints no national total, so Germany needs a second card dataset that the other four countries do not. On Environment 2, one set of files for the whole build out of eighteen, five answers. Naming a file that is not a source costs as much as missing one that is. The skill is judging what a dataset measures against what the build needs.
Score: 0.43 on Environment 1, 0.53 on Environment 2; across both, gpt-6-astra 0.72, claude-fable-5-1 0.29, gemini-3.8-flash 0.38.
Where the misses are: on Environment 1, every model named the cross-check dataset as a source for the four countries where the build does not use it: nine of nine runs, four wrong answers each. The consumer survey's latest wave does not cover two of the five countries, and six of nine runs dropped it for those two even though the build still needs it there; gpt-6-astra kept it. claude-fable-5-1 then named a published summary of the survey as a source in its place, on all three runs, which is why it scores below gemini-3.8-flash with the same first two errors. On Environment 2, six of nine runs named the first group of files they found and stopped, missing the survey and the file the cross-border share comes from; gpt-6-astra named all five on every run. The nearest miss was gemini-3.8-flash's third run, four of the five and one file too many.
Example: claude-fable-5-1's first run on Environment 2. It named the card file and nothing else. In the same run's full build from the brief, the other four sources are never mentioned, although their file names appeared in six of its tool results across five turns. It never looked at the other four. Its final figures on that build are the sums of one component of five.
Conclusion: the models fail here in two opposite ways. On Environment 1 every model declared a dataset the build only reads to cross-check, so their source lists were too long; six of nine runs then dropped the survey where it does not cover a country, so the lists were also too short. On Environment 2 two of the three models built from the first group of files they found and never looked at the rest. This is the lowest-scoring stage on both environments, and a source left out here cannot be recovered later in the build.
4.3 Pinpoint the exact series
The task: handed the chosen files, the model names the exact series to read in each. The payment statistics publish the same count at many combinations of dimensions (who paid, where they were, how the payment was made, the card type, the period), and the model has to fix every dimension to one value: 21 such values on Environment 1, 34 on Environment 2. On Environment 1 it also sorts the 58 merchant categories into goods, digital, fuel, services or money (fuel stations, for example), marks each of 14 survey purpose codes as physical goods or not, and decides for each of 12 payment-method codes whether it is already inside the card count (a wallet charged to a card is; a bank transfer is not): 105 answers, each scored exactly against the key. The skill is mapping a market definition onto statistics published in their own categories, code by code.
Score: 0.61 on Environment 1, 0.59 on Environment 2; across both, gpt-6-astra 0.86, claude-fable-5-1 0.93, gemini-3.8-flash 0.04 (three of three Environment 1 runs did not submit).
Where the misses are: on Environment 1, every one of the six runs that submitted put one or two of the twelve payment-method codes on the wrong side of the line between card and not card, ten of 72 code answers. claude-fable-5-1 named all 63 values right across its three runs; gpt-6-astra 59 of 63, two runs with two values each that point at a series that exists but is not the one the build reads. The merchant categories and purpose codes were right on every submitting run. gemini-3.8-flash's three runs all ran out of turns before submitting, which is its 0.04. On Environment 2, gemini-3.8-flash did not submit on two of three runs. Of the seven that submitted: the statistics count the same payments from both ends, the buyer's and the seller's, and four of seven runs took the wrong end with every other part of the series right; claude-fable-5-1 took one end on its first run and the other on its two later ones. Every submitting run hedged at least once, listing several values where one was asked, and a hedged answer counts as no answer. One pair of values was reached by no run of any model, and five runs had both right values inside a longer list and lost them by hedging.
Example: claude-fable-5-1 on Environment 2. The statistics count the same payments from both ends. On its first run it read one end, with a stated reason in its transcript; on its two later runs it read the other, with every other part of the series right each time. With the same task and the same data, the model had no fixed view of which end the market is counted from.
Conclusion: given the sources, two of the three models can name every series exactly, 63 of 63 for claude-fable-5-1, and lose points only on the classification of one or two payment codes per run; gemini-3.8-flash used all 120 turns on every attempt, where the other two finished in 9 to 28. On Environment 2 the misses were which end of the statistics to read, and hedging, several values offered where one was asked, which counts as no answer.
4.4 Extract the figures
The task: handed the exact series, the model reads the figures in them for every country at the reference year and sums them where a cell is a total over categories: 70 cells on Environment 1 (for example, card purchases per merchant category per country, summed from the source table), 15 on Environment 2. Where the source leaves a cell blank (suppressed, in the statistics' own term), the model has to decide what to do. The skill is careful reading and summing, and knowing what to do when a number is not where it should be.
Score: 0.94 on Environment 1, 0.73 on Environment 2; across both, gpt-6-astra 0.93, claude-fable-5-1 0.93, gemini-3.8-flash 0.87.
Where the misses are: on Environment 1 the misses come down to one cell. One country's base count is blank in the table the model is pointed to, and the figure is published in another file. Five of nine runs reported the blank as the answer (gemini-3.8-flash three times, gpt-6-astra once, claude-fable-5-1 once); four of nine went looking and found the figure. Every other cell was right on every run, nine of nine on 69 of 70. On Environment 2, two cells, the same two on every run of every model: the source publishes nothing in the table given, the key recovers each cell from figures the source does publish, and every run reported the gap instead.
Example: claude-fable-5-1 on Environment 1. Its first and second runs found the missing country's base in another file; its third read the main file, found the blank and stopped. The same model with the same data found the figure twice and gave up once, so the figure was within reach.
Conclusion: reading a figure from a given series is done right, 69 of 70 cells on every run. The one cell missed is blank where the model is pointed and has to be found in another file, and five of nine runs reported the blank instead of looking.
4.5 Fill the coverage gaps
The task: the card data counts card-paid purchases only, and the market is purchases by any payment method. To scale up, the build needs the share of goods purchases paid by card in each country, which comes from the consumer survey, and the survey's latest wave does not cover two of the five countries. Handed the extracted figures and told this, the model produces the missing shares: on Environment 1 by the method stated on the task, eleven answers; on Environment 2 by first choosing from listed options how the two unmeasured components are scaled, then computing ten survey shares and the two components per country, 21 answers. The skill is estimating by a stated method; on Environment 2, choosing the method too.
Score: 1.00 on both environments, every model.
Where the misses are: there were none. Nine of nine runs scored 1.00 on Environment 1, 99 of 99 answers, and nine of nine on Environment 2, 189 of 189.
Example: the case for this stage is on the full build from the brief on Environment 1, where nobody tells the model that the survey has a gap. There, seven of seven submitting runs reported the two uncovered countries as unavailable instead of filling the gap the way every one of them filled it here. Ten of the 25 cells on the build from the plan go the same way on six of eight runs.
Conclusion: told the survey does not cover two countries and asked to fill the gap, every run filled it the key's way. Asked for the whole build, none of them noticed the same gap, and seven of seven reported those countries as unavailable. The models can fill the gap when told it is there. None of them noticed it on their own.
4.6 Set the market perimeter
The task: a purchase counted in one country's data may have gone to a seller abroad, and the client wants each country's market split between sellers at home and sellers abroad. Handed the figures so far, the model establishes for each country the share of its online buying or selling that crosses the border, on the basis the task names: five answers on Environment 1, ten on Environment 2 including how each share was obtained. The skill is measuring a cross-border share on the right basis.
Score: 0.89 on Environment 1, 1.00 on Environment 2; across both, gpt-6-astra 0.89, claude-fable-5-1 1.00, gemini-3.8-flash 1.00.
Where the misses are: there were none on the answers. On Environment 1, eight of nine runs scored 1.00; the ninth, a gpt-6-astra run, made no submission because the model's provider returned an empty reply, no text and no action, on the turn after the model's own tool output had printed all five shares right to six decimals. On Environment 2, nine of nine runs scored 1.00.
Conclusion: all three models have this stage solved, 17 of 18 runs at 1.00. The one zero is a provider returning nothing after the model had already printed the right answer in its own tool output.
4.7 Finalize the inputs
The task: the card purchase counts are not the market yet, and this stage turns them into the build's inputs. On Environment 1, 85 cells: drop the merchant categories classified as not goods, split the goods count by product segment, take the card share per segment from two survey waves, scale card-paid purchases up to all payment methods, and set the average ticket. On Environment 2, 30 answers: for each component, whether and where the source publishes the amounts that have to come out, and the 25 exclusion figures themselves. The skill is carrying the definition through to the numbers: what is scaled up, what is taken out.
Score: 0.98 on Environment 1, 0.64 on Environment 2; across both, gpt-6-astra 0.86, claude-fable-5-1 0.87, gemini-3.8-flash 0.94.
Where the misses are: on Environment 1 there was one error. The older survey wave has a payment category the newer one dropped, and four of nine runs computed shares from the older wave without handling that category (gpt-6-astra all three, claude-fable-5-1 once), the same three wrong figures each time; twelve of 765 cells. On Environment 2 there were three. The source publishes three categories of amounts to take out; one of them was reported by no run of any model, eight of eight runs that submitted and forty of forty cells, while the two beside it in the source's own list were answered. A second category, ten cells, was left blank by five of eight runs. claude-fable-5-1 wrote "not published" on every run for a component whose amounts the source does publish, at a broader level. gpt-6-astra's third run made no submission, an empty reply on its twelfth turn.
Example: claude-fable-5-1 on Environment 2. At this stage each of its three runs reported one exclusion category, left the second and third blank entirely, and scored 0.55. In the same runs' full build from the plan it built the two measured components from the gross published figures and subtracted nothing, not even the exclusions it had itself just reported at this stage; seven of its fourteen wrong cells there are, to the sixth decimal, the published figure plus the exclusions it had reported.
Conclusion: the models remove the exclusions they are told about. One category sits in the source's own list and none of the eight runs found it. Reporting an exclusion at this stage did not mean applying it later: claude-fable-5-1 reported exclusions here and then built the same components in its full build with nothing removed.
4.8 Compute the result
The task: handed the finished inputs, the model produces the market per country, purchases times average purchase value, 25 cells on Environment 1; on Environment 2, each component net of its exclusions, the subtotal and the total per country, 35 cells, with a gap declared wherever an input does not exist. The skill is arithmetic.
Score: 0.89 on Environment 1, 1.00 on Environment 2; across both, gpt-6-astra 0.86, claude-fable-5-1 1.00, gemini-3.8-flash 1.00.
Where the misses are: there were none on the arithmetic. On Environment 1, eight of nine runs scored 1.00; gpt-6-astra's second run made no submission, the same empty reply from the provider as at stage 6, on a different run, two turns in. On Environment 2, nine of nine runs scored 1.00, 315 of 315 cells.
Conclusion: given the right inputs, no model got the arithmetic wrong on either environment, 315 of 315 cells on Environment 2. Every wrong final figure across all runs was made before this stage.
4.9 The full build from the brief
The task: handed the brief and the files available and nothing else, the model delivers the final market figure per country and says which datasets and series it used: on Environment 1 the five market values plus the fourteen datasets it accepted or rejected, with the series it names scored against the 21 values in the key; on Environment 2 the three published final figures and two gaps, against 38. There is no calculation check, because only the final figure is submitted. It is every stage above in one run, with no stage answers supplied and no prompt to stop and decide.
Score, both environments together: sources right 56%, figures right 0 of 44, no run with every figure right; gpt-6-astra 92% on sources, claude-fable-5-1 51%, gemini-3.8-flash 25% (three of six runs did not submit).
Where the misses are: fifteen of eighteen runs submitted; gemini-3.8-flash did not on two Environment 1 runs and one Environment 2 run. On Environment 1, two things account for every lost cell. The two countries the survey does not cover were reported as unavailable rather than filled, by seven of seven submitting runs, fourteen of 35 final cells. The three covered countries were wrong on every run, 21 of 21 cells: we recomputed one run (claude-fable-5-1, first run) and found five differences from the key, each alone enough to lose the cell: which payment codes count as card, which average ticket to use, the base count read from a series that exists but is not the one the build reads, one merchant category counted inside a broader one it already belongs to, and one divider taken at the wrong level. The first two moved that run's figure for one country by 33% to 59%; the base count and the double count moved it by under 1%. Six of seven runs read the base from that same wrong series; five of those six had named the right one at stage 3 in the same run. On Environment 2 the error is scope: five of eight submitting runs built the market from one component, or one plus a single survey series (claude-fable-5-1 three times, gemini-3.8-flash twice). gpt-6-astra declared all five sources on all three runs, 35 to 37 of 38 values, and still wrote no right figure: one run in raw units, a million times off, two in the right magnitude and 12% to 52% short.
Example: gpt-6-astra's first run on Environment 2 had 37 of 38 values right and all five sources declared, and produced three wrong final figures. It found the components, the survey shares and the cross-border source, and still got all three figures wrong.
Conclusion: from the brief alone, no run produced a right figure on either environment, and the recomputation shows why: on the runs we reconstructed, between two and five separate decisions, each one enough on its own to lose the cell, and the model flagged none of them. gpt-6-astra found every source on Environment 2, 37 of 38 values, and was still a million times off on one run. The sources were found. The figures were lost on the decisions made between the sources and the figure, and no model flagged that it was making one.
4.10 The full build from the plan and datasets
The task: handed the brief, the plan and the chosen files with their roles (and, on Environment 1, the six product-segment definitions), the model delivers the final figures and the intermediate cells: 25 cells, 21 values and ten calculation checks on Environment 1; 35 cells, 38 values and seven checks on Environment 2. It is stages 3 to 8 in one run.
Score, both environments together: sources right 71%, calculations hold 60%, figures right 1 of 42, no run with every figure right; gpt-6-astra 98% and 73%, claude-fable-5-1 93% and 69%, gemini-3.8-flash 21% (three of six runs did not submit) and 37%.
Where the misses are: fifteen of eighteen runs submitted. On Environment 1, the two uncovered countries were reported as unavailable on all ten of their cells by six of eight runs and on eight of ten by a seventh; in claude-fable-5-1's transcripts the idea of filling the gap never comes up. The three covered countries were wrong on eight of eight runs: which payment codes count as card and which average ticket to use, on gpt-6-astra and claude-fable-5-1; wrong series on gemini-3.8-flash. gpt-6-astra's three runs score 21 of 21 on sources, six of ten on calculation checks and 0 on the figures: the whole loss is those two choices. claude-fable-5-1's third run named 21 of 21 values right, its first run with every value right, and still scored 0. On Environment 2 the error is which base the unmeasured components are scaled from. Two of gpt-6-astra's three runs took the exclusions out of the measured components first and then scaled the unmeasured ones from the netted figures; the key scales them from the other base. That one choice cost 17 of 35 cells, with 38 of 38 values and seven of seven calculation checks right. claude-fable-5-1's first run did the opposite, built the measured components gross with nothing removed, fourteen wrong cells every one equal to a published figure or a sum of them, and still holds the best figures score on this build of any run, five of 35; its two later runs started removing exclusions and broke the cells the first had right. gemini-3.8-flash's one submitting run pulled the wrong series for two components, one of them 667 times the key.
Example: gpt-6-astra's first run on Environment 2 had 38 of 38 values and seven of seven calculation checks right and lost 17 of 35 cells; the difference is one decision: which base the unmeasured components scale from.
Conclusion: handing over the plan and the datasets lifts sources right to 71% and calculations hold to 60%, and does not lift the figure at all, one right figure in 42. gpt-6-astra scored 38 of 38 on sources and 7 of 7 on calculation checks on one run and lost 17 of 35 cells to one decision, which base to scale from. The plan fixed the sources and most of the arithmetic. The decisions inside the build were still the model's, and they were still wrong.
4.11 The full build from the extracted figures
The task: handed the extracted figures, the model delivers the final figures and the intermediate cells. On Environment 1 the coverage shares are handed too, so stages 6 to 8 remain, perimeter, finalizing and arithmetic: 25 cells and ten calculation checks. On Environment 2 only the three measured components are handed, so stages 5 to 8 remain, scaling the unmeasured components, perimeter, exclusions and arithmetic: 35 cells and seven checks. Sources are not scored, because they were handed over.
Score, both environments together: calculations hold 72%, figures right 10 of 71, two of eighteen runs with every figure right; gpt-6-astra 86%, claude-fable-5-1 76%, gemini-3.8-flash 53%.
Where the misses are: eighteen of eighteen runs submitted. All ten right figures are gpt-6-astra's two exact runs on Environment 1. Otherwise on Environment 1: claude-fable-5-1, three runs, 20 of 25 cells wrong each time, on two errors we reproduced to the sixth decimal: it never scaled the card-paid count up to purchases by all payment methods, although the all-methods figure was given for every country (it copied it into its first draft and dropped it in the rewrite), and it applied one set of shares to all five countries, past a scope note printed six times beside those numbers saying which countries they are for; the third run fixed the second error and kept the first. gemini-3.8-flash, three runs, 20 of 25 wrong, one calculated cell reported as not published, and its own arithmetic inconsistent by 24% to 50% across four countries. One gpt-6-astra run reported the ten cells that split each market between sellers at home and abroad as unavailable. On Environment 2, 0 of 26 figures right. The measured components were carried through gross, with no exclusions removed, in eight of nine runs; gpt-6-astra's one run that did remove them then scaled the unmeasured components from the netted base, the same choice as in 4.10, and lost them that way. The two unmeasured components were reported as a gap in four countries by six of nine runs (claude-fable-5-1 and gemini-3.8-flash on every run), which also fails four of the seven calculation checks; gpt-6-astra's other two runs computed them by a method the key does not use.
Example: on claude-fable-5-1's first run on Environment 1, every figure it read off the page is right, five of 25, and every figure it derived is wrong, 20 of 25, because of one scale-up it left out although the figure it needed was on the page, and one scope note it read past.
Conclusion: even with every extracted figure handed over, 16 of 18 runs missed the final figure, and the two that got it were both gpt-6-astra on Environment 1. The misses are the two decisions of stage 7: what to take out, and which base to scale up from. With nothing left to find, those two decisions alone cost 16 of 18 runs the figure.
4.12 Full builds
Environment 1
| The model is given | Sources right | Calculations hold | Figures right, net | Runs with every figure exact | Distance from the right figure | Within 25% | More than double |
|---|---|---|---|---|---|---|---|
| The brief only | Sources right0.69 | Calculations hold– | Figures right, net0.37 | Runs with every figure exactnone | Distance from the right figure33% | Within 25%8 of 21 | More than double1 of 21 |
| The build plan and datasets | Sources right0.70 | Calculations hold0.58 | Figures right, net0.00 | Runs with every figure exactnone | Distance from the right figure39% | Within 25%3 of 26 | More than double8 of 26 |
| The extracted figures | Sources right– | Calculations hold0.81 | Figures right, net0.22 | Runs with every figure exact2 of 9 (gpt-6-astra) | Distance from the right figurethe rest 24% | Within 25%31 of 45 | More than doublenone |
Sources right: the share of the series the build needs that the model named correctly; not scored where the sources were handed over with the extracted figures. Calculations hold: the share of the model's own calculations that hold across the cells it submitted; not scored from the brief, where only the final figure is submitted. Figures right, net: right final cells minus wrong ones, floored at zero, as a share of the cells asked. On Environment 1, figures right from the brief alone is the mean of two parts: the final figures, which were 0 on every run, and the datasets the build refuses. The runs that submitted identified eleven to fourteen of the fourteen. Runs with every figure exact: runs where every final figure matched the key. Distance from the right figure is not a grader score: it is measured afterwards, per final figure, against the key, and the median is shown; 0% is exact.
Environment 2
| The model is given | Sources right | Calculations hold | Figures right, net | Runs with every figure exact | Distance from the right figure | Within 25% | More than double |
|---|---|---|---|---|---|---|---|
| The brief only | Sources right0.49 | Calculations hold– | Figures right, net0.00 | Runs with every figure exactnone | Distance from the right figure45% | Within 25%6 of 23 | More than double4 of 23 |
| The build plan and datasets | Sources right0.71 | Calculations hold0.62 | Figures right, net0.02 | Runs with every figure exactnone | Distance from the right figure9% | Within 25%12 of 16 | More than double2 of 16 |
| The extracted figures | Sources right– | Calculations hold0.59 | Figures right, net0.00 | Runs with every figure exactnone | Distance from the right figure5% | Within 25%24 of 26 (14 of 26 within 5%) | More than doublenone |
The counts are of final figures submitted.
The whole build, per model
Each model asked for the complete set of final figures, from three starting points, three runs each. Cells: figures right (net) / runs with every figure exact / distance from the right figure (figures counted).
| The model is given | gpt-6-astra | claude-fable-5-1 | gemini-3.8-flash |
|---|---|---|---|
| Environment 1, the brief only | gpt-6-astra0.50 / 0 of 3 / 17% (9) | claude-fable-5-10.43 / 0 of 3 / 43% (9) | gemini-3.8-flash0.17 / 0 of 3 / 35% (3) |
| Environment 1, the build plan and datasets | gpt-6-astra0.00 / 0 of 3 / 31% (9) | claude-fable-5-10.00 / 0 of 3 / 80% (11) | gemini-3.8-flash0.00 / 0 of 3 / 87% (6) |
| Environment 1, the extracted figures | gpt-6-astra0.67 / 2 of 3 / 0% (15) | claude-fable-5-10.00 / 0 of 3 / 24% (15) | gemini-3.8-flash0.00 / 0 of 3 / 24% (15) |
| Environment 2, the brief only | gpt-6-astra0.00 / 0 of 3 / 48% (8) | claude-fable-5-10.00 / 0 of 3 / 48% (9) | gemini-3.8-flash0.00 / 0 of 3 / 40% (6) |
| Environment 2, the build plan and datasets | gpt-6-astra0.02 / 0 of 3 / 3% (8) | claude-fable-5-10.05 / 0 of 3 / 16% (5) | gemini-3.8-flash0.00 / 0 of 3 / 149% (3) |
| Environment 2, the extracted figures | gpt-6-astra0.01 / 0 of 3 / 24% (8) | claude-fable-5-10.00 / 0 of 3 / 5% (9) | gemini-3.8-flash0.00 / 0 of 3 / 5% (9) |
The distance is the median over the figures in brackets; with three to fifteen figures it swings, so read it with the count. gemini-3.8-flash made no submission on two of its three Environment 1 runs from the brief and one with the plan, and on one Environment 2 run from the brief and two with the plan. gpt-6-astra got every figure right on two of its three runs with the extracted figures on Environment 1, ten of fifteen figures exact; no other run on either environment did. Its 3% on Environment 2 with the plan is the run in 4.10: near on every cell, right on none.
4.13 Per model and cost
Cost is from exact token counts at list prices, range over three runs. The score is the unweighted mean of the eight stage scores, so a four-answer stage counts as much as a 105-answer one. gemini-3.8-flash is at a promotional rate that ends on 31 December 2026, after which its cost doubles; its 0.78 on Environment 1 carries a 0.00 on pinpoint the exact series, three runs that ran out of turns, worth 0.12 of its mean.
Stages, mean of three runs, Environment 1 / Environment 2
| Stage | gpt-6-astra | claude-fable-5-1 | gemini-3.8-flash |
|---|---|---|---|
| Structure the model | gpt-6-astra0.89 / 1.00 | claude-fable-5-10.89 / 1.00 | gemini-3.8-flash0.89 / 1.00 |
| Select the sources | gpt-6-astra0.60 / 1.00 | claude-fable-5-10.27 / 0.33 | gemini-3.8-flash0.43 / 0.27 |
| Pinpoint the exact series | gpt-6-astra0.88 / 0.77 | claude-fable-5-10.96 / 0.85 | gemini-3.8-flash0.00 / 0.16 |
| Extract the figures | gpt-6-astra0.97 / 0.73 | claude-fable-5-10.97 / 0.73 | gemini-3.8-flash0.90 / 0.73 |
| Fill the coverage gaps | gpt-6-astra1.00 / 1.00 | claude-fable-5-11.00 / 1.00 | gemini-3.8-flash1.00 / 1.00 |
| Set the market perimeter | gpt-6-astra0.67 / 1.00 | claude-fable-5-11.00 / 1.00 | gemini-3.8-flash1.00 / 1.00 |
| Finalize the inputs | gpt-6-astra0.95 / 0.60 | claude-fable-5-10.98 / 0.55 | gemini-3.8-flash1.00 / 0.77 |
| Compute the result | gpt-6-astra0.67 / 1.00 | claude-fable-5-11.00 / 1.00 | gemini-3.8-flash1.00 / 1.00 |
gemini-3.8-flash: 0.00 on Environment 1's stage 3, pinpoint the exact series, is three runs that ran out of turns without submitting. gpt-6-astra: the two 0.67 on Environment 1 are one run each that got an empty reply from the provider after its own tool output had printed the answer; not retried.
A full run of all eleven costs $4.8 to $6.4 with gpt-6-astra on Environment 1 and $6.2 to $7.7 on Environment 2, $11.4 to $12.2 and $9.8 to $10.7 with gemini-3.8-flash, and $19.0 to $21.0 and $19.9 to $21.7 with claude-fable-5-1. claude-fable-5-1 returns an identical score on six of the eight stages across all three runs on both environments. gpt-6-astra has the best single run on both environments and the widest spread between runs. gemini-3.8-flash ran out of turns eleven times across the two environments.
5. Takeaways
The stages hold and the whole does not. Six of eight stages score 0.89 or better and no full build from the brief reaches a right figure. What the models lose the figure on is a few decisions that carry through the build, each worth 5% to 59% of the figure, not many small slips. The two stages where the models are furthest apart, and furthest from the key, are select the sources and pinpoint the exact series. Cost per run ranged from about $5 to about $21 across the three models, and none reached a right figure from the brief.
6. What is next
New environments are added continuously, each one a new market question built to the same depth, and each adds to the findings on this page. Every new frontier model release gets three runs on every environment. The score is exact and reproduces from the stored submission, so it works as a reward signal today. As the set grows we train on some environments and hold out the rest, and publish how much of the improvement holds on markets the model was not trained on.