Scorekey

Scorekey Market Sizing Environments

Evaluating models on complete market sizing builds, every answer scored against a key

The two environments in this study are complete market sizing builds from public data, each with a key that holds the right answer for every cell. The build is tested in eleven pieces: the eight stages of the work, each asked on its own with the earlier stages' answers supplied, and three full builds from different starting points. An answer is credited only if it matches the key within one part in ten thousand. Nothing is scored by a rubric or by another model. Environment 1 is the online goods market in the public demo; Environment 2 is a second market question built the same way.

We ran gpt-6-astra, claude-fable-5-1, gemini-3.8-flash, Mistral Medium 3.5 and grok-4.7 on both environments, three runs of each of the eleven pieces, fifteen runs per environment. On the mechanical stages, where the inputs are given and the work is to read, fill and compute, extracting the figures, filling the coverage gaps and computing the result, the models score 0.73 to 0.95. On the two stages that require choices about the data, which sources to build from and which exact series to read in them, they score 0.40 to 0.69. Asked for the whole build from the brief, no run of any model on either environment produced a right final figure.

Below: what the environments cover, one stage walked through, how scoring works, the results, and what comes next.

Each run is one whole build with three checks; three runs per model.

Where models go wrong

  1. Declares a recoverable figure unavailable: Where the source prints no figure in the table given, the key records the route that recovers it from published totals, and the reason. Every model on every run declared the figure unavailable instead. That one decision, taken the other way by every run, caps extract the figures at 0.73 on Environment 2 for every model.
  2. Builds the market from one table: It sizes the part of the market that sits in the largest table and leaves out the parts that sit in other files. Nine of the fourteen runs that submitted built Environment 2's market from the brief from a single source, or one source plus a single survey series; gpt-6-astra named all five sources on every run and still produced no right figure.
  3. Reads the statistics from the other side: Statistical sources publish the same payments counted from both ends, the payer's and the payee's, and the key records which end the market is counted from, and why. On Environment 2 at stage 3, four of the thirteen runs that submitted read the other end with every other part of the series right, gpt-6-astra on all three of its runs and claude-fable-5-1 on one; nine of thirteen used it somewhere.
  4. Names a source it does not need and keeps what it should take out: On Environment 1, ten of fifteen runs declared as a source a dataset the build only reads to cross-check. On Environment 2, one category of amounts the source itself lists for removal was reported by none of the fourteen runs that submitted, and in the full build from the extracted figures fourteen of fifteen runs carried the measured components through with nothing removed.
  5. Writes a figure where the total cannot be finished: Where one input needed to complete a country's figure is not published, the right answer is to declare the gap. On Environment 2 from the brief, eleven of the fourteen runs that submitted wrote a figure for at least one of the two such countries, seven for both; gpt-6-astra declared the gap for both on all three of its runs.

1. Coverage

The eight stages are the stages of sizing a market in a commercial due diligence, in the order the work is done, from the brief to the figure that goes to the investment committee. Each environment takes one market question through all eight, about 400 decisions on Environment 1 and about 250 on Environment 2.

Get the data

  1. Structure the model. Fix what is being sized (product, buyer, channel, geography, period) and the formula that produces it: volume times value, spend read directly, or supply side. Every later stage supplies an input to this formula.
  2. Select the sources. Decide which of the candidate datasets the build stands on. Every model has more candidate sources than it uses, and the wrong choice is the most expensive error in the build.
  3. Pinpoint the exact series. Statistics are published in their own categories, not the market's. Pin every series to read, dimension by dimension, and where the sources publish codes, decide for each whether it is inside the definition, outside it, or already counted elsewhere.
  4. Extract the figures. Pull the raw numbers at the pinned addresses and deal with suppressed or missing cells.

Refine it to the market

  1. Fill the coverage gaps. Every dataset covers the definition only partly: one payment method, some countries, a sample, a year. Derive the factors that cover the rest, from related sources, by a stated method.
  2. Set the market perimeter. Every source measures a population, and the market's perimeter rarely matches it exactly. Decide how the perimeter maps onto the measured population, and the share of the measured activity that lies inside it.
  3. Finalize the inputs. Turn the extracted figures into the inputs the formula needs: establish what falls outside the definition and take it out, and bring what is only partly covered up to the whole, with factors from stage 5 or established here.

Result

  1. Compute the result. Bring the inputs together, apply the exclusions and the stage 6 share, run the stage 1 formula, and declare where an input does not exist.

How much each stage weighs depends on the market; none can be skipped without saying why.

How it is tested. The build is tested stage by stage and as a whole. Stage by stage, each of the eight is asked on its own, with the earlier stages' verified answers supplied, and every answer inside it is scored: 324 assessed answers across the eight on Environment 1, from 5 on the market perimeter to 105 on the exact series. As a whole, the model is asked for the complete build three times, from the brief alone, from the brief with the build plan and the datasets, and from all of that with the extracted figures; each is scored on its sources, its calculations and its final figures, 40, 56 and 35 assessed answers. One run of all eleven is about 1,240 decisions and 455 assessed answers on Environment 1, about 760 and 319 on Environment 2. Decision counts are our count, rounded; assessed answers are the grader's exact count.

2. Sample stage: pinpoint the exact series

This is stage 3 of 8 in Environment 1, the online goods market.

What the model is given. The brief: a client wants the market for physical goods bought online in five European countries, one figure per country, reference year 2024. The data landscape, the files the model may read: thirteen extracts of the ECB's payment statistics, three waves of a consumer payment survey with their codebooks, and the codelists, about 120 MB, read-only in a Python sandbox with no web access and at most 120 turns. The ECB's payment statistics count every card payment, transfer and other payment made in the euro area, and publish the counts broken down by many dimensions at once: who paid and who received, where each of them was, how the payment was started, the payment scheme, the type of card, and the reporting period. A single figure the build needs is the count at one combination of those dimensions, and the whole build's precision comes from choosing the right combination for every series it reads. The menu of every dataset, column and value an answer may name. And the verified answers of the two earlier stages, the build's structure and which datasets it uses per country, so an error in an earlier stage does not count against this one.

What it has to decide and submit. Two things, 105 answers in all. First, for each of the five datasets the build uses, the exact series to read: every dimension the file carries fixed to one value (frequency, counterparty area, terminal location, transaction type, channel, remote or not, scheme, card function, unit), 21 values. Second, what counts as the market, code by code: each of the main source's 58 merchant categories classed as goods, digital, fuel, services or money; 14 of the survey's 15 purpose codes as physical goods or not (one is not scored); each of its 12 payment-method codes as already inside the base count or not. One JSON object, one answer per question. Two answers to one question count as a format error.

What the grader checks. Each of the 105 answers against the key, exact match. The score is the share right, with no credit for what guessing would earn, so one wrong classification costs more than its one cell. There is no partial credit on a series: one dimension wrong is a wrong series, and naming a column the dataset does not carry counts as wrong.

What a wrong answer looks like. A series one dimension off, usually the frequency or the counterparty area (where the other side of the payment is located: same country, or abroad), which selects a real series that is not the one the build needs. A merchant category put on the wrong side of the goods line, or a survey purpose code counted as goods when it is not. A payment-method code classified as outside the base when it is already inside it, so the purchases it covers are added a second time.

What the five models scored. On this stage, gpt-6-astra scored 0.88, claude-fable-5-1 0.96, gemini-3.8-flash 0.00, Mistral Medium 3.5 0.70 and grok-4.7 0.90, out of 1.00. The zero is three runs that ran out of turns without submitting. gpt-6-astra's three runs got two, four and five of the 105 answers wrong; claude-fable-5-1's got one, the same one on all three runs; grok-4.7 pinned every series on every run and lost its points on the payment codes and on one merchant category in two runs; Mistral Medium 3.5 lost points on the series, the payment codes, the merchant categories and the purpose codes. The scores are lower than those counts suggest because a classification earns credit only above what guessing would give, so one wrong code costs more than one cell.

Extract the figures and the full build, from the brief and from the plan, are in the demo, open on GitHub and scored with a token.

3. How scoring works

Five design choices define the scoring: no rubric and no model grading a model; one key value per cell, fixed before any model runs; every decision on record with its alternative; one author for the whole build; every score re-derivable from the stored submission. The general method, for any assignment type, is on the Methods page.

Public data on a dated snapshot. Every figure in a key traces to a published table in the data landscape, a read-only snapshot of the public sources fixed on a date. Anyone with the packet can follow the trace, and any submission can be scored again to the same result. The menu lists every dataset, column and value an answer may name, which is what makes an answer checkable by machine. The model works in a Python sandbox over the landscape files, writes an answer file, and the grader scores it afterwards.

Exact match, no rubric. An answer is credited only if it matches the key within one part in ten thousand. A stage score is the share of answers credited, with no credit for what guessing would earn, so one wrong classification costs more than its one cell. A full build returns three separate numbers that are never combined: sources right, the share of the right sources and series declared; calculations hold, the share of the model's own calculations that hold together; and figures right, net, right figures minus wrong ones, floored at zero. A wrong figure is penalized, and so is a figure given where the source publishes none. A gap declared where the source publishes none counts as right. The grader is plain Python over the packet.

Every decision on record. Where the brief leaves a choice a senior analyst has to make, the key takes the position a senior analyst would take and records it in the decision record (27 entries on Environment 1, 11 on Environment 2) with the alternative and the reason. A model on the other side is scored as wrong on that cell, and the record says which decision it was and shows the alternative. That is how a decision becomes scorable: it is fixed and written down before any model runs, and the model is measured against it rather than against a grader's reading. Anyone with the sources can challenge a position.

One author for the whole build. Each environment is made end to end by senior consultants who led this work on transactions from the hundreds of millions to the billions and answered for the figures that went to the investment committee. The same analyst chooses the sources, makes every decision and builds the figure; the key files, the grader and the runner are produced from those positions by AI agents in separate roles and checked against the sources by the release gate. It is not a set of tasks written separately by many contributors and assembled afterwards.

Checked before release. Every score re-derives from the stored submission and the release grader, 334 of 334 on 20 September 2026. The packet matches its checksums. Every recorded position is applied in the grader. Reproduced without the key: gpt-6-astra reached every figure on the build from the extracted figures twice on Environment 1, 25 of 25 cells within one part in ten thousand, and the figures of the builds taken apart in 4.9 to 4.11 were reproduced to six decimals from the sources and the models' own routes, except where the record says it could not be done.

4. Results, stage by stage and build by build

The brief is a client engagement: a PE client wants the consumer market for physical goods bought online in five countries, to support a growth and expansion strategy. The environment covers the first module of that work, the data module: establish what the market can be measured from in the public statistics, and assemble the inputs the market build needs, for the base year. The growth view, the segments and the strategy all use these inputs, so they have to be right.

Each of the eleven parts below covers one stage or one full build under the same five headings: the task, the score, where the misses are, an example, and a conclusion. Where a stage had no market error there is no example. A stage is one step of the work tested on its own, with the earlier steps' answers supplied; a full build is the whole module in one run, handed only its starting point, the brief, the plan and datasets, or the extracted figures.

The counts cover the five models tested, three runs each, fifteen runs per environment. Each run has up to 120 turns, one model action per turn. A run that ran out of turns, or got an empty reply from the model's provider partway through, is counted as not submitted, not as a wrong answer. A cell is one figure in the answer table. Every cell is scored against one key. Where the brief leaves a choice a senior analyst has to make, the key takes the position a senior analyst would take, states it in the decision record, and scores against it.

4.1 Structure the model

The task: the brief says what the client wants and what the module has to deliver, the base-year market per country. Before opening any data, the model gives twelve scored answers to four questions on the plan by picking from listed options: which purchases are in scope, what each of three ways of building the figure would need (purchases times an average purchase value, spending figures read directly, or seller-side data), how to handle the five countries, and which of the three ways is the most granular. On Environment 2 there are four questions. The skill is reading a brief for its exact scope and planning inside it.

Score: 0.90 on Environment 1, 0.98 on Environment 2; 0.92 across both. By model: gpt-6-astra 0.89 / 1.00, claude-fable-5-1 0.89 / 1.00, gemini-3.8-flash 0.89 / 1.00, Mistral Medium 3.5 0.96 / 0.88, grok-4.7 0.89 / 1.00.

Where the misses are: on Environment 1 one error accounts for nearly all of it. The brief asks for physical goods, but when the models described what two of the three build approaches would need, they described a build of all online purchases, goods or not. Twelve of fifteen runs did so for both approaches, gpt-6-astra, claude-fable-5-1, gemini-3.8-flash and grok-4.7 on all three runs each, every one at 0.89; Mistral Medium 3.5 left the term out of one approach on two runs and kept it on both in its third. Answers lost to the scope term: 26 of 180. gemini-3.8-flash's three submissions were identical byte for byte, and grok-4.7 filed the same six answers on all three runs. On Environment 2, fourteen of fifteen runs were right on all four answers, 58 of 60; the one miss is Mistral Medium 3.5's first run, which pooled the five countries where the key builds each on its own and sums them.

Example: gpt-6-astra on Environment 1 got the scope, the country handling and the most granular approach right on all three runs, and in all three its plan sized all online buying rather than goods.

Conclusion: everything in the plans was right except the market itself. The brief's first sentence says physical goods; four of the five models planned for all online buying on every run, and the fifth on two of three.

4.2 Select the sources

The task: handed the brief, the plan and the files available, fifteen datasets on Environment 1 with their documentation, the model decides for each country which files the market is built from and which are not sources. That is eleven answers. A file the build only reads to cross-check is not a source, and Germany's main card table prints no national total, so Germany needs a second card dataset that the other four countries do not. On Environment 2, one set of files for the whole build out of eighteen, five answers. Naming a file that is not a source costs as much as missing one that is. The skill is judging what a dataset measures against what the build needs.

Score: 0.46 on Environment 1, 0.40 on Environment 2; 0.44 across both. By model: gpt-6-astra 0.60 / 1.00, claude-fable-5-1 0.27 / 0.33, gemini-3.8-flash 0.43 / 0.27, Mistral Medium 3.5 0.47 / 0.20, grok-4.7 0.56 / 0.20.

Where the misses are: on Environment 1, ten of fifteen runs named the cross-check dataset as a source for all four countries where the build does not use it, gpt-6-astra, claude-fable-5-1 and gemini-3.8-flash on every run and grok-4.7 once: 40 wrong answers of 60 possible. The consumer survey's latest wave does not cover two of the five countries, and nine of fifteen runs dropped it for those two even though the build still needs it there, claude-fable-5-1, gemini-3.8-flash and grok-4.7 on every run; claude-fable-5-1 named a published summary of the survey in its place on all three runs, and Mistral Medium 3.5 named the summary beside the survey once. Three runs dropped the base dataset itself: grok-4.7's second run for the three covered countries, a choice it states in its transcript, and Mistral Medium 3.5's first and third runs for all five. On Environment 2, ten of fifteen runs named exactly one file, gemini-3.8-flash twice, claude-fable-5-1 twice, Mistral Medium 3.5 and grok-4.7 on every run, grok-4.7's three identical; twelve of fifteen fell short of the five the key holds; gpt-6-astra alone named the exact five, on all three runs. The nearest miss was claude-fable-5-1's second run, three of the five and nothing extra; gemini-3.8-flash's third run named three and added one file that is not a source.

Example: claude-fable-5-1's first run on Environment 2. It named the card file and nothing else. In the same run's full build from the brief, the other four sources are never mentioned, although their file names appeared in six of its tool results across five turns. It never looked at the other four. Its final figures on that build are the sums of one component of five.

Conclusion: the models fail here in two opposite ways. On Environment 1 ten of fifteen runs declared a dataset the build only reads to cross-check, so their source lists were too long; nine of fifteen dropped the survey where it does not cover a country, so the lists were also too short. On Environment 2 twelve of fifteen runs built from one file or one family of files and never looked at the rest. This is the lowest-scoring stage on both environments, and a source left out here cannot be recovered later in the build.

4.3 Pinpoint the exact series

The task: handed the chosen files, the model names the exact series to read in each. The payment statistics publish the same count at many combinations of dimensions (who paid, where they were, how the payment was made, the card type, the period), and the model has to fix every dimension to one value: 21 such values on Environment 1, 34 on Environment 2. On Environment 1 it also sorts the 58 merchant categories into goods, digital, fuel, services or money (fuel stations, for example), marks each of 14 survey purpose codes as physical goods or not, and decides for each of 12 payment-method codes whether it is already inside the card count (a wallet charged to a card is; a bank transfer is not): 105 answers, each scored exactly against the key. The skill is mapping a market definition onto statistics published in their own categories, code by code.

Score: 0.69 on Environment 1, 0.60 on Environment 2; 0.67 across both. By model: gpt-6-astra 0.88 / 0.77, claude-fable-5-1 0.96 / 0.85, gemini-3.8-flash 0.00 / 0.16 (its three Environment 1 runs and two Environment 2 runs did not submit), Mistral Medium 3.5 0.70 / 0.44, grok-4.7 0.90 / 0.78.

Where the misses are: on Environment 1, three of fifteen runs did not submit, all gemini-3.8-flash, out of turns. Every one of the twelve that submitted put at least one of the twelve payment-method codes on the other side of the card line from the key, 28 of 144 code answers (gpt-6-astra 7, claude-fable-5-1 3, grok-4.7 7, Mistral Medium 3.5 11); which side a wallet or app payment sits on is a decision the key records, with the reason, and these runs decided otherwise. The series were pinned right on 238 of 252 values across the twelve: grok-4.7 and claude-fable-5-1 63 of 63 each, gpt-6-astra 59, Mistral Medium 3.5 53. The 58 merchant categories were right on eight of the twelve (grok-4.7 one category wrong on two runs; Mistral Medium 3.5 fourteen wrong on one run and one on another), the purpose codes on nine of twelve (Mistral Medium 3.5 wrong on all three). On Environment 2, two runs did not submit, both gemini-3.8-flash. Of the thirteen that did: the statistics count the same payments from both ends, the payer's and the payee's, and the key records which end the market is counted from, and why; four of thirteen read the other end with every other part of the series right, gpt-6-astra on all three runs and claude-fable-5-1 on one, and nine of thirteen used it somewhere. Ten of thirteen hedged at least once, listing several values where one was asked, and a hedged answer counts as no answer; eight of thirteen had both right values inside a longer list and lost them that way. One pair of survey values was reached by no run of any model. The best set on record is grok-4.7's second run, 32 of 34. Over the runs that submitted the means are 0.86 on Environment 1 and 0.69 on Environment 2; the scores above count the non-submissions as zero.

Example: claude-fable-5-1 on Environment 2. The statistics count the same payments from both ends. On its first run it read one end, with a stated reason in its transcript; on its two later runs it read the other, with every other part of the series right each time. With the same task and the same data, the model had no fixed view of which end the market is counted from; the key records one.

Conclusion: given the sources, two of the five models named every series exactly on every run, grok-4.7 and claude-fable-5-1 at 63 of 63; claude-fable-5-1 lost points only on the payment codes, a decision the key records, and grok-4.7 also put one merchant category in the wrong class on two runs. gemini-3.8-flash used all 120 turns on every Environment 1 attempt, where the others finished in 9 to 52. On Environment 2 the misses were which end of the statistics to read, a decision the key records, and hedging, several values offered where one was asked, which counts as no answer.

4.4 Extract the figures

The task: handed the exact series, the model reads the figures in them for every country at the reference year and sums them where a cell is a total over categories: 70 cells on Environment 1 (for example, card purchases per merchant category per country, summed from the source table), 15 on Environment 2. Where the source leaves a cell blank (suppressed, in the statistics' own term), the model has to decide what to do. The skill is careful reading and summing, and knowing what to do when a number is not where it should be.

Score: 0.95 on Environment 1, 0.73 on Environment 2; 0.91 across both. By model: gpt-6-astra 0.97 / 0.73, claude-fable-5-1 0.97 / 0.73, gemini-3.8-flash 0.90 / 0.73, Mistral Medium 3.5 0.92 / 0.73, grok-4.7 1.00 / 0.73.

Where the misses are: on Environment 1 the misses come down to one cell. One country's base count is blank in the table the model is pointed to, and the figure is published in another file. Seven of fifteen runs reported the blank as the answer; eight went looking and found the figure. Fourteen of fifteen runs had no miss outside that cell, eight at 1.00 and six at 0.90; the exception is Mistral Medium 3.5's first run, which also named a component the source does not publish on five rows. On Environment 2, two cells, the same two on every run of every model, fifteen of fifteen at exactly 0.73: the source publishes nothing in the table given, the key records a route to recover each cell from figures the source does publish, and the reason, and every run declared the gap instead. That one decision, taken the other way by every run, is the whole of the gap between 0.73 and 1.00.

Example: claude-fable-5-1 on Environment 1. Its first and second runs found the missing country's base in another file; its third read the main file, found the blank and stopped. The same model with the same data found the figure twice and gave up once, so the figure was within reach.

Conclusion: reading a figure from a given series is done right; fourteen of fifteen runs had no miss on Environment 1 outside one cell. That cell is blank where the model is pointed and has to be found in another file, and seven of fifteen runs reported the blank instead of looking. On Environment 2 every run declared unavailable the two cells the key recovers from published totals, and that one position caps every model at 0.73.

4.5 Fill the coverage gaps

The task: the card data counts card-paid purchases only, and the market is purchases by any payment method. To scale up, the build needs the share of goods purchases paid by card in each country, which comes from the consumer survey, and the survey's latest wave does not cover two of the five countries. Handed the extracted figures and told this, the model produces the missing shares: on Environment 1 by the method stated on the task, eleven answers; on Environment 2 by first choosing from listed options how the two unmeasured components are scaled, then computing ten survey shares and the two components per country, 21 answers. The skill is estimating by a stated method; on Environment 2, choosing the method too.

Score: 0.93 on Environment 1, 0.85 on Environment 2; 0.88 across both. By model: gpt-6-astra 1.00 / 1.00, claude-fable-5-1 1.00 / 1.00, gemini-3.8-flash 1.00 / 1.00, Mistral Medium 3.5 0.67 / 0.60, grok-4.7 1.00 / 0.67.

Where the misses are: thirteen of fifteen runs scored 1.00 on Environment 1 and eleven of fifteen on Environment 2; the six misses are five of Mistral Medium 3.5's and one of grok-4.7's. On Environment 1, Mistral Medium 3.5's first and third runs scored 0.50. On Environment 2, Mistral Medium 3.5's first run and grok-4.7's third scored 0.00, each having scaled the two unmeasured components by a method other than the one the key records, grok-4.7 also with its two cells for one country wrong; Mistral Medium 3.5's second and third runs scored 0.90, having written figures for the country the survey does not cover instead of deriving them.

Example: the case for this stage is on the full build from the brief on Environment 1, where nobody tells the model that the survey has a gap. There, twelve of the thirteen runs that submitted reported the two uncovered countries as unavailable instead of filling the gap the way almost every one of them filled it here; the exception, Mistral Medium 3.5's first run, wrote figures at 0.47 and 2.14 times the key. On the build from the plan, ten of fourteen submitting runs did the same on all ten of those cells, and claude-fable-5-1's second run on eight of ten.

Conclusion: told the survey does not cover two countries and asked to fill the gap, 24 of 30 runs filled it the key's way, and the misses are one model's and one method choice. Asked for the whole build, almost none of them noticed the same gap: twelve of thirteen submitting runs reported those countries as unavailable.

4.6 Set the market perimeter

The task: a purchase counted in one country's data may have gone to a seller abroad, and the client wants each country's market split between sellers at home and sellers abroad. Handed the figures so far, the model establishes for each country the share of its online buying or selling that crosses the border, on the basis the task names: five answers on Environment 1, ten on Environment 2 including how each share was obtained. The skill is measuring a cross-border share on the right basis.

Score: 0.73 on Environment 1, 0.93 on Environment 2; 0.86 across both. By model: gpt-6-astra 0.67 / 1.00, claude-fable-5-1 1.00 / 1.00, gemini-3.8-flash 1.00 / 1.00, Mistral Medium 3.5 0.00 / 0.63, grok-4.7 1.00 / 1.00.

Where the misses are: on Environment 1, eleven of fifteen runs scored 1.00. One gpt-6-astra run made no submission because the model's provider returned an empty reply, no text and no action, on the turn after the model's own tool output had printed all five shares right to six decimals. Mistral Medium 3.5 scored 0.00 on all three runs, by two routes: its first run read the rows for payments made in person as cross-border, and its second and third double-counted the base by grouping on the location column; its second run's Germany cell also departs from a decision the key records. On Environment 2, thirteen of fifteen runs scored 1.00; Mistral Medium 3.5's first and third scored 0.45, three cells each: the key records that the share bought from abroad is the complement of the own-country share, and these runs summed the published destinations instead.

Conclusion: four of the five models have this stage solved, 23 of 23 runs that submitted at 1.00 across both environments. The losses are one model's, by misreading the source on Environment 1 and by a decision taken the other way on Environment 2, and one provider returning nothing after the model had already printed the right answer.

4.7 Finalize the inputs

The task: the card purchase counts are not the market yet, and this stage turns them into the build's inputs. On Environment 1, 85 cells: drop the merchant categories classified as not goods, split the goods count by product segment, take the card share per segment from two survey waves, scale card-paid purchases up to all payment methods, and set the average ticket. On Environment 2, 30 answers: for each component, whether and where the source publishes the amounts that have to come out, and the 25 exclusion figures themselves. The skill is carrying the definition through to the numbers: what is scaled up, what is taken out.

Score: 0.85 on Environment 1, 0.55 on Environment 2; 0.78 across both. By model: gpt-6-astra 0.95 / 0.60, claude-fable-5-1 0.98 / 0.55, gemini-3.8-flash 1.00 / 0.77, Mistral Medium 3.5 0.33 / 0.35, grok-4.7 1.00 / 0.50.

Where the misses are: on Environment 1, eight of fifteen runs scored 1.00 on all 85 cells, claude-fable-5-1 twice, gemini-3.8-flash and grok-4.7 three times each. The older survey wave has a payment category the newer one dropped, and four of fifteen runs computed shares from the older wave without handling it (gpt-6-astra all three, claude-fable-5-1 once), the same three wrong figures each time, twelve of 1,275 cells. Mistral Medium 3.5 scored 0.67, 0.17 and 0.17: it reproduced the segment counts it was given without ever applying the split, and on two runs reported the older wave's share as unavailable without opening the survey. On Environment 2, one gpt-6-astra run made no submission, an empty reply from the provider on its twelfth turn; fourteen submitted. The source publishes three categories of amounts to take out; one of them was reported by no run of any model, none of fourteen, while the two beside it in the source's own list were answered. A second category, ten cells, was left unanswered by eleven of fourteen. Four runs wrote "not published at any level" for a component whose amounts the source does publish at a broader level, claude-fable-5-1 on every run and grok-4.7 once. Over the fourteen that submitted the mean is 0.59; the 0.55 above counts the non-submission as zero.

Example: claude-fable-5-1 on Environment 2. At this stage each of its three runs reported one exclusion category, left the second and third blank entirely, and scored 0.55. In the same runs' full build from the plan it built the two measured components from the gross published figures and subtracted nothing, not even the exclusions it had itself just reported at this stage; seven of its fourteen wrong cells there are, to the sixth decimal, the published figure plus the exclusions it had reported.

Conclusion: the models remove the exclusions they are told about. One category sits in the source's own list and none of the fourteen runs found it. Reporting an exclusion at this stage did not mean applying it later: claude-fable-5-1 reported exclusions here and then built the same components in its full build with nothing removed.

4.8 Compute the result

The task: handed the finished inputs, the model produces the market per country, purchases times average purchase value, 25 cells on Environment 1; on Environment 2, each component net of its exclusions, the subtotal and the total per country, 35 cells, with a gap declared wherever an input does not exist. The skill is arithmetic.

Score: 0.93 on Environment 1, 0.86 on Environment 2; 0.89 across both. By model: gpt-6-astra 0.67 / 1.00, claude-fable-5-1 1.00 / 1.00, gemini-3.8-flash 1.00 / 1.00, Mistral Medium 3.5 1.00 / 0.29, grok-4.7 1.00 / 1.00.

Where the misses are: on Environment 1, fourteen of fifteen runs scored 1.00, every run that submitted, 350 of 350 cells; gpt-6-astra's second run made no submission, the same empty reply from the provider as at stage 6, on a different run, two turns in. On Environment 2, twelve of fifteen runs scored 1.00; Mistral Medium 3.5 scored 0.00, 0.03 and 0.83, errors in the arithmetic itself.

Conclusion: given the right inputs, four of the five models got the arithmetic right on every run that submitted, on both environments. In the full builds the figures were lost earlier, on the sources and the decisions, as 4.9 to 4.11 show; where a full build's own arithmetic did not hold, the calculation checks in 4.12 record it.

4.9 The full build from the brief

The task: handed the brief and the files available and nothing else, the model delivers the final market figure per country and says which datasets and series it used: on Environment 1 the five market values plus the fourteen datasets it accepted or rejected, with the series it names scored against the 21 values in the key; on Environment 2 the three published final figures and two gaps, against 38. There is no calculation check, because only the final figure is submitted. It is every stage above in one run, with no stage answers supplied and no prompt to stop and decide.

Score: 27 of 30 runs submitted, thirteen of fifteen on Environment 1 and fourteen of fifteen on Environment 2, the three that did not all gemini-3.8-flash, out of turns. Final figures right: none, on any of the 27. Sources right, pin by pin: 49% on Environment 1 (155 of 315), 37% on Environment 2 (209 of 570), 41% across both. On Environment 1 the datasets the build correctly leaves out are scored separately, 14 of 14 on every gpt-6-astra run, none filed by Mistral Medium 3.5 on two runs; see 4.12.

Where the misses are: on Environment 1, two things account for every lost cell. The two countries the survey does not cover were reported as unavailable rather than filled by twelve of thirteen submitting runs, 24 of 65 final cells. The three covered countries were wrong on every run, 39 of 39 cells: we recomputed one run (claude-fable-5-1, first run) and found five differences from the key, each alone enough to lose the cell: which payment codes count as card, which average ticket to use, the base count read from a series that exists but is not the one the build reads, one merchant category counted inside a broader one it already belongs to, and one divider taken at the wrong level. The first two moved that run's figure for one country by 33% to 59%; the base count and the double count moved it by under 1%. Six of thirteen runs read the base from a half-yearly series where the build reads the quarterly one, gpt-6-astra and claude-fable-5-1 on every run. On Environment 2 the error is scope: nine of fourteen submitting runs built the market from one component, or one plus a single survey series (gemini-3.8-flash twice, claude-fable-5-1 three times, Mistral Medium 3.5 once, grok-4.7 three times). gpt-6-astra declared all five sources on all three runs and still wrote no right figure: one run in raw units, a million times off, two in the right magnitude and 12% to 52% short. gemini-3.8-flash's two submitting runs reached the same three figures by different routes.

Example: gpt-6-astra's first run on Environment 2 had 37 of 38 values right and all five sources declared, and produced three wrong final figures. It found the components, the survey shares and the cross-border source, and still got all three figures wrong.

Conclusion: from the brief alone, no run of any model produced a right figure on either environment, 27 of 27 that submitted, and the recomputation shows why: on the runs we reconstructed, between two and five separate decisions, each one enough on its own to lose the cell, and the model flagged none of them. gpt-6-astra found every source on Environment 2 and was still a million times off on one run. The sources were found. The figures were lost on the decisions made between the sources and the figure, and no model flagged that it was making one.

4.10 The full build from the plan and datasets

The task: handed the brief, the plan and the chosen files with their roles (and, on Environment 1, the six product-segment definitions), the model delivers the final figures and the intermediate cells: 25 cells, 21 values and ten calculation checks on Environment 1; 35 cells, 38 values and seven checks on Environment 2. It is stages 3 to 8 in one run.

Score: 27 of 30 runs submitted, fourteen of fifteen on Environment 1 and thirteen of fifteen on Environment 2, the three that did not all gemini-3.8-flash. Final figures right: none on Environment 1 in fourteen runs; on Environment 2 no run right; the one right final figure in 27 runs is the France total on claude-fable-5-1's second run, whose score is 0.00 because its wrong cells outnumber its right ones; the best figures score on this build is claude-fable-5-1's first run, 0.14, on component cells; gpt-6-astra's two scoring runs are at 0.03. Calculations hold: 87 of 150 checks on Environment 1 and 68 of 105 on Environment 2, 61% across both. Sources right, pin by pin: 58% on Environment 1, 66% on Environment 2 (375 of 570), 63% across both.

Where the misses are: on Environment 1, the two uncovered countries were reported as unavailable on all ten of their cells by ten of fourteen runs and on eight of ten by claude-fable-5-1's second; in claude-fable-5-1's transcripts the idea of filling the gap never comes up. The three covered countries were wrong on fourteen of fourteen runs. Four runs named every series right, gpt-6-astra's three and claude-fable-5-1's third, grok-4.7's second next at 0.94; claude-fable-5-1's third run had 21 of 21 values right and still scored 0 on the figures. gpt-6-astra's whole loss is two choices, which payment codes count as card and which average ticket to use. On Environment 2 the error is which base the unmeasured components are scaled from. Two of gpt-6-astra's three runs took the exclusions out of the measured components first and then scaled the unmeasured ones from the netted figures; the key scales them from the other base. That one choice cost 17 of 35 cells, with 38 of 38 values and seven of seven calculation checks right. grok-4.7's first and third runs had 37 of 38 values and seven of seven checks right and no right figure. claude-fable-5-1's first run did the opposite, built the measured components gross with nothing removed, fourteen wrong cells every one equal to a published figure or a sum of them, and still holds the best figures score on this build of any run, five of 35; its two later runs started removing exclusions and broke the cells the first had right. gemini-3.8-flash's one submitting run pulled the wrong series for two components, one of them 667 times the key.

Example: gpt-6-astra's first run on Environment 2 had 38 of 38 values and seven of seven calculation checks right and lost 17 of 35 cells; the difference is one decision: which base the unmeasured components scale from.

Conclusion: handing over the plan and the datasets lifts sources right to 63% and calculations hold to 61%, and does not lift the figure: no run right on Environment 1, one right final figure on Environment 2, in 27 runs. gpt-6-astra scored 38 of 38 on sources and 7 of 7 on calculation checks on one run and lost 17 of 35 cells to one decision, which base to scale from. The decisions inside the build were still the model's, and they were still wrong.

4.11 The full build from the extracted figures

The task: handed the extracted figures, the model delivers the final figures and the intermediate cells. On Environment 1 the coverage shares are handed too, so stages 6 to 8 remain, perimeter, finalizing and arithmetic: 25 cells and ten calculation checks. On Environment 2 only the three measured components are handed, so stages 5 to 8 remain, scaling the unmeasured components, perimeter, exclusions and arithmetic: 35 cells and seven checks. Sources are not scored, because they were handed over.

Score: 30 of 30 runs submitted. Runs with every final figure right: 2 of 30, gpt-6-astra's two Environment 1 runs at 25 of 25 cells; 28 of 30 missed. Final figures right on Environment 1: 10 of 75, all ten gpt-6-astra's; on Environment 2 none. Calculations hold: 118 of 150 checks on Environment 1 and 54 of 105 on Environment 2, 67% across both.

Where the misses are: on Environment 1, claude-fable-5-1 and gemini-3.8-flash, three runs each, 20 of 25 cells wrong each time, as before: claude-fable-5-1 on two errors we reproduced to the sixth decimal, it never scaled the card-paid count up to purchases by all payment methods although the all-methods figure was given for every country (it copied it into its first draft and dropped it in the rewrite), and it applied one set of shares to all five countries past a scope note printed six times beside those numbers; gemini-3.8-flash with one calculated cell reported as not published and its own arithmetic inconsistent by 24% to 50% across four countries. grok-4.7, three runs, 20 of 25 wrong, with submissions identical to the last decimal on three separate runs, the five tickets exact and ten of ten calculation checks right, on the same omitted scale-up plus a seller split taken from the payments file's terminal location. Mistral Medium 3.5, three runs, the same five counts each time, matching claude-fable-5-1's rows digit for digit, the omitted scale-up with a pooled divider applied to all five countries on top. Four of fifteen runs reported the ten cells that split each market between sellers at home and abroad as unavailable, gpt-6-astra once and Mistral Medium 3.5 three times. The scale-up was omitted on ten of the twelve runs we read. On Environment 2, the measured components were carried through gross, with no exclusions removed, in fourteen of fifteen runs; the exception, one gpt-6-astra run, removed them exactly and then scaled the unmeasured components from the netted base, the same choice as in 4.10. The two unmeasured components were reported as a gap by ten of fifteen runs, which also fails four of the seven calculation checks for eight of them and all seven for Mistral Medium 3.5's second and third; Mistral Medium 3.5's first run wrote zeros instead, a committed wrong figure. gpt-6-astra's other two runs computed them by a method other than the one the key records.

Example: on claude-fable-5-1's first run on Environment 1, every figure it read off the page is right, five of 25, and every figure it derived is wrong, 20 of 25, because of one scale-up it left out although the figure it needed was on the page, and one scope note it read past.

Conclusion: even with every extracted figure handed over, 28 of 30 runs missed the final figure, and the two that got it were both gpt-6-astra on Environment 1. The misses are the two decisions of stage 7: what to take out, and which base to scale up from. With nothing left to find, those two decisions alone cost 28 of 30 runs the figure.

4.12 Full builds

Environment 1

The model is givenSources rightCalculations holdFinal figures right, netRuns with every figure exactDistance from the right figureWithin 25%More than double
The brief onlySources right0.45Calculations hold–Final figures right, net0.00 (datasets correctly left out 0.59)Runs with every figure exactnoneDistance from the right figure35%Within 25%16 of 41More than double2 of 41
The build plan and datasetsSources right0.54Calculations hold0.58Final figures right, net0.00Runs with every figure exactnoneDistance from the right figure37%Within 25%13 of 45More than double11 of 45
The extracted figuresSources right–Calculations hold0.79Final figures right, net0.13Runs with every figure exact2 of 15 (gpt-6-astra)Distance from the right figurethe rest 24%Within 25%46 of 75More than doublenone

Sources right: the share of the series the build needs that the model named correctly; not scored where the sources were handed over with the extracted figures. Calculations hold: the share of the model's own calculations that hold across the cells it submitted; not scored from the brief, where only the final figures are submitted. Final figures right, net: right final cells minus wrong ones, floored at zero, as a share of the cells asked; the full-builds panel's 'final figures exactly right' is a different measure, right figures over figures submitted. On Environment 1, the five final market figures and the datasets the build correctly leaves out are scored separately from the brief; none of the five figures matched the key on any run of any model. Datasets correctly left out is the share of the fourteen datasets the key rejects that the model also rejected, right minus wrong, floored at zero. Runs with every figure exact: runs where every final figure matched the key. Distance from the right figure is not a grader score: it is measured afterwards, per final figure, against the key, and the median is shown; 0% is exact.

Environment 2

The model is givenSources rightCalculations holdFinal figures right, netRuns with every figure exactDistance from the right figureWithin 25%More than double
The brief onlySources right0.37Calculations hold–Final figures right, net0.00Runs with every figure exactnoneDistance from the right figure55%Within 25%7 of 40More than double10 of 40
The build plan and datasetsSources right0.66Calculations hold0.65Final figures right, net0.01Runs with every figure exactnoneDistance from the right figure25%Within 25%17 of 32More than double11 of 32
The extracted figuresSources right–Calculations hold0.51Final figures right, net0.00Runs with every figure exactnoneDistance from the right figure5%Within 25%32 of 35More than doublenone

The counts are of final figures submitted. Nine of the eleven figures more than double from the plan are Mistral Medium 3.5's, 13 to 95 times the key.

The whole build, per model

Each model asked for the complete set of final figures, from three starting points, three runs each. Cells: final figures right, net / runs with every figure exact / distance from the right figure (figures counted); on the Environment 1 brief row, datasets correctly left out is shown on its own row.

The model is givengpt-6-astraclaude-fable-5-1gemini-3.8-flashMistral Medium 3.5grok-4.7
Environment 1, the brief onlygpt-6-astra0.00 / 0 of 3 / 17% (9)claude-fable-5-10.00 / 0 of 3 / 43% (9)gemini-3.8-flash0.00 / 0 of 3 / 35% (3)Mistral Medium 3.50.00 / 0 of 3 / 98% (11)grok-4.70.00 / 0 of 3 / 10% (9)
Environment 1, the brief only, datasets correctly left outgpt-6-astra1.00claude-fable-5-10.86gemini-3.8-flash0.33Mistral Medium 3.50.00grok-4.70.76
Environment 1, the build plan and datasetsgpt-6-astra0.00 / 0 of 3 / 31% (9)claude-fable-5-10.00 / 0 of 3 / 80% (11)gemini-3.8-flash0.00 / 0 of 3 / 87% (6)Mistral Medium 3.50.00 / 0 of 3 / 97% (10)grok-4.70.00 / 0 of 3 / 8% (9)
Environment 1, the extracted figuresgpt-6-astra0.67 / 2 of 3 / 0% (15)claude-fable-5-10.00 / 0 of 3 / 24% (15)gemini-3.8-flash0.00 / 0 of 3 / 24% (15)Mistral Medium 3.50.00 / 0 of 3 / 34% (15)grok-4.70.00 / 0 of 3 / 24% (15)
Environment 2, the brief onlygpt-6-astra0.00 / 0 of 3 / 48% (8)claude-fable-5-10.00 / 0 of 3 / 48% (9)gemini-3.8-flash0.00 / 0 of 3 / 40% (6)Mistral Medium 3.50.00 / 0 of 3 / 72% (9)grok-4.70.00 / 0 of 3 / 345% (8)
Environment 2, the build plan and datasetsgpt-6-astra0.02 / 0 of 3 / 3% (8)claude-fable-5-10.05 / 0 of 3 / 16% (5)gemini-3.8-flash0.00 / 0 of 3 / 149% (3)Mistral Medium 3.50.00 / 0 of 3 / 49 times the key (9)grok-4.70.00 / 0 of 3 / 25% (7)
Environment 2, the extracted figuresgpt-6-astra0.01 / 0 of 3 / 24% (8)claude-fable-5-10.00 / 0 of 3 / 5% (9)gemini-3.8-flash0.00 / 0 of 3 / 5% (9)Mistral Medium 3.50.00 / 0 of 3 / no figure submitted (0 of 9)grok-4.70.00 / 0 of 3 / 9% (9)

The distance is the median over the figures in brackets; with three to fifteen figures it varies a lot, so read it with the count. gemini-3.8-flash made no submission on two of its three Environment 1 runs from the brief and one with the plan, and on one Environment 2 run from the brief and two with the plan. gpt-6-astra got every figure right on two of its three runs with the extracted figures on Environment 1, ten of fifteen figures exact; no other run on either environment did. Its 3% on Environment 2 with the plan is the run in 4.10: close on every cell and exact on none. Mistral Medium 3.5's 98% and 97% on Environment 1 are figures about one fiftieth of the key. On Environment 2 its figures from the brief are 13% to 68% of the key, from the plan 13 to 95 times it, and from the extracted figures it submitted no market figure on any run. grok-4.7's 8% to 10% on Environment 1 cover France, Spain and Italy only; it reported Germany and the Netherlands as unavailable on all six of those runs. Datasets correctly left out is the share of the fourteen datasets the key rejects that the model also rejected, right minus wrong, floored at zero; a run that did not submit counts zero. gemini-3.8-flash's 0.33 is 14 of 14 on its one submitting run and zero on two runs that ran out of turns.

4.13 Per model and cost

0510152025300.500.600.700.800.90Cost per full run of all eleven, USDMean of the eight stage scoresgpt-6-astra Env 1gpt-6-astra Env 2claude-fable-5-1 Env 1claude-fable-5-1 Env 2gemini-3.8-flash Env 1gemini-3.8-flash Env 2Mistral Medium 3.5 Env 1Mistral Medium 3.5 Env 2grok-4.7 Env 1grok-4.7 Env 2
Environment 1Environment 2

Cost is from exact token counts at list prices, range over three runs. The score is the unweighted mean of the eight stage scores, so a four-answer stage counts as much as a 105-answer one. gemini-3.8-flash is at a promotional rate that ends on 31 December 2026, after which its cost doubles; its 0.78 on Environment 1 carries a 0.00 on pinpoint the exact series, three runs that ran out of turns, worth 0.12 of its mean.

Stages, mean of three runs, Environment 1 / Environment 2

Stagegpt-6-astraclaude-fable-5-1gemini-3.8-flashMistral Medium 3.5grok-4.7
Structure the modelgpt-6-astra0.89 / 1.00claude-fable-5-10.89 / 1.00gemini-3.8-flash0.89 / 1.00Mistral Medium 3.50.96 / 0.88grok-4.70.89 / 1.00
Select the sourcesgpt-6-astra0.60 / 1.00claude-fable-5-10.27 / 0.33gemini-3.8-flash0.43 / 0.27Mistral Medium 3.50.47 / 0.20grok-4.70.56 / 0.20
Pinpoint the exact seriesgpt-6-astra0.88 / 0.77claude-fable-5-10.96 / 0.85gemini-3.8-flash0.00 / 0.16Mistral Medium 3.50.70 / 0.44grok-4.70.90 / 0.78
Extract the figuresgpt-6-astra0.97 / 0.73claude-fable-5-10.97 / 0.73gemini-3.8-flash0.90 / 0.73Mistral Medium 3.50.92 / 0.73grok-4.71.00 / 0.73
Fill the coverage gapsgpt-6-astra1.00 / 1.00claude-fable-5-11.00 / 1.00gemini-3.8-flash1.00 / 1.00Mistral Medium 3.50.67 / 0.60grok-4.71.00 / 0.67
Set the market perimetergpt-6-astra0.67 / 1.00claude-fable-5-11.00 / 1.00gemini-3.8-flash1.00 / 1.00Mistral Medium 3.50.00 / 0.63grok-4.71.00 / 1.00
Finalize the inputsgpt-6-astra0.95 / 0.60claude-fable-5-10.98 / 0.55gemini-3.8-flash1.00 / 0.77Mistral Medium 3.50.33 / 0.35grok-4.71.00 / 0.50
Compute the resultgpt-6-astra0.67 / 1.00claude-fable-5-11.00 / 1.00gemini-3.8-flash1.00 / 1.00Mistral Medium 3.51.00 / 0.29grok-4.71.00 / 1.00

gemini-3.8-flash: 0.00 on Environment 1's stage 3, pinpoint the exact series, is three runs that ran out of turns without submitting. gpt-6-astra: the two 0.67 on Environment 1 are one run each that got an empty reply from the provider after its own tool output had printed the answer; not retried. Mistral Medium 3.5 is mistral-medium-2604 on Mistral's API. Its 0.29 on compute the result, Environment 2, is three runs at 0.00, 0.03 and 0.83. grok-4.7's 0.67 on fill the coverage gaps, Environment 2, is 1.00, 1.00 and 0.00; that stage there is a single scored choice, so a run is either right or wrong. grok-4.7's 0.56 on select the sources, Environment 1, is 0.77, 0.47 and 0.43.

A full run of all eleven costs $2.9 to $4.1 with Mistral Medium 3.5 on Environment 1 and $4.1 to $4.6 on Environment 2, $4.8 to $6.4 and $6.2 to $7.7 with gpt-6-astra, $11.4 to $12.2 and $9.8 to $10.7 with gemini-3.8-flash, $18.1 to $19.2 and $21.8 to $27.2 with grok-4.7, and $19.0 to $21.0 and $19.9 to $21.7 with claude-fable-5-1. grok-4.7 takes 350 to 390 model turns per run and its reasoning is billed as output. Mistral Medium 3.5's first Environment 1 run, before response caching was enabled, cost $16.7 and is outside the range. claude-fable-5-1 returns an identical score on six of the eight stages across all three runs on both environments. gpt-6-astra has the best single run on both environments and the widest spread between runs. gemini-3.8-flash ran out of turns eleven times across the two environments.

5. Takeaways

Across the five models, five of eight stages score 0.86 or better, and no full build from the brief reaches a right figure. The figure is lost on a few decisions that carry through the build, each moving it by 5% to 59% on the runs we reconstructed. The two stages where the models score lowest, and differ most from each other, are select the sources and pinpoint the exact series. A full run costs about $3 to $27 depending on the model.

6. What is next

New environments follow, on market sizing and on other assignments in finance and strategy, each built to the same depth. Major model releases get three runs on every environment. The score is exact and reproduces from the stored submission, so it can serve as a reward signal today. As the set grows we train on some environments and hold out the rest, and publish how much of the improvement holds on assignments the model was not trained on.