matplotgym

1. Overview

matplotgym measures whether an agent can turn a scientific dataset into the figure the paper's authors drew. Each of the 171 tasks comes from a published paper whose authors released both their data and the code that drew the figure. The agent receives the data and a one-sentence claim, and decides what to plot.

The benchmark tests analytical judgment, not plotting skill. What separates the models is choosing the right subset, aggregation, encoding and panel layout. Nine models were evaluated with at least three attempts per task, for 6,147 scored episodes.

2. Source material

2.1 Real figures, real decisions

The reference is the figure produced by running the authors' own plotting code, unmodified, on their released data. We do not redraw, simplify or invent figures. The ground truth is a decision a working scientist made about their own results. The authors' plotting code is used only to build the reference spec; it is not shipped with the task.

2.2 Raw data, not answers

The agent receives the rows as released, including columns and values the figure does not use. In the example task the data holds runs at two network sizes, and the authors plot only the 16-node runs. A gate blocks any task whose files contain a pre-aggregated table, a column named after the conclusion, or a header that describes the comparison.

2.3 Licensing

Every file (task.yaml, reference_spec.json and the workspace) is under a license that permits commercial use. We check each task's shipped files rather than the repository as a whole, and read each upstream's own terms.

3. What is tested

3.1 Six decisions

Every task is scored on the decisions below. A task exercises the ones its reference figure makes; all six are present in 45 of the 171 tasks, and every task has at least three.

A weight is that decision’s share of a task’s score; the six add up to 1. Each decision is scored from 0 to 1, and the task score is their weighted average. A task only counts the decisions its reference figure makes. When one is missing, its weight is dropped and the rest are scaled up to fill the gap. On a task with no overlays, for example, data selection counts for 0.37 / 0.85, about 44% of the score.

3.2 What each decision measures, and why

3.3 What the claim gives away

The claim names the conditions and quantities at issue. It never names a chart type, an axis assignment or an aggregation, because those are the decisions under test.

3.4 Fields

The tasks span nine fields. ML & systems holds 81 of them; five fields hold fewer than ten each, too few to rank models on alone. The results table above can be split by field.

4. Environment interface

4.1 What the agent receives

The claim, a listing of its working directory, data/SCHEMA.md describing columns and units, the data files, and a one-line make_figure.py stub. It never receives the reference spec, the authors' plotting code, the published figure, or the task's list of scored decisions.

4.2 Actions

One action per turn, as a fenced block: bash runs a shell command, python runs a throwaway script, write <path> writes a file, and done ends the episode. Python has numpy, pandas and matplotlib. The budget is 14 turns and 120 seconds per command.

The protocol is plain text and identical for every model. Native tool calling is off for all of them, so no provider's tool format is an advantage.

4.3 Submission

Only make_figure.py is scored. When the episode ends, the harness runs the script again in a fresh process and scores the figure object it builds. The image the agent saved is never looked at, and the agent sees no score during the episode.

5. Reward design

5.1 Structure

Both figures are reduced to a spec: panels, marks, the values each mark plots, axis scales and overlays. The six decisions are compared one by one and combined by weight. Decisions a task does not exercise leave the denominator, so a task with no error bars gives no credit for drawing none. A script that crashes or draws nothing scores 0.

5.2 More than one right answer

• Style is never read. Palette, fonts, figure size, tick placement, legend position and resolution appear nowhere in the scoring code. Nor do series order or plotting idiom.
• Near-equivalent encodings earn partial credit. Bars where the authors drew points, or a histogram drawn as bars, earn half on the chart-type signal rather than zero.
• Faithful additions cost less than omissions. An extra series costs half what a missing one costs. Leaving out what the claim is read against is a failure; adding context is at worst clutter.
• Units do not matter where shape does. Heatmaps and scalar fields are compared up to a change of scale and offset.

5.3 Where models lose points

Across all 6,147 episodes, choosing the wrong data costs more than everything else except encoding combined.

5.4 Observed model failures

Across all episodes on the benchmark, we have observed the following failure modes:

5.4.1 Choosing the wrong data

• Plotting the right series with the wrong filter, subset or aggregation
• Reducing a series over time to a single average
• Plotting conditions or runs the authors left out
• Dropping a condition the authors plotted

5.4.2 Encoding the data differently

• Drawing a series as a different kind of mark
• Stacking bars the authors grouped, or joining points they left separate
• Dropping the colour mapping that tells conditions apart

5.4.3 Leaving out overlays

• Leaving out error bars or confidence bands
• Leaving out a fitted line or a reference line the claim is read against

5.4.4 Laying out panels differently

• Splitting one comparison across panels so the series no longer share axes
• Merging the authors’ panels into one crowded plot
• Dropping some of the authors’ panels

5.4.5 Choosing different axis scales

• Adding a log scale the authors didn’t use
• Using a linear scale where the authors used log
• Sharing axis limits across panels the authors scaled separately, or the reverse

5.4.6 Failing to complete the task

• Ending the episode without submitting make_figure.py
• Drawing the figure in a scratch command but never writing it to the submission
• Submitting a script that crashes
• Submitting a script that runs but draws nothing

6. False positives and reward hacking

A score is only useful if it cannot be earned without doing the task. Each promise below says how we know.

• The agent cannot read the answer. Four static scans run on every task, and a task that fails any of them is blocked: everything the agent can see is searched for the answer's tokens, the reference's derived columns are checked against the shipped ones, filenames and headers are linted, and claims that list their own answer are flagged. We also audited every command and every output across all 6,147 episodes: no reference spec, hidden task field or author code ever reached an agent.
• The scorer cannot be tampered with. Scoring re-runs the submitted script after the episode, so a saved image or a printed number cannot stand in for a figure. None of the 5,983 submitted scripts we can read imports the harness or patches matplotlib. The other 164 episodes submitted no script and scored 0.
• A non-answer scores well below a real attempt. We build the lazy figures ourselves (plot every column, plot nothing, plot the raw table) and score them on all 171 tasks. A task where one of them scores too close to a real attempt is flagged, because it can no longer tell a non-answer from analysis.
• No credit for style or volume. The scoring code reads nothing cosmetic (5.2) and charges for extra series (D6), so neither polish nor plotting everything raises a score.
• The reward is deterministic. No model and no network call sits in the scoring path. We re-scored recorded scripts and got the same score on every run.
• Every task can be solved. Each reference, scored against itself, gets 1.000 on all 171 tasks, so no attempt is charged for a fault in the harness. We also check that each task can be solved in practice: a strong model is told the decisions the authors made (which rows, which chart, which panels) and asked to draw the figure. It has to score 0.95 or higher, or the task is blocked: if it cannot, the fault is in the task, not the model.
• Every task discriminates. Five independent attempts must produce a real spread of scores, and deliberately degraded versions of the reference must score clearly lower. A task that fails either check is blocked.
• The criteria are stated up front. The agent's prompt lists the decisions it is judged on and the ones it is not (palette, fonts, size, ticks, legend position). There is no hidden criterion.

7. Human review

The automated gates check that a task is solvable, discriminating and free of leaks. They cannot check that a claim is faithful to the paper it came from; only an author knows that. So for a selected set of about twenty tasks, we met with their authors to review each claim against the work it came from, and asked for their input on the weights of the rubric's components.

8. Fairness invariants

An agent is charged only for a decision it could have made correctly from what it was given.

• One mistake costs once. The six decisions are scored independently, so a wrong choice is charged in the decision responsible and nowhere else.
• Only exercised decisions count. A decision the reference does not make is not scored.
• Format slips are not failures. The parser accepts a mislabelled code fence, such as sh for bash. A strict parser once cost one provider about 40% of its turns, which measured formatting rather than judgment.
• One reminder to submit. An agent that ends with the stub untouched is told once that only make_figure.py is scored. The reminder says nothing about the figure, and its turn counts against the budget.
• The agent tests on the scorer's interpreter. The python the agent runs is the one that runs its script for scoring, with the same libraries.
• Solved means every time. A task counts as solved only when all three attempts reach 95, so a single lucky attempt is not a solve.

9. Limits

• The open-weight runs were audited, not sealed. Gemma 4 31B and Qwen3.8-27B ran on our GPU workers, where the shell could reach the task store. In 118 episodes Qwen3.8-27B ran commands outside its workspace, mostly while looking for its own files; one printed 50 lines of harness source code, which contains no answers. No episode opened an answer file (section 6).
• Small fields do not rank. Five fields have fewer than ten tasks, and the results table marks them as too small to rank models on.