matplotlib
grade
reliability
overall
Matplotlib is a programmatic plotting library that enables agents to generate charts, diagrams, and posters by executing Python code. The tool requires writing layout, styling, and structural commands into a script and running it to export files which then require external validation. Because execution runs headlessly, the agent cannot directly view the visual output during the design process.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
The agent designs standard visualizations using Matplotlib's object-oriented interface. Standard methods like subplots, bar charts, and scatter plots work adequately because their parameters correspond directly to data series and match predictable API patterns. Creating advanced multi-axis layouts using methods like twinx and applying styling options like hatch patterns also works adequately, as these properties can be configured explicitly in Python code.
However, constructing multi-element compositions like data cards and custom legends works poorly. The agent must position titles, borders, and text blocks by specifying explicit coordinate systems and bounding boxes. Because the agent lacks direct visual feedback, calculating precise spatial offsets is highly iterative and error-prone, where minor alterations frequently cause overlapping text or clipped borders without throwing structural errors.
State persistence across multiple sequential executions works poorly. Repeated attempts to run or modify code in an ongoing session frequently fail due to execution environment unreliability. Conversely, resuming development from a clean state works well, since a fresh agent resuming from saved code files can modify the layout without breaking previously established structures.
Fit
Matplotlib is a good fit for agents executing automated data visualization tasks that require standard charts and structured layouts defined through explicit code parameters. It is effective when the layout is simple or when exact coordinate alignment is not critical to the final output.
The tool is a poor fit for tasks that demand complex page designs, custom templates, or precise canvas layouts. When an agent must design visually dense media containing multi-axis charts, multi-column metric boxes, and precise text callouts, the lack of real-time visual feedback makes the process inefficient.
Notes
This diagram illustrates the agent's iterative write-execute-verify workflow when generating artifacts with Matplotlib.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 9 of 16 cross-checked cells (69%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| recursion.coherence | -0.85 | +0.70 | 1.55 |
| interface.coherence | -0.70 | +0.80 | 1.50 |
| disclosure.verifiability | -0.40 | +0.60 | 1.00 |
| human.verifiability | -0.20 | +0.80 | 1.00 |
| loop.determinism | -0.30 | +0.60 | 0.90 |
| human.coherence | -0.20 | +0.60 | 0.80 |
| human.safety | -0.10 | +0.40 | 0.50 |
Ground truth
1 of 1 requirements met · deterministic checker, no model judgment- poster produced — gapminder_infographic.pdf
Reliability (pass^k): one attempt succeeds 40% of the time · three in a row 0% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
| | 23 | 16/23 | 6 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.8619
- hard-measured cells
- 53%
- teaching source
- matplotlib (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.