matplotlib
grade
reliability
overall
matplotlib: plotting with Python
Matplotlib is a Python library that allows an agent to programmatically generate and save static graphic files like PNG or SVG. The tool operates through Python scripts using an object-oriented syntax of figures and axes. This interface works well for an agent because its standard API is highly documented and produces predictable, reproducible results across repeated script executions.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
Teaching sources compared
Teaching source barely matters for matplotlib: sources land within a grade of each other (docs B, skill B).
The agent performed better when instructed by the specialized skill material than when using the standard reference documentation. The specialized skill produced higher quality results, leading to a visible gap in human evaluation.
The concrete reason for this difference is that the specialized skill provides direct, task-oriented examples that match the agent's constraints. While the general documentation covers basic plotting mechanics, it lacks the targeted formatting instructions needed for specialized visuals. The skill-based instructions guide the agent to configure advanced choices like colorblind-safe palettes and precise text alignments, resulting in cleaner and more compliant figures.
| Source | Grade | Renders | Disclosure | Interface | Loop | Recursion | Human |
|---|---|---|---|---|---|---|---|
| official docs ★ | B | ✓ | +0.30 | +0.42 | +0.49 | +0.30 | +0.32 |
| skill | B | ✓ | +0.39 | +0.59 | +0.33 | +0.25 | +0.15 |
Same harness, same reference agent, same briefs, same trial count — the only variable is how the agent was taught the tool. ★ = best source.
The experience
The agent interacts with the tool by writing Python execution scripts that sequence loading data, plotting markers, configuring labels, and saving files. This workflow flows well because the standard plotting API is highly predictable, meaning the agent can structure a complete visualization script from start to finish without requiring incremental execution. The library's core file-saving operations work well since they write files directly to the disk directory, presenting no safety risks to the runner environment.
However, the agent fights with visual feedback because the tool does not provide built-in programmatic evaluation of the rendered output. Since the agent cannot inspect the final PNG or SVG file directly, it cannot detect issues like visual overlaps, label clipping, or incorrect tick placements without human feedback or external vision verification. This lacks verifiability, forcing the agent to rely entirely on code correctness rather than visual checking.
Furthermore, managing complex coordinate systems and layout configurations often leads to fights. Setting up specialized alignments, such as placement of legends or text annotations at specific data coordinates, requires exact positioning logic that can break if the input data range shifts. When plotting multiple series with varying scales, the API requires explicit adjustments like setting a logarithmic scale manually, which can cause layout clutter if omissions are made.
Fit
Matplotlib is a good fit for agents operating in isolated code-execution environments tasked with transforming raw datasets into static, standard reports. Because it requires no interactive graphics framework, the agent can reliably prepare charts like scatter plots or bar graphs using basic Python scripts. The abundance of online training data for this library ensures the agent produces syntax-correct code on its first attempt.
The library is a poor fit for agents that require real-time visual adjustments or lack external visual verification channels. Since the agent cannot programmatically inspect the spatial coordination of the rendered elements, it is unable to resolve layout defects such as overlapping legend text or squeezed axes. It is also a poor fit for tasks that demand highly responsive, web-based interactive charts where static image outputs are insufficient.
Notes
This diagram illustrates the agent's python execution loop when generating and correcting a plot.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 7 of 16 cross-checked cells (76%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| recursion.coherence | -0.85 | +0.40 | 1.25 |
| interface.coherence | -0.70 | +0.50 | 1.20 |
| disclosure.verifiability | -0.40 | +0.50 | 0.90 |
| human.coherence | -0.20 | +0.50 | 0.70 |
| recursion.economy | +0.84 | +0.20 | 0.64 |
| human.verifiability | +0.10 | +0.70 | 0.60 |
| recursion.determinism | +1.00 | +0.50 | 0.50 |
| recursion.verifiability | +0.00 | +0.50 | 0.50 |
| loop.economy | +0.62 | +0.20 | 0.42 |
Ground truth
3 of 4 requirements met · deterministic checker, no model judgment- figure produced — figure.svg
- chart source runs clean — exit 0
- title/axis/legend labels (3+ text) — 0 text nodes
- 2+ series colours — 7 colours
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-06| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 25 | 24/25 | 13 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.8691
- hard-measured cells
- 53%
- teaching source
- matplotlib (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.