plotnine
grade
reliability
overall
The plotnine library provides a Python implementation of the ggplot2 grammar of graphics, allowing agents to declare visualizations by joining layers, scales, and themes. While the codebase syntax is highly compatible with the agent's training, the tool is highly prone to producing outputs that do not match the agent's internal expectations. This results in a silent failure loop where the agent believes it has generated a highly detailed, annotated plot when the actual visual output lacks primary labels, titles, and legends.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
Data parsing and initial plotting setup work poorly because of environment and input sensitivities. The agent required ten turns to parse a non-standard CSV file containing leading lines and column labels. Implementing the plotting layout proceeded adequately in terms of code syntax since the agent wrote correct imports and methods like ggplot, aes, geom_line, annotate, and theme_minimal.
Execution stability and state persistence work poorly. The library is unreliable run-to-run, leading to unexpected script failures on repeated attempts without code changes. Furthermore, a fresh agent resuming from saved code files often regresses on prior progress, indicating that the tool's runtime behavior creates significant friction for multi-turn workflows.
Visual consistency between the agent's intent and the rendered artifact works poorly. The agent's generated code claimed to output custom colorblind-safe legends, log scales, axis headers, and a main title, yet the actual rendered file lacked title text, axis names, and legends. Plotnine fails to notify the execution environment of missing metadata or visual clipping, meaning errors in visual rendering are silent and impossible for the agent to detect without image verification tools.
Layout management works poorly when placing annotations manually. Because the library places annotations using coordinate-based values, elements like text labels can be clipped or pushed outside the rendering frame. The agent's global maximum annotation marker was partially cut off at the right edge of the chart without triggering any warning, leading to a flawed final output.
Fit
From an agent's perspective, plotnine is a poor fit for autonomous, closed-loop environments. The tool's runtime instability, silent rendering failures, and lack of visual feedback prevent agents from correcting layout errors or confirming text additions without manual human oversight.
Conversely, the library is an adequate fit for interactive setups where a human developer can inspect the generated files and iterate on the layout. It is also suitable for agents paired with visual feedback tools that can read rendered files and provide descriptive error messages about clipped text or missing legends.
Notes
This diagram illustrates the agent's write-compile-render loop and the visual discrepancies introduced by a lack of rendering feedback.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 8 of 16 cross-checked cells (73%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| recursion.economy | +0.82 | -0.40 | 1.22 |
| disclosure.verifiability | -0.40 | +0.60 | 1.00 |
| recursion.determinism | +0.97 | +0.00 | 0.97 |
| loop.economy | +0.43 | -0.50 | 0.93 |
| interface.coherence | -0.38 | +0.50 | 0.88 |
| disclosure.economy | -0.31 | +0.50 | 0.81 |
| human.verifiability | -0.20 | +0.50 | 0.70 |
| recursion.coherence | -0.69 | +0.00 | 0.69 |
Ground truth
3 of 4 requirements met · deterministic checker, no model judgment- figure produced — figure.svg
- chart source runs clean — exit 0
- title/axis/legend labels (3+ text) — 0 text nodes
- 2+ series colours — 9 colours
Reliability (pass^k): one attempt succeeds 80% of the time · three in a row 40% · all runs 0%. A fresh agent resuming from the files alone sometimes regressed prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 21/23 | 7 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.744
- hard-measured cells
- 53%
- teaching source
- plotnine (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.