plotly
grade
reliability
overall
Plotly is a Python graphing library that enables an AI agent to programmatically generate interactive web charts and static vector graphics. The agent uses the tool by writing scripts that parse CSV files, configure visualization traces, and export output files. While the resulting charts render cleanly without visual corruption, the visualization workflow is poorly reliable due to frequent failures across repeated execution runs.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
The agent constructs charts by writing standard Python code that leverages the 'plotly.graph_objects' API, which works well due to its integration with standard data libraries like 'pandas'. For example, the agent can parse a custom format with commented lines and dynamically construct coordinates using loops. Defining scatter plots, markers, and sizing attributes programmatically allows the agent to control multi-dimensional bubble charts cleanly.
For file saving and distribution, the tool works adequately. It provides a robust HTML export function that does not require additional system-level packages, running reliably in minimal environments. If static formats like SVG or PNG are needed, the library relies on the external 'kaleido' engine. When 'kaleido' is absent, the agent must catch the exception and fall back to HTML generation.
However, the tool works poorly regarding consistency and reliability across multiple execution attempts, as run-to-run behavior is unstable and often leads to runtime errors or missing elements. Specifically, the library failed to satisfy the requirement of producing multiple distinct series colors, even when the agent explicitly provided a custom Okabe-Ito colorway palette. The complexity of adjusting the internal plot state and legend traces programmatically often leads to silent visual discrepancies.
The tool works adequately when a task must be handed off to a subsequent agent. A fresh agent resuming the task from the generated scripts and output files alone can modify the script without breaking preexisting work. The declarative nature of the saved visualization code prevents state regression or compounding format errors.
Fit
Plotly is a good fit for agents operating as automated data analysts or report generators that deliver interactive web dashboards to human users. The programmatic trace-add structure matches typical script-writing workflows, allowing fast iterations on plot elements.
It is a poor fit for highly constrained runtimes that cannot install binary dependencies like 'kaleido' yet require static image exports. It is also a poor fit for production systems requiring strict determinism in automated layout validation, as complex multi-trace styling often behaves inconsistently.
Notes
The diagram illustrates the agent's decision loop for building and exporting the visualization file depending on library dependencies.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 7 of 16 cross-checked cells (73%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| recursion.coherence | -0.62 | +0.50 | 1.12 |
| disclosure.verifiability | -0.40 | +0.60 | 1.00 |
| human.verifiability | -0.20 | +0.80 | 1.00 |
| interface.coherence | -0.24 | +0.70 | 0.94 |
| human.coherence | -0.20 | +0.60 | 0.80 |
| recursion.economy | +0.70 | +0.10 | 0.60 |
| human.safety | -0.10 | +0.50 | 0.60 |
| recursion.verifiability | +0.20 | +0.70 | 0.50 |
| disclosure.economy | +0.19 | +0.60 | 0.41 |
Ground truth
3 of 4 requirements met · deterministic checker, no model judgment- figure produced — figure.svg
- chart source runs clean — exit 0
- title/axis/legend labels (3+ text) — 21 text nodes
- 2+ series colours — 0 colours
Reliability (pass^k): one attempt succeeds 80% of the time · three in a row 40% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 22 | 17/22 | 7 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.786
- hard-measured cells
- 53%
- teaching source
- plotly (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.