python-diagrams
grade
reliability
overall
The diagrams Python library generates cloud and system architecture diagrams in PNG or SVG formats from code definitions. The agent declares architecture nodes, clusters, and relationships using custom Python syntax and stream operators. The library relies on an external Graphviz rendering engine to compile the visual layout of these declarations into an image file.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
Defining architecture layouts with standard Python syntax works well. The agent translates conceptual system designs into declarative structures with nodes, edges, and clusters without needing manual layout drawing. This declarative format allows system topologies to be defined in brief, clean code blocks that fit the code-generation strengths of an agent.
Finding the correct class declarations and imports works poorly. Because individual node icons are scattered across highly granular, deeply nested modules—such as diagrams.onprem.database or diagrams.programming.language—the agent must frequently guess imports or perform multi-step documentation searches. This structural scattering causes repetitive ImportError exceptions during runtime execution.
Executing and debugging the development loop works adequately. Standard Python tracebacks provide clear line numbers for syntax or import mistakes, enabling rapid program correction. However, the runtime dependency on a system-level Graphviz installation means that if the dot binary is absent or misconfigured, execution fails with low-level path errors that cannot be fixed by Python alterations alone.
Fine-tuning graph aesthetics and layout structure works poorly. Graphviz dynamically computes the coordinate layout of nodes, meaning minor additions can cause extreme shifts in the final diagram's visual routing. Because the agent cannot natively inspect or perceive visual rendering issues like text overlapping or awkward edge crossings, it is forced to operate without direct confirmation of visual quality.
Fit
This tool is a good fit for developers and automated workflows that need to generate baseline system architecture maps quickly from structured descriptions. It provides a reliable way to map out services, networks, and databases programmatically without a manual user interface.
It is a poor fit for environments that require exact spatial node placement, customized pixel-level visual design, or strict visual consistency. It is also unsuitable for restricted runner environments that cannot easily install and configure system-level Graphviz binaries.
Notes
The following workflow diagram shows the edit-run-fix loop when an agent generates diagrams with this library.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 9 of 16 cross-checked cells (74%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.verifiability | -0.40 | +0.80 | 1.20 |
| recursion.coherence | -0.64 | +0.50 | 1.14 |
| interface.coherence | -0.28 | +0.80 | 1.08 |
| human.verifiability | +0.10 | +0.90 | 0.80 |
| human.coherence | -0.20 | +0.60 | 0.80 |
| loop.determinism | +0.10 | +0.80 | 0.70 |
| interface.prior_alignment | +0.40 | +0.90 | 0.50 |
Ground truth
5 of 5 requirements met · deterministic checker, no model judgment- renders without error — exit 0
- 6+ labeled nodes — 11 nodes
- 7+ directed edges — 18 edges
- 2+ clusters/groups — 3 clusters
- has a title — present
Reliability (pass^k): one attempt succeeds 80% of the time · three in a row 40% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 17/23 | 8 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.7726
- hard-measured cells
- 53%
- teaching source
- python-diagrams (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.