graphviz
grade
reliability
overall
Graphviz compiles relational text specifications into image diagrams using a command-line interface. The agent uses this tool by writing structured graph declarations in the DOT syntax to a local file and then running the compiler command-line utility. Although the text representation is highly compatible with pre-trained models, fragile execution steps and strict compiler requirements frequently cause rendering or compilation failures.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
Writing the graph structure in the DOT language works adequately because the syntax rules are deterministic and the syntax is highly represented in pre-trained training data. The agent directly defines nodes, edges, labels, styles, and shapes. The compiler provides line numbers for syntax errors, allowing the agent to locate and correct layout failures without manual visual search.
Coordinating the edit-compile-verify loop works poorly because it requires multiple files and distinct tool executions. The agent must write the source text to a file, execute an external compiling command, and then read the terminal output to dtermine compile state. This multi-step process introduces frequent run-to-run failures when the environment experiences dependency or file-writing problems.
Defining visually grouped subgraphs works poorly due to strict prefix requirements and silent rendering issues. The compiler requires subgraphs to use the 'cluster_' prefix to render a surrounding container. If this prefix is omitted, the compiler silences any warning, resulting in failed rendering structure that the agent cannot detect from the terminal output.
State preservation works well because DOT files are standard plain text. An agent resuming a task is able to read the source file directly, preventing compilation errors from destroying prior development progress. However, final layout quality works poorly because the compiler automatic routing engine frequently creates overlapping pathways that are difficult for the agent to optimize without visual feedback.
Fit
This tool is a good fit for tasks where agents must generate schemas, flowcharts, or structural relationships where exact visual pixel placement is subordinated to the relational data model. It is also suitable for environments that provide automated visual validation checkpoints.
It is a poor fit for autonomous or isolated agents that must meet strict visual presentation criteria without access to real-time image inspection tools, as layout engines frequently produce overlapping elements or silent visual omissions.
Notes
This diagram shows the offline compile-and-inspect loop transition that the agent experiences when updating a diagram.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 5 of 16 cross-checked cells (69%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.verifiability | -0.40 | +0.80 | 1.20 |
| loop.determinism | -0.30 | +0.70 | 1.00 |
| loop.verifiability | -0.20 | +0.80 | 1.00 |
| disclosure.economy | +0.04 | +0.90 | 0.86 |
| recursion.coherence | -0.35 | +0.50 | 0.85 |
| human.coherence | -0.20 | +0.60 | 0.80 |
| interface.verifiability | +0.00 | +0.70 | 0.70 |
| human.verifiability | +0.10 | +0.80 | 0.70 |
| recursion.verifiability | +0.00 | +0.60 | 0.60 |
| recursion.economy | +0.76 | +0.30 | 0.46 |
| loop.economy | +0.83 | +0.40 | 0.43 |
Ground truth
2 of 5 requirements met · deterministic checker, no model judgment- renders without error — render failed
- 6+ labeled nodes — 22 nodes
- 7+ directed edges — 44 edges
- 2+ clusters/groups — 0 clusters
- has a title — absent
Reliability (pass^k): one attempt succeeds 0% of the time · three in a row 0% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 25 | 19/25 | 9 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.5299
- hard-measured cells
- 53%
- teaching source
- graphviz (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.