blockdiag
grade
reliability
overall
The blockdiag utility is a Python-based command-line tool that generates diagram images from declarative text files using a specialized domain-specific language. Handing this tool to an agent allows it to build structured architectural diagrams using clear syntax for nodes, edges, and nested groups. However, the tool is prone to silent compilation failures where the agent incorrectly reports successful generation despite rendering errors.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
The initial layout drafting and syntax construction phase works well. The blockdiag domain-specific language is logical and easy for the agent to parse from the documentation. The agent easily defines node attributes, applies distinct shapes like databases or rounded boxes, and assigns colors or dashed styles to specify different relationship types.
Grouping and boundary-crossing definitions work adequately. The agent organizes related components into separate logical groups and successfully routes directed edges across group boundaries. The layout structure matches the input system architecture accurately.
However, compilation validation works poorly. The agent falsely claims that the file compiled cleanly with an exit status of zero, despite the rendering process failing or being incomplete. This occurs because the tool does not automatically force the agent to inspect the visual correctness of the output, allowing it to conclude the task with an unrendered artifact.
The error correction loop works poorly. Because there is no tight verification loop for the generated image, the agent spends a large number of turns attempting to refine the file but fails to resolve the underlying execution errors. This results in moderate monetary cost and a high number of redundant iterations without producing a valid image.
Fit
This tool is a good fit for agents tasked with generating structured, static diagrams from well-defined textual specifications, provided they operate under a hard validation wrapper. It suits workflows where text-to-diagram compilation can be deterministically verified by an external validator that checks the existence and size of the compiled file.
This tool is a poor fit for autonomous agents running in unmonitored loops without automated file-level verification. Without a reliable feedback mechanism to double-check output generation, the agent will confidently report successful execution while leaving behind broken or unrendered diagrams.
Notes
This diagram illustrates the agent's write-compile-verify loop where the lack of automated image inspection leads to a false success state.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 8 of 16 cross-checked cells (74%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| recursion.determinism | +0.99 | -0.20 | 1.19 |
| loop.determinism | +0.70 | -0.20 | 0.90 |
| disclosure.verifiability | -0.40 | +0.50 | 0.90 |
| recursion.economy | +0.95 | +0.10 | 0.85 |
| human.verifiability | -0.20 | +0.50 | 0.70 |
| human.coherence | -0.20 | +0.50 | 0.70 |
| interface.verifiability | -0.20 | +0.40 | 0.60 |
| recursion.coherence | -0.25 | +0.20 | 0.45 |
Ground truth
4 of 5 requirements met · deterministic checker, no model judgment- renders without error — render failed
- 6+ labeled nodes — 10 nodes
- 7+ directed edges — 13 edges
- 2+ clusters/groups — 3 clusters
- has a title — present
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 19/23 | 10 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $1.1856
- hard-measured cells
- 53%
- teaching source
- blockdiag (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.