mermaid
grade
reliability
overall
Generation of diagrams like flowcharts or sequence diagrams from text in a similar manner as markdown
The Mermaid tool allows an agent to design structured system diagrams using a declarative text format and compile them into static image assets. The agent specifies diagram components, groupings, and styled directional paths within a plain text file, which is then compiled via a command-line interface. This setup provides an automated loop where syntax and structural layout can be validated programmatically before final output creation.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
Teaching sources compared
Teaching source barely matters for mermaid: sources land within a grade of each other (skill A, docs A).
The tool was evaluated using a specialized skill guide and general reference documentation. Both teaching sources served the agent adequately and yielded nearly identical rendering results. This uniform performance is due to the tool's reliance on standard declarative syntax, which is well-supported by both the structured reference files and the agent's pre-existing patterns.
| Source | Grade | Renders | Disclosure | Interface | Loop | Recursion | Human |
|---|---|---|---|---|---|---|---|
| skill ★ | A | ✓ | +0.31 | +0.64 | +0.55 | +0.38 | +0.31 |
| official docs | A | ✓ | +0.23 | +0.62 | +0.63 | +0.36 | +0.35 |
Same harness, same reference agent, same briefs, same trial count — the only variable is how the agent was taught the tool. ★ = best source.
The experience
Developing visual models with the command-line compiler works well. The agent uses precise style declarations to define custom stroke colors, widths, and dash arrays, allowing clear visual paths to differentiate primary logic from secondary relationships. Subgraphs successfully cluster related nodes, allowing the automated layout engine to organize the final image cleanly without the need for absolute coordinate positioning.
State management across turns is reliable. Because the output source file is plain text, a fresh agent can read the existing diagram schema and make subsequent updates without corrupting existing layout rules or breaking the style formatting. This statelessness reduces the risk of regression in complex diagrams during multi-step revisions.
The compilation loop operates adequately. Running the command-line compiler on the generated text file provides immediate validation of the diagram syntax. Potential layout errors are caught and reported by the compiler, allowing the agent to correct structural issues immediately before committing the final rendered files.
Fit
The tool is a good fit for agents that need to programmatically document software architectures, database schemas, or step-by-step processes in automated environments. It works well when the priority is logical flow, structure, and declarative verification rather than custom design control.
It is a poor fit for tasks that demand precise spatial layouts or pixel-perfect styling constraints. Because the compiler engine dynamically positions nodes, the agent has no direct way to enforce specific physical coordinates or custom pixel distances between elements.
Notes
This diagram shows the iterative compilation and validation flow used when working with the tool.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 10 of 16 cross-checked cells (83%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| human.verifiability | +0.10 | +0.90 | 0.80 |
| recursion.coherence | -0.32 | +0.40 | 0.72 |
| recursion.economy | +0.89 | +0.20 | 0.69 |
| human.safety | -0.10 | +0.50 | 0.60 |
| recursion.determinism | +1.00 | +0.50 | 0.50 |
| recursion.verifiability | +0.00 | +0.50 | 0.50 |
Ground truth
5 of 5 requirements met · deterministic checker, no model judgment- renders without error — exit 0
- 6+ labeled nodes — 19 nodes
- 7+ directed edges — 31 edges
- 2+ clusters/groups — 2 clusters
- has a title — present
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-06| Model | Harness | Runs | Solved | Turns | Knows it |
|---|---|---|---|---|---|
 glm-5.2 |  PI | 25 | 25/25 | 14 | 0% |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn).
Hover a model — here or in the graph — to isolate its runs.
“Knows it” = a closed-book quiz (20 questions, no docs) on whether the model already knows this tool from pre-training — 62% across the panel.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $1.198
- hard-measured cells
- 53%
- teaching source
- design-doc-mermaid (skills.sh)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.