plantuml
grade
reliability
overall
PlantUML is a command-line application that compiles structured text declarations into visual diagram files. The agent interacts with the tool by writing diagram specifications in a declarative language and calling a terminal command to generate graphical formats such as SVG or PNG. This compilation workflow runs locally and deterministically, producing consistent output images from identical input scripts.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
Teaching sources compared
The skill source produces a rendering artifact; docs does not — the teaching source is decisive for plantuml.
The tool succeeded under the skill teaching source but failed under the reference documentation. The skill teaching source was the decisive factor, enabling the agent to compile a clean, visually verified SVG diagram. In contrast, the reference documentation only covered generating plain-text ASCII art, which did not prepare the agent for compiling visual image files.
As a result, learning via the reference documentation left the agent unable to meet the graphic rendering requirements directly. The skill source provided a reliable guide for the rendering process, whereas the documentation forced the agent to bypass the provided instructions and rely on prior training.
| Source | Grade | Renders | Disclosure | Interface | Loop | Recursion | Human |
|---|---|---|---|---|---|---|---|
| skill ★ | A | ✓ | +0.35 | +0.62 | +0.58 | +0.38 | +0.48 |
| official docs | F | ✗ | +0.30 | +0.31 | +0.28 | +0.45 | +0.41 |
Same harness, same reference agent, same briefs, same trial count — the only variable is how the agent was taught the tool. ★ = best source.
The experience
The agent initiates diagram generation by writing a text file containing standard UML declarations, including nodes, clusters, and labeled relationship lines. It then calls the command-line tool with flags specifying the preferred target format, such as vector SVG or raster PNG. This process is compact and efficient, enabling the creation of complex spatial layouts from minimal text descriptions without manual coordinate calculations.
A major friction point occurs when the host environment lacks Java in its default path. Because PlantUML requires a Java runtime execution environment, the agent must check the system paths, locate a compatible installed JDK, and set the appropriate environment variable before compiling. Once the binary dependencies are resolved, the terminal command executes predictably, rendering artifacts with reliable zero exit codes.
The agent faced difficulty due to the provided documentation, which focused muddying the objective by focusing exclusively on exporting plain-text ASCII and Unicode art. Since the practical task required rendering vector images, the agent was forced to work around the documentation and locate instructions internally to compile standard pixel and vector graphics.
Fit
This tool is a strong fit for agents designed to document software architectures, database schemas, or step-by-step logic workflows. Because the layout engine resides entirely within the compiler, the agent only needs to define relationships and system logic in clean text blocks rather than managing placement coordinates or rendering style.
It is a poor fit for agents operating in lightweight runtimes that lack a Java Virtual Machine or restrict environmental path changes. It is also poorly suited for projects requiring highly specific aesthetic designs, non-standard node shapes, or precise pixel-level layout placements, as the utility manages all spacing and layouts using its own engine.
Notes
The following diagram illustrates the agent loop for writing markup and managing path variables to compile a PlantUML graphic.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 11 of 16 cross-checked cells (82%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| human.verifiability | -0.20 | +0.90 | 1.10 |
| recursion.verifiability | -0.32 | +0.50 | 0.82 |
| recursion.economy | +0.98 | +0.20 | 0.78 |
| recursion.coherence | -0.11 | +0.50 | 0.61 |
| recursion.determinism | +1.00 | +0.50 | 0.50 |
Ground truth
5 of 5 requirements met · deterministic checker, no model judgment- renders without error — exit 0
- 6+ labeled nodes — 15 nodes
- 7+ directed edges — 20 edges
- 2+ clusters/groups — 2 clusters
- has a title — present
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-06| Model | Harness | Runs | Solved | Turns | Knows it |
|---|---|---|---|---|---|
 glm-5.2 |  PI | 25 | 22/25 | 13 | 60% |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn).
Hover a model — here or in the graph — to isolate its runs.
“Knows it” = a closed-book quiz (20 questions, no docs) on whether the model already knows this tool from pre-training — 59% across the panel.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.8987
- hard-measured cells
- 53%
- teaching source
- plantuml-ascii (skills.sh)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.