lilypond
grade
reliability
overall
LilyPond is a command-line utility that compiles plain-text musical notation files into vector and raster graphic formats. Handing this tool to an agent establishes a highly deterministic development cycle based on local source code compilation. The agent drafts structured notation markup, invokes the compiler via command-line arguments, and processes standard exit codes or generated files.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
The agent interacts with the tool by writing music notation in a structured markup format. This drafting phase works well because the standard LilyPond syntax matches the agent's pre-trained capabilities, allowing it to define multi-voice staves, tempo markings, and dynamics. The structured format allows the agent to construct the document layout incrementally before invoking the compiler.
The command-line compilation interface works well due to its predictable execution behavior. The command-line commands accept simple parameters to generate specific output types, and the tool returns clear success or failure indicators through standard exit codes. Each compilation run produces identical visual output from the same source file, enabling reliable version control.
Error handling works adequately because the compiler differentiates between fatal translation errors and non-fatal warnings. However, when syntax errors occur, the agent must parse raw log output to pinpoint the failing line and token, which can interrupt the development flow. Correcting these errors requires the agent to perform multiple search-and-replace iterations on the source file.
Visual spatial verification works poorly. Although the compiler produces clean output files, it lacks spatial collision detection and will return a successful exit code even if text labels overlap or graphic bars extend directly through the musical staff. Because the agent cannot natively evaluate these layout issues without visual analysis tools, it may produce technically valid files that are visually illegible.
Fit
This tool is a good fit for applications that require structured, programmatic sheet music generation where automatic layout formatting is acceptable. It suits automated execution environments where agents must generate standard music documents from database input query outputs or plain text sources.
The tool is a poor fit for projects that demand precise custom formatting or graphic design quality. Because the compiler does not raise errors for visual overlap or collision of elements, an agent cannot guarantee final aesthetic quality. It is also unsuitable for tasks requiring immediate audio verification, since the silent MIDI output cannot be directly verified within standard environments.
Notes
This diagram outlines the iterative write-compile-check loop of the agent when compiling LilyPond files.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 10 of 16 cross-checked cells (81%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.verifiability | -0.40 | +0.50 | 0.90 |
| recursion.economy | +0.73 | +0.00 | 0.73 |
| recursion.coherence | -0.25 | +0.30 | 0.55 |
| recursion.determinism | +1.00 | +0.50 | 0.50 |
| human.verifiability | +0.10 | +0.60 | 0.50 |
| loop.verifiability | +0.94 | +0.50 | 0.44 |
Ground truth
7 of 7 requirements met · deterministic checker, no model judgment- notation source produced — water_cycle.ly
- engraves without error — ok
- has a title — —
- 2+ staves/voices — 3 voices
- tempo or meter set — —
- dynamics markings — —
- 8+ measures — 34 bar marks
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 20/23 | 6 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $1.0029
- hard-measured cells
- 53%
- teaching source
- lilypond (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.