groff
grade
reliability
overall
Groff is a command-line typesetting utility that compiles plain-text markup containing ms macros into documents. For an agent, it serves as a lightweight and highly deterministic document generator that runs via simple terminal execution flags. The utility produces consistent and repeatable layouts without requiring graphical desktop environments or web browser runtime dependencies.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
An agent interacts with groff by writing plain-text files containing structural macros and executing compilation commands. The compilation stage works well because compilation commands using standard preprocessors are highly predictable, execution has a low blast radius, and the compiler returns reliable exit codes. This allows the agent to verify compilation success deterministically and guarantees identical visual output across multiple runs.
Writing metadata and basic layout sections works adequately. Structural macros provide direct indicators for titles, body paragraphs, and numbered headings, which are straightforward for an agent to generate. However, the ms macro set is obscure to modern models, requiring strict adherence to exact spacing, margins, and uppercase commands, which can complicate block alignment.
Constructing structured tables works poorly and requires multiple refinement iterations. The tbl preprocessor requires a rigid syntax for declaring column alignment, borders, and separators that must map to the input data layout perfectly. Syntactic errors and visual column misalignments are difficult for the agent to debug from compiler output alone, causing the agent to take multiple tool turns to write correct tables.
Fit
Groff is an excellent fit for headless agent workflows that need to produce highly standardized, plain documents such as invoices or brief reports without graphical overhead. It is a poor fit for applications requiring modern, complex designs or when the agent must parse dynamic upstream data without intermediate transformation scripts.
Notes
This diagram illustrates the agent loop when compiling a groff document from raw input data.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 6 of 16 cross-checked cells (74%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| human.verifiability | -0.20 | +0.80 | 1.00 |
| recursion.coherence | -0.43 | +0.50 | 0.94 |
| disclosure.verifiability | -0.40 | +0.50 | 0.90 |
| recursion.economy | +0.95 | +0.20 | 0.75 |
| recursion.verifiability | +0.00 | +0.60 | 0.60 |
| human.safety | -0.10 | +0.50 | 0.60 |
| loop.economy | +0.89 | +0.30 | 0.59 |
| interface.coherence | +0.13 | +0.70 | 0.57 |
| disclosure.economy | -0.26 | +0.30 | 0.56 |
| human.coherence | -0.20 | +0.30 | 0.50 |
Ground truth
4 of 4 requirements met · deterministic checker, no model judgment- renders to final format (pdf/html) — ok
- document identifier present — INV-2024-0417
- all line items present — 4/4 items
- correct computed total — expect subtotal 1673.00
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 23/23 | 8 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.7923
- hard-measured cells
- 53%
- teaching source
- groff (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.