org
grade
reliability
overall
The Org toolchain consists of authoring structured documents in Org-mode text files and compiling them to HTML or PDF targets using Pandoc. An agent receives a plain-text template requiring specific markup, including header nesting with asterisks, metadata headers, double-bracketed hyperlinks, and pipe-delimited data tables. The final output is generated via shell commands that invoke Pandoc alongside LaTeX engines like Tectonic.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
The agent operates in an iterative write-compile-verify loop. It first writes document content using specific Org metadata headers and block identifiers like #+BEGIN_SRC. It then runs Pandoc compilation commands through a bash interface to generate the target document. Because Pandoc conversion is deterministic and locally contained, the compilation step executes reliably across multiple attempts with no risk of destructive side effects.
Challenges occur when parsing compilation logs from the underlying rendering engine. When rendering to PDF using Tectonic, the terminal output often includes verbose messages, such as cosmetic table-width notices and font warnings. The agent must parse these notifications to distinguish benign aesthetic warnings from fatal compilation failures.
Syntactically, the structured block-style format of Org-mode maps directly to structured data formats. The pipe-and-hyphen table borders allow the agent to format complex data grids that align predictably. However, the unique double-bracket link notation and asterisk heading hierarchy differ slightly from Markdown, requiring the agent to strictly adhere to the provided documentation to avoid structural rendering errors.
Fit
This toolchain is effective for agents tasked with generating highly structured documents, tables, and reports where output formatting must be deterministic. The predictable execution path of local Pandoc compilation makes it easy for an agent to automate, verify, and correct layout issues without risk of state corruption.
It is less suitable for simple document tasks with short execution time limits. Because Org syntax contains specific block delimiters and hyperlink rules that differ from standard Markdown, the compilation process introduces an additional verification layer and warning logs that the agent must analyze.
Notes
This diagram illustrates the workflow loop of authoring Org-mode document source, running local compiling processes, and sorting through log warnings to ensure a clean render.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 8 of 16 cross-checked cells (75%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.economy | -0.32 | +0.90 | 1.22 |
| disclosure.verifiability | -0.40 | +0.70 | 1.10 |
| recursion.coherence | -0.49 | +0.50 | 0.99 |
| recursion.verifiability | +0.00 | +0.70 | 0.70 |
| human.verifiability | +0.10 | +0.80 | 0.70 |
| human.safety | -0.10 | +0.50 | 0.60 |
| interface.coherence | +0.02 | +0.50 | 0.48 |
| loop.economy | +0.75 | +0.30 | 0.45 |
Ground truth
7 of 7 requirements met · deterministic checker, no model judgment- renders/compiles cleanly — ok
- has a title — —
- nested sections (2+ levels) — —
- a table — yes
- a code block — —
- a link — —
- an image/figure — yes
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
| | 23 | 20/23 | 6 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.7877
- hard-measured cells
- 53%
- teaching source
- org (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.