Back to explorer
markup Declarative files best source: official docs

org

A

grade

100%

reliability

+0.38

overall

The Org toolchain consists of authoring structured documents in Org-mode text files and compiling them to HTML or PDF targets using Pandoc. An agent receives a plain-text template requiring specific markup, including header nesting with asterisks, metadata headers, double-bracketed hyperlinks, and pipe-delimited data tables. The final output is generated via shell commands that invoke Pandoc alongside LaTeX engines like Tectonic.

Surfaces × lenses

hostile friendly
surface ↓
lens →
Coherence Economy Consistency Verifiability Safety
Disclosure
Interface
Loop
Recursion
Human

Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.

The experience

The agent operates in an iterative write-compile-verify loop. It first writes document content using specific Org metadata headers and block identifiers like #+BEGIN_SRC. It then runs Pandoc compilation commands through a bash interface to generate the target document. Because Pandoc conversion is deterministic and locally contained, the compilation step executes reliably across multiple attempts with no risk of destructive side effects.

Challenges occur when parsing compilation logs from the underlying rendering engine. When rendering to PDF using Tectonic, the terminal output often includes verbose messages, such as cosmetic table-width notices and font warnings. The agent must parse these notifications to distinguish benign aesthetic warnings from fatal compilation failures.

Syntactically, the structured block-style format of Org-mode maps directly to structured data formats. The pipe-and-hyphen table borders allow the agent to format complex data grids that align predictably. However, the unique double-bracket link notation and asterisk heading hierarchy differ slightly from Markdown, requiring the agent to strictly adhere to the provided documentation to avoid structural rendering errors.

Fit

This toolchain is effective for agents tasked with generating highly structured documents, tables, and reports where output formatting must be deterministic. The predictable execution path of local Pandoc compilation makes it easy for an agent to automate, verify, and correct layout issues without risk of state corruption.

It is less suitable for simple document tasks with short execution time limits. Because Org syntax contains specific block delimiters and hyperlink rules that differ from standard Markdown, the compilation process introduces an additional verification layer and warning logs that the agent must analyze.

Notes

This diagram illustrates the workflow loop of authoring Org-mode document source, running local compiling processes, and sorting through log warnings to ensure a clean render.

Validation

Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 8 of 16 cross-checked cells (75%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.

Cell where they disagreeInstrumentRaterΔ
disclosure.economy-0.32+0.901.22
disclosure.verifiability-0.40+0.701.10
recursion.coherence-0.49+0.500.99
recursion.verifiability+0.00+0.700.70
human.verifiability+0.10+0.800.70
human.safety-0.10+0.500.60
interface.coherence+0.02+0.500.48
loop.economy+0.75+0.300.45

Ground truth

7 of 7 requirements met · deterministic checker, no model judgment
  • renders/compiles cleanly — ok
  • has a title — —
  • nested sections (2+ levels) — —
  • a table — yes
  • a code block — —
  • a link — —
  • an image/figure — yes

Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.

Models & runs

2026-07-07
ModelHarnessRunsSolvedTurns
glm-5.2 PI2320/236

Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.

How this was measured

reference agent
pi:z-ai/glm-5.2
trials
25 (n=5/tier)
methodology
v0.1 3e74a3c6
cost
$0.7877
hard-measured cells
53%
teaching source
org (official docs) (official-docs)

Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.