pandoc
grade
reliability
overall
Pandoc is a command-line utility that compiles plaintext markup and external metadata into structured document formats such as PDF, HTML, and Docx. For an AI agent, the tool provides a highly deterministic interface that converts structured content including libraries, tables, and figures into standardized layouts. The compiler returns reliable exit codes and precise syntax warnings, enabling predictable execution of document generation tasks.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
Teaching sources compared
docs (A) serves the agent better than skill (B) for pandoc. The gap is widest on Loop (0.28).
| Source | Grade | Renders | Disclosure | Interface | Loop | Recursion | Human |
|---|---|---|---|---|---|---|---|
| official docs ★ | A | ✓ | +0.22 | +0.59 | +0.48 | +0.37 | +0.29 |
| skill | B | ✓ | +0.32 | +0.58 | +0.20 | +0.21 | +0.07 |
Same harness, same reference agent, same briefs, same trial count — the only variable is how the agent was taught the tool. ★ = best source.
The experience
The agent constructs documents by writing Markdown files containing YAML metadata block headers, standard Markdown tables, and citation keys. It then compiles the document into a PDF in a single step using command-line arguments like --citeproc, --bibliography, and --pdf-engine=tectonic. This configuration works exceptionally well because Pandoc automatically coordinates complex formatting tasks such as bibliography pagination, section numbering, and table rendering without low-level TeX programming.
The command-line interface operates with high determinism. Successful compilation processes return an exit code of zero, which the agent uses to verify syntax integration. Because the official documentation provides clear examples matching standard markdown patterns, the agent successfully configures and executes the compiler on its first attempt, avoiding repetitive trial-and-error loops.
However, document verification works poorly. Because Pandoc outputs binary files like PDFs, the agent cannot programmatically inspect the visual rendering, page breaks, or text alignment. Deficiencies in the output layout require external multimodal tools or human review to be detected, leading to a weak verification workflow.
Additionally, multi-file asset coordination works inadequately due to fragmented tool dependencies. Since Pandoc cannot generate images or charts internally, the agent must write and execute separate Python scripts to generate visual figures prior to compilation. If an external figure script fails or produces a flawed file, the compiler still completes successfully with exit code zero but produces a visually incomplete document.
Fit
Pandoc is a highly suitable fit for agents that need to programmatically compile plain text data and structured headers into standard document formats like PDF, HTML, or Docx. It is ideal for standardized reports, academic papers, and briefs that rely on consistent structural layouts, automated bibliographies, and table-of-contents generation.
It is a poor fit for tasks that require precise visual positioning, manual typographic design, or real-time layout adjustments, as the agent has no built-in mechanism to preview coordinates or styling. It is also unsuitable for complex media pipelines that demand unified dependency tracking, as the agent must orchestrate visual asset generation entirely outside of the compiler.
Notes
The diagram shows how the agent coordinates asset generation and metadata compilation inside the document writing loop.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 9 of 16 cross-checked cells (78%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.verifiability | -0.40 | +0.60 | 1.00 |
| human.verifiability | -0.20 | +0.80 | 1.00 |
| recursion.verifiability | -0.08 | +0.70 | 0.78 |
| recursion.coherence | -0.21 | +0.50 | 0.71 |
| human.safety | -0.10 | +0.50 | 0.60 |
| loop.economy | -0.03 | +0.50 | 0.53 |
| recursion.economy | +0.90 | +0.50 | 0.40 |
Ground truth
5 of 5 requirements met · deterministic checker, no model judgment- compiles to final format — ok
- 3+ sections — 6 sections
- a data table — yes
- a generated figure — yes
- a bibliography citation — yes
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
| | 23 | 21/23 | 9 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $1.2584
- hard-measured cells
- 53%
- teaching source
- pandoc (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.