Back to explorer
documents Declarative files best source: official docs

pandoc

measured sources: official docs Askill B — added context lifts it A → B
A

grade

100%

reliability

+0.48

overall

Pandoc is a command-line utility that compiles plaintext markup and external metadata into structured document formats such as PDF, HTML, and Docx. For an AI agent, the tool provides a highly deterministic interface that converts structured content including libraries, tables, and figures into standardized layouts. The compiler returns reliable exit codes and precise syntax warnings, enabling predictable execution of document generation tasks.

Surfaces × lenses

hostile friendly
surface ↓
lens →
Coherence Economy Consistency Verifiability Safety
Disclosure
Interface
Loop
Recursion
Human

Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.

Teaching sources compared

docs (A) serves the agent better than skill (B) for pandoc. The gap is widest on Loop (0.28).

SourceGradeRendersDisclosureInterfaceLoopRecursionHuman
official docs ★A+0.22+0.59+0.48+0.37+0.29
skillB+0.32+0.58+0.20+0.21+0.07

Same harness, same reference agent, same briefs, same trial count — the only variable is how the agent was taught the tool. ★ = best source.

The experience

The agent constructs documents by writing Markdown files containing YAML metadata block headers, standard Markdown tables, and citation keys. It then compiles the document into a PDF in a single step using command-line arguments like --citeproc, --bibliography, and --pdf-engine=tectonic. This configuration works exceptionally well because Pandoc automatically coordinates complex formatting tasks such as bibliography pagination, section numbering, and table rendering without low-level TeX programming.

The command-line interface operates with high determinism. Successful compilation processes return an exit code of zero, which the agent uses to verify syntax integration. Because the official documentation provides clear examples matching standard markdown patterns, the agent successfully configures and executes the compiler on its first attempt, avoiding repetitive trial-and-error loops.

However, document verification works poorly. Because Pandoc outputs binary files like PDFs, the agent cannot programmatically inspect the visual rendering, page breaks, or text alignment. Deficiencies in the output layout require external multimodal tools or human review to be detected, leading to a weak verification workflow.

Additionally, multi-file asset coordination works inadequately due to fragmented tool dependencies. Since Pandoc cannot generate images or charts internally, the agent must write and execute separate Python scripts to generate visual figures prior to compilation. If an external figure script fails or produces a flawed file, the compiler still completes successfully with exit code zero but produces a visually incomplete document.

Fit

Pandoc is a highly suitable fit for agents that need to programmatically compile plain text data and structured headers into standard document formats like PDF, HTML, or Docx. It is ideal for standardized reports, academic papers, and briefs that rely on consistent structural layouts, automated bibliographies, and table-of-contents generation.

It is a poor fit for tasks that require precise visual positioning, manual typographic design, or real-time layout adjustments, as the agent has no built-in mechanism to preview coordinates or styling. It is also unsuitable for complex media pipelines that demand unified dependency tracking, as the agent must orchestrate visual asset generation entirely outside of the compiler.

Notes

The diagram shows how the agent coordinates asset generation and metadata compilation inside the document writing loop.

Validation

Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 9 of 16 cross-checked cells (78%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.

Cell where they disagreeInstrumentRaterΔ
disclosure.verifiability-0.40+0.601.00
human.verifiability-0.20+0.801.00
recursion.verifiability-0.08+0.700.78
recursion.coherence-0.21+0.500.71
human.safety-0.10+0.500.60
loop.economy-0.03+0.500.53
recursion.economy+0.90+0.500.40

Ground truth

5 of 5 requirements met · deterministic checker, no model judgment
  • compiles to final format — ok
  • 3+ sections — 6 sections
  • a data table — yes
  • a generated figure — yes
  • a bibliography citation — yes

Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.

Models & runs

2026-07-07
ModelHarnessRunsSolvedTurns
glm-5.2 PI2321/239

Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.

How this was measured

reference agent
pi:z-ai/glm-5.2
trials
25 (n=5/tier)
methodology
v0.1 3e74a3c6
cost
$1.2584
hard-measured cells
53%
teaching source
pandoc (official docs) (official-docs)

Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.