markdown
grade
reliability
overall
The markdown tool is a document compilation system that uses Pandoc, Tectonic, and BibTeX to build text documents, data tables, and dynamic visual assets into PDF and HTML outputs. It interacts with the agent via command-line utilities and standard file operations in the workspace. The agent is responsible for writing the manuscript source, generating data plots via auxiliary scripts, and running shell commands to compile the final documents.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
Basic typographical layout and tabular structures work well because standard Pandoc syntax compiles predictably with high reliability. When formatting headers, basic text styles, and markdown grids, the system processes them without rendering errors on the first attempt, allowing the agent to establish standard layouts immediately. Similarly, task resumption and state recovery work well because the workflow is entirely file-based. If an execution loop is interrupted, a fresh agent can read the existing source files and resume compilation without breaking prior work or requiring complex database rollbacks.
Multi-step asset coordination works adequately but introduces operational overhead. To insert a data chart, the agent must write an auxiliary Python script, execute it to save a visual image, reference that image file in the Markdown source, and compile the entire package. While this multi-step loop completes successfully, it consumes a large amount of tokens and tool executions to coordinate code output with document compilation.
Bibliography management and academic citation validation work poorly because the compilation engine fails to provide structural feedback when references do not match. Although the system supports BibTeX databases via the citeproc filter, the agent often produces reports with missing or unrendered bibliography citations. Because the compiler does not output clear warnings about unresolved reference keys in a format the agent can parse, formatting discrepancies are difficult to diagnose and correct autonomously.
Fit
This tool is a good fit for agents producing standard corporate briefs, software manuals, and visual data summaries that require dual-format HTML and PDF outputs. Its standard Markdown syntax matches model training data, resulting in highly predictable document structures.
It is a poor fit for resource-constrained agents because compiling visual reports with dynamic assets is expensive, requiring a large volume of tokens to manage multi-file dependencies. It is also unsuitable for complex layout designs requiring precise academic bibliography matching, as the compiler does not offer enough structured syntax feedback for autonomous adjustment.
Notes
This diagram shows the iterative sequence for generating supporting charts, writing the source report, and compiling the final documents.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 9 of 16 cross-checked cells (81%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.verifiability | -0.40 | +0.60 | 1.00 |
| human.verifiability | -0.20 | +0.70 | 0.90 |
| recursion.coherence | -0.20 | +0.50 | 0.70 |
| interface.verifiability | +1.00 | +0.40 | 0.60 |
| human.safety | -0.10 | +0.50 | 0.60 |
| disclosure.economy | -0.18 | +0.40 | 0.58 |
| recursion.verifiability | +0.08 | +0.50 | 0.42 |
Ground truth
4 of 5 requirements met · deterministic checker, no model judgment- compiles to final format — ok
- 3+ sections — 6 sections
- a data table — yes
- a generated figure — yes
- a bibliography citation — no
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 23/23 | 7 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.9413
- hard-measured cells
- 53%
- teaching source
- markdown (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.