rst
grade
reliability
overall
The rst toolchain compiles reStructuredText files into structured target formats like HTML using CLI tools such as rst2html.py or pandoc. Handing this tool to an agent establishes an explicit, file-based edit-compile-verify loop driven by strict formatting syntax that leads to deterministic output. However, the parser and compilation utilities suffer from a high rate of unexpected failures and run-to-run execution instability.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
An agent works with this tool through a standard file-based editing and compilation pipeline. It drafts source text files with .rst extensions and then triggers shell commands like rst2html.py or pandoc to render them. When the compiler encounters formatting problems, it outputs explicit syntactic error messages containing specific line numbers, allowing the agent to systematically locate and correct errors in a closed loop.
The markup syntax supports robust automated editing because rules on directives, table layouts, and section headers are concrete. For example, directives like .. list-table:: rely on consistent indentation and explicit options like :header-rows: 1. Underline headers must span at least the length of the title text. These strict, checkable constraints allow the agent to verify its formatting deterministically and ensure that complex elements like code blocks, hyperlinked text, and images render correctly in the final HTML.
The state of the document is entirely preserved within the plaintext .rst file, which benefits multi-turn workflows and agent handoffs. If a new agent session resumes from existing files, it can build upon or modify the source without breaking prior formatting or introducing compounding syntax errors. This flat-file structure prevents state corruption and simplifies agent memory constraints.
However, the toolchain behaves unreliably across repeated executions. Even when the source code remains identical and correct, repeated compilation runs frequently fail due to environment or utility instability. This unpredictable execution profile disrupts automated feedback loops, forcing the agent to handle tool failures that do not stem from its own syntax mistakes.
Fit
This tool is a good fit for agents tasked with generating deeply structured documents containing nested sections, tables, and nested code blocks. Its explicit syntax rules and informative compiler warnings allow the agent to systematically author and repair complex technical documents. It is also well-suited for asynchronous workflows where progress must be safely handed off to subsequent agent sessions using simple, plaintext source files.
It is a poor fit for workflows that require highly stable and repeatable execution pipelines. The run-to-run instability of the compiler utilities makes it unsuitable for pipelines where intermittent CLI crashes or infrastructure rendering failures cannot be tolerated. Agents in these environments will waste processing steps attempting to resolve environment errors rather than document content issues.
Notes
An agent utilizes an edit-compile-execute loop to author reStructuredText documents, compile them, and resolve syntax errors reported by the parser.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 8 of 16 cross-checked cells (72%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.verifiability | -0.40 | +0.80 | 1.20 |
| recursion.coherence | -0.75 | +0.40 | 1.15 |
| interface.coherence | -0.50 | +0.60 | 1.10 |
| disclosure.economy | -0.17 | +0.90 | 1.07 |
| human.verifiability | -0.20 | +0.80 | 1.00 |
| recursion.verifiability | +0.06 | +0.70 | 0.64 |
| loop.determinism | +0.10 | +0.60 | 0.50 |
| human.safety | -0.10 | +0.40 | 0.50 |
Ground truth
7 of 7 requirements met · deterministic checker, no model judgment- renders/compiles cleanly — ok
- has a title — —
- nested sections (2+ levels) — —
- a table — yes
- a code block — —
- a link — —
- an image/figure — yes
Reliability (pass^k): one attempt succeeds 80% of the time · three in a row 40% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
| | 23 | 19/23 | 6 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.7772
- hard-measured cells
- 53%
- teaching source
- rst (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.