mediawiki
grade
reliability
overall
The mediawiki utility is a markup language compiler system that converts text files into HTML documents. The agent writes document structure, tables, syntax-highlighted code, and hyperlinks inside a text file, then compiles it using a command-line utility. The tool functions predictably because translation from the defined markup to HTML is deterministic.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
The text conversion process works well because Pandoc compiles core markup elements like lists, links, and tables in a highly deterministic manner. Syntax parsing also works well because the agent's pre-training data aligns with standard MediaWiki syntax patterns, allowing it to write valid markup files without syntax-related errors. Safety and reversibility work well because document rendering is highly isolated, meaning syntax mistakes are easily corrected and do not cause compound host errors.
In contrast, visual verification works poorly because text-based environments prevent the agent from directly inspecting the rendered layout or validating styling choices. This limitation makes detecting CSS or image scale anomalies difficult during the development loop.
Asset integration works adequately. While the agent can use custom scripts to embed local images as base64 data URIs, managing external styles, scripts, and media files introduces substantial file management overhead to the design loop.
Fit
This tool works well for agents that need to generate structured, text-heavy documentation or database articles with standard section layers. It operates highly reliably when transforming raw text input into structured pages according to predefined schemas.
It works poorly for agents that need to create highly dynamic user interfaces, complex visual animations, or layout-sensitive graphic designs. Without real-time visual rendering feedback, the agent cannot adjust precise margins, typographic scales, or pixel-perfect layout flows.
Notes
The following diagram illustrates the agent's sequential process of writing, compiling, and verifying a self-contained MediaWiki document.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 9 of 16 cross-checked cells (76%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| interface.coherence | -0.64 | +0.50 | 1.14 |
| recursion.coherence | -0.82 | +0.30 | 1.12 |
| disclosure.verifiability | -0.40 | +0.70 | 1.10 |
| human.verifiability | -0.20 | +0.60 | 0.80 |
| disclosure.economy | -0.25 | +0.40 | 0.65 |
| recursion.determinism | +1.00 | +0.50 | 0.50 |
| recursion.verifiability | +0.00 | +0.50 | 0.50 |
Ground truth
7 of 7 requirements met · deterministic checker, no model judgment- renders/compiles cleanly — ok
- has a title — —
- nested sections (2+ levels) — —
- a table — yes
- a code block — —
- a link — —
- an image/figure — yes
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 22/23 | 7 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $1.0761
- hard-measured cells
- 53%
- teaching source
- mediawiki (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.