typst
grade
reliability
overall
Typst is a typesetting tool that compiles structured markup source files into PDF documents. Providing this tool to an agent allows for local document compilation and deterministic file rendering using short command-line invocations. However, while generating basic structured text succeeds, the agent struggles to correctly format advanced elements like tables and native figures, resulting in unmet requirements despite extensive iterative efforts.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
The basic document compilation workflow works well because the command-line compiler is deterministic and lightweight. The agent writes a plaintext markup file and compiles it with a single command, incurring low computational overhead. Typst catches syntax errors at compile time and provides immediate feedback, allowing the agent to safely test and correct syntax issues without causing system-wide side effects.
Designing complex components such as tables and graphical figures works poorly because the agent struggles to coordinate low-level primitives. While standard text formatting is straightforward, drawing custom diagrams requires manual calculation of spatial coordinates. When external packages are missing from the local environment, the agent must construct graphics by positioning shapes manually. This limitation results in long editing loops that often fail to render the requested data visualizations or tables.
Task handoff and state restoration work adequately because the entire document's state is fully transparent and file-based. All structural source text and citation keys are stored in a small set of local files. A fresh agent resuming a task can read these files and run the compiler without encountering hidden state corruption or breaking prior formatting work.
Fit
Typst is a good fit for agents tasked with generating linear, text-heavy documents containing standard elements like headings, bulleted lists, standard bibliographies, and simple tables. The clear, minimal syntax matches standard programming patterns, making it easy for the agent to assemble structured content.
It is a poor fit for tasks that require complex, data-driven visualizations or sophisticated graphic layouts unless auxiliary pre-rendered SVGs are supplied. If an agent must draw charts from scratch without verified local packages, the spatial reasoning required is too high, leading to excessive, unsuccessful feedback loops that consume execution steps.
Notes
The diagram outlines how the agent develops and tests Typst documents, highlighting the loop where coordinate calculation failures occur.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 10 of 16 cross-checked cells (79%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| recursion.determinism | +1.00 | -0.20 | 1.20 |
| recursion.economy | +0.76 | -0.40 | 1.16 |
| disclosure.verifiability | -0.40 | +0.50 | 0.90 |
| disclosure.economy | -0.07 | +0.40 | 0.47 |
| loop.verifiability | +0.16 | -0.30 | 0.46 |
| loop.economy | +0.04 | -0.40 | 0.44 |
Ground truth
3 of 5 requirements met · deterministic checker, no model judgment- compiles to final format — ok
- 3+ sections — 4 sections
- a data table — no
- a generated figure — no
- a bibliography citation — yes
Reliability (pass^k): one attempt succeeds 80% of the time · three in a row 40% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 17/23 | 7 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.9149
- hard-measured cells
- 53%
- teaching source
- typst (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.