latex
grade
reliability
overall
The latex tool is an automated LaTeX paper editing and compilation interface that runs compilation engines such as Tectonic and executes Python auditing scripts for document checks. It processes LaTeX manuscripts by routing tasks to specialized scripts for bibliography verification, syntax compilation, and structural formatting. The tool succeeds in producing cleanly rendered documents but suffers from poor run-to-run reliability and high execution overhead during iteration.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 62% of this matrix is hard-measured.
The experience
The agent operates the tool by reading the modular schema and invoking python utilities such as compile.py or running the tectonic command directly to compile manuscripts. The Tectonic engine works adequately because it automatically fetches missing packages and runs BibTeX without manual dependency management. This automated resolution reduces manual setup steps.
However, the iterative compile-and-repair loop works poorly due to high run-to-run unreliability. Minor syntax changes in the LaTeX source often trigger compilation errors, which requires the agent to perform multiple retries to achieve a clean render. Feedback verifiability is hostile because the tool does not provide a direct visual check of the generated PDF content to the agent, meaning the agent cannot easily confirm if the visual layout is correct.
The structured module router provides clear commands but performs poorly for document generation from scratch. The documentation and Python analysis scripts are tailored for auditing existing IEEE or ACM manuscripts. When asked to write a new paper, the agent is guided toward auditing scripts that do not apply to unwritten sections, forcing it to write raw LaTeX source without utilizing the helper scripts.
State recovery works well during continuous integration or agent handoff. Because files are updated directly on disk and compilation does not corrupt existing assets, subsequent agent attempts can resume the work without breaking previous progress. All compilation tasks are isolated and reversible, ensuring that build failures do not damage the file system.
Fit
The tool is a good fit for engineering teams that need to automate the maintenance, auditing, and repair of existing LaTeX papers. It works adequately for standard tasks such as validating bibliographies, formatting citations, and compiling stable documents with Tectonic. It is also suitable for workflows that involve multi-agent handoffs, as the deterministic file-system state allows new runs to resume seamlessly without losing progress.
It is a poor fit for workflows that require generating new documents from scratch or demand fast, low-overhead iteration. The lack of visual layout feedback and the high frequency of compilation errors make it expensive and slow for real-time generation. Furthermore, teams requiring high run-to-run reliability will find the compile loop too fragile for fully autonomous execution.
Notes
This diagram represents the typical write, compile, and repair loop executed by the agent using the LaTeX compilation tool.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 7 of 13 cross-checked cells (75%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.verifiability | -0.50 | +0.50 | 1.00 |
| loop.economy | -1.00 | +0.00 | 1.00 |
| recursion.economy | +1.00 | +0.00 | 1.00 |
| loop.determinism | -0.30 | +0.50 | 0.80 |
| recursion.coherence | -0.25 | +0.50 | 0.75 |
| recursion.determinism | +1.00 | +0.50 | 0.50 |
Ground truth
5 of 5 requirements met · deterministic checker, no model judgment- compiles to final format — ok
- 3+ sections — 5 sections
- a data table — yes
- a generated figure — yes
- a bibliography citation — yes
Reliability (pass^k): one attempt succeeds 20% of the time · three in a row 0% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns | Knows it |
|---|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 10/23 | 7 | 85% |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn).
Hover a model — here or in the graph — to isolate its runs.
“Knows it” = a closed-book quiz (20 questions, no docs) on whether the model already knows this tool from pre-training — 89% across the panel.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $1.4355
- hard-measured cells
- 62%
- teaching source
- latex-paper-en (skills.sh)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.