react-email
grade (soft)
reliability
overall
π Build and send emails using React
The react-email tool is a development framework that compiles React components into HTML email templates. Handing this tool to an agent presents an immediate compatibility barrier because the provided documentation conflicts with the installed environment version. The mismatch between unified imports and scoped packages forces the agent into an unproductive loop that fails to output any files.
Surfaces Γ lenses
| surface β lens β | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 47% of this matrix is hard-measured.
The experience
The agent attempts to generate email components like Html, Head, and Tailwind wrappers by following the guide instructions. The documentation works poorly because it directs the agent to import from a single, unified react-email package. The underlying environment actually runs version four, which requires importing from scoped packages like @react-email/components. This drift triggers immediate import errors during validation.
The model's dependency resolution mechanism works poorly. When faced with import errors, the agent tries to install @react-email/components and @react-email/render to reconcile the version split. The agent successfully installs these dependencies, but the environment fails to stabilize, forcing the agent into an expensive cycle of modifying imports and re-running commands.
Saving compilation results to disk works poorly. Despite correction attempts, the loop fails to write the final compiled template files to the target directory. The agent remains trapped in execution trials without producing any output file for the user.
Fit
This tool works poorly for fully autonomous agents that need to compile and output email templates without human intervention. The version mismatch between the provided documentation and the runtime environment halts agent progress immediately.
The tool works adequately only as an interactive autocomplete assistant for a human developer. In this scenario, the user can manually resolve the version mismatch, install the modern scoped packages, and verify the resulting directory structure themselves.
Notes
The diagram illustrates the loop failure caused by documentation drift and package split in the runtime environment.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 4 of 14 cross-checked cells (60%). Two independent methods landing on the same number is our accuracy signal β not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Ξ |
|---|---|---|---|
| loop.verifiability | +1.00 | -0.50 | 1.50 |
| recursion.economy | +0.92 | -0.50 | 1.42 |
| loop.determinism | +0.70 | -0.70 | 1.40 |
| interface.verifiability | +1.00 | -0.30 | 1.30 |
| recursion.coherence | +0.47 | -0.50 | 0.97 |
| interface.coherence | +0.43 | -0.50 | 0.94 |
| human.coherence | +0.30 | -0.50 | 0.80 |
| human.safety | +0.60 | +0.00 | 0.60 |
| human.verifiability | +0.10 | -0.50 | 0.60 |
| loop.safety | +0.50 | +0.00 | 0.50 |
Models & runs
2026-07-06| Model | Harness | Runs | Solved | Turns | Knows it |
|---|---|---|---|---|---|
 glm-5.2 |  PI | 25 | 19/25 | 19 | 83% |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn).
Hover a model β here or in the graph β to isolate its runs.
βKnows itβ = a closed-book quiz (20 questions, no docs) on whether the model already knows this tool from pre-training β 86% across the panel.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $1.8908
- hard-measured cells
- 47%
- teaching source
- react-email (official)
Same brief, same reference agent, and the same trial count for every tool in a category β a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.