ffmpeg
grade (soft)
reliability
overall
FFmpeg is a command-line utility that processes media assets through CLI commands and nested filter chains. Handing an agent this tool requires it to construct long, syntactically complex arguments that are prone to parsing errors. Without highly specific documentation, the agent struggles to manage the complex filter syntax and frequently fails to produce correct video compositions.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 47% of this matrix is hard-measured.
Teaching sources compared
docs (B+) serves the agent better than skill (C) for ffmpeg. The gap is widest on Disclosure (0.19).
| Source | Grade | Renders | Disclosure | Interface | Loop | Recursion | Human |
|---|---|---|---|---|---|---|---|
| official docs ★ | B+ | · | +0.24 | +0.42 | +0.33 | +0.28 | +0.21 |
| skill | C | · | +0.44 | +0.34 | +0.15 | +0.22 | +0.27 |
Same harness, same reference agent, same briefs, same trial count — the only variable is how the agent was taught the tool. ★ = best source.
The experience
For basic image operations like cropping, scaling, and brightness adjustments, the tool works adequately because the CLI syntax maps inputs to explicit filters using direct key-value assignments. The execution is fast, deterministic, and highly reliable once the parameters are set correctly.
For video composition and multi-asset transition tasks, the tool works poorly. The agent struggles to construct FFmpeg's filter graph syntax, which uses strict colon and comma separators to chain multiple media inputs. When the documentation covers only simple image pipelines, the agent cannot easily extrapolate how to build video transitions or register multi-track overlays.
The tool's error reporting works poorly for complex pipelines. When a nested filter like overlay or xfade fails, the parser returns generic syntax errors that do not pinpoint the invalid parameter, causing the agent to run multiple redundant debugging loops. This lack of diagnostic granularity causes the agent to consume a large amount of token context as it repeatedly modifies inputs and retries the command.
Post-execution verification also works poorly. Because FFmpeg provides only textual log summaries of output streams, the agent has no direct way to inspect visual alignment issues or pixelation during execution. The agent must rely on auxiliary scripting or layout analysis to detect these rendering failures, which often happens too late to correct.
Fit
FFmpeg is a good fit for agents performing automated, single-step media utilities, such as resizing static images, transcribing audio streams, or converting file formats. When tasks are constrained to these direct commands with explicit parameter structures, the execution is deterministic and fast.
It is a poor fit for agents tasked with dynamic creative production or complex video editing, especially when precise, task-aligned documentation is not provided. The high syntactic burden of debugging nested filtergraphs, coupled with the lack of direct visual feedback during runtime, causes the agent to default to fragile workarounds that fail to produce high-quality output.
Notes
This diagram displays the iterative loop an agent runs when constructing layered media overlays and processing them through the command line.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 5 of 14 cross-checked cells (66%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| loop.verifiability | +1.00 | -0.50 | 1.50 |
| interface.verifiability | +1.00 | -0.50 | 1.50 |
| recursion.coherence | +0.50 | -0.50 | 1.00 |
| recursion.economy | +0.31 | -0.60 | 0.91 |
| interface.coherence | +0.50 | -0.40 | 0.90 |
| loop.safety | +0.70 | +0.00 | 0.70 |
| human.verifiability | +0.35 | -0.30 | 0.65 |
| loop.determinism | +0.24 | -0.40 | 0.64 |
| loop.economy | -0.26 | -0.70 | 0.44 |
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 21/23 | 6 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $1.0166
- hard-measured cells
- 47%
- teaching source
- ffmpeg (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.