ffmpeg
grade
reliability
overall
FFmpeg enables an agent to perform image processing tasks such as scaling, cropping, tone adjustment, and text overlay via a single command-line interface. The tool processes still images by applying video filters and outputs them to target formats like WebP or JPEG. While the filter calculations are deterministic, configuring the exact filter chaining syntax often leads to execution failures and requires many iterations to produce the desired output.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Consistency | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
The agent interacts with the tool by chaining command-line arguments within filter-graphs to scale, crop, and adjust colors. Combining these visual modifications works adequately because the filters execute deterministically based on input parameters. However, assembling the syntax of a complex filter-graph is fragile. A single incorrect punctuation mark or parameter name in the filter string causes the entire command to fail, which forces the agent to use many interactive turns to resolve the errors.
Rendering text watermarks or overlays using the 'drawtext' filter performs poorly. The command requires explicit paths to system fonts, which frequently vary or are missing in different environments. Because FFmpeg does not automatically fall back to standard system fonts when a specified font is unavailable, the command errors out. The agent must spend a large amount of effort debugging font-path parameters and positioning syntax like text-width and height offsets.
Controlling file sizes and format options works well. The agent easily exports to WebP or JPEG and adjusts file sizes to fit within budgets by using explicit quality flags like '-quality' or '-q:v'. File safety and task recovery also work well. While the '-y' flag overwrites destination files, the agent limits the blast radius of destructive changes by writing to distinct intermediate files and successfully deleting them when the pipeline finishes.
Overall execution reliability behaves poorly because the tool is unstable run-to-run and repeated attempts often fail. However, maintaining workflow state works adequately. When an execution fails or a new agent is initialized, the agent can resume the task from the remaining disk files alone without breaking existing progress or corrupting successful prior work.
Fit
FFmpeg is a poor fit for agents performing quick, single-turn image tasks or those operating in highly restricted environments where system font paths cannot be verified. The fragile command syntax and unreliability on repeated runs make it difficult for an agent to execute tasks efficiently without human oversight.
It is an adequate fit for agents that have the ability to run multiple trial-and-error loops, inspect intermediate file sizes, and safely clean up working directories. For environments where deterministic scaling and color modifications are required, an agent with robust state tracking can successfully manage the tool by writing discrete step-by-step commands instead of a single, complex pipeline.
Notes
This diagram illustrates the agent's iterative workflow when building, debugging, and verifying an FFmpeg image processing pipeline.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 7 of 16 cross-checked cells (74%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.verifiability | -0.40 | +0.70 | 1.10 |
| loop.determinism | -0.20 | +0.60 | 0.80 |
| recursion.coherence | -0.25 | +0.50 | 0.75 |
| human.safety | -0.10 | +0.60 | 0.70 |
| human.verifiability | +0.10 | +0.80 | 0.70 |
| human.coherence | -0.20 | +0.50 | 0.70 |
| loop.safety | -0.10 | +0.50 | 0.60 |
| recursion.economy | +0.88 | +0.30 | 0.58 |
| interface.verifiability | +0.10 | +0.60 | 0.50 |
Ground truth
4 of 4 requirements met · deterministic checker, no model judgment- WebP output produced — 40KB
- JPEG fallback produced — 62KB
- cropped to ~1.91:1 frame — 1200×630 (1.90:1)
- each output under 200KB — 40KB, 62KB
Reliability (pass^k): one attempt succeeds 60% of the time · three in a row 10% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 23 | 15/23 | 8 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.9832
- hard-measured cells
- 53%
- teaching source
- ffmpeg (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.