Back to explorer
data-visualization SDKs & libraries best source: official docs

altair

A

grade

100%

reliability

+0.31

overall

Vega-Altair is a declarative statistical visualization library for Python that translates chart specifications into Vega-Lite JSON format. The library allows AI agents to construct charts by chaining encoding methods and layering distinct graphic marks together. Its strict schema-driven API ensures that generated chart code operates deterministically, though agents require additional export dependencies to generate static image formats such as PNG or SVG.

Surfaces × lenses

hostile friendly
surface ↓
lens →
Coherence Economy Consistency Verifiability Safety
Disclosure
Interface
Loop
Recursion
Human

Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.

The experience

The agent constructs charts by declaring global parameters such as datasets and titles, then chaining encoding properties with specific data-type suffixes. This declarative style reduces code complexity and minimizes syntax errors during the initial composition phase. The API is robust enough that repeated executions output program files cleanly, demonstrating high reliability under autonomous execution.

The main workflow bottleneck occurs during visual verification. Because the agent cannot directly view the rendered output, it must inspect raw SVG text or assume the visual layout is correct based on the lack of compilation errors. This lack of direct visual feedback causes the agent to perform multiple iterative attempts to align complex requirements, such as placing annotations precisely offset from data points.

Layering multiple chart elements, such as adding custom text labels to a peak data point, requires a specific workflow of filtering datasets, creating separate sub-charts, and merging them with addition syntax. Writing these compound structures often increases execution turns because the agent must carefully align encoding dimensions across the merged layers. Any mismatch in specifications between the base chart and the label chart results in rendering failures.

When refining an existing plot over several iterations, keeping state cohesive is difficult. Since the agent cannot inspect visual changes, subsequent edits to properties such as scale types or domain limits often overwrite previous formatting rules. The agent lacks a mechanism to confirm whether a localized adjustment to one marker has inadvertently displaced other elements, leading to code that diverges from requirements over long conversations.

Fit

This tool works well for agents performing autonomous data visualization tasks where the visual format can be rigorously specified in advance and verified programmatically. It is ideal for standardized dashboard components and automated reporting pipelines that require deterministic, repeatable chart code.

The tool is poorly suited for interactive design environments or tasks requiring fine-grained aesthetic adjustments from an agent. Without human-in-the-loop validation or a multimodal agent capable of image inspection, real-time visual alignment remains highly expensive in developer loops.

Notes

This diagram illustrates the iterative workflow of an agent writing Altair code, compiling it to Vega-Lite, exporting it to SVG, and executing blind alignment adjustments.

Validation

Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 7 of 16 cross-checked cells (77%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.

Cell where they disagreeInstrumentRaterΔ
disclosure.verifiability-0.40+0.500.90
recursion.coherence-0.58+0.300.88
interface.coherence-0.16+0.600.76
loop.verifiability+1.00+0.400.60
interface.verifiability+1.00+0.500.50
recursion.determinism+1.00+0.500.50
recursion.economy+0.60+0.100.50
recursion.verifiability-0.20+0.300.50
loop.economy+0.55+0.100.45

Ground truth

2 of 2 requirements met · deterministic checker, no model judgment
  • figure produced — figure.png
  • chart source runs clean — exit 0

Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.

Models & runs

2026-07-07
ModelHarnessRunsSolvedTurns
glm-5.2 PI2318/237

Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.

How this was measured

reference agent
pi:z-ai/glm-5.2
trials
25 (n=5/tier)
methodology
v0.1 3e74a3c6
cost
$0.9706
hard-measured cells
53%
teaching source
altair (official docs) (official-docs)

Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.