# What AI-assisted data visualization benchmarks actually measure

Status: state-of-the-field crosswalk, evidence cut 14 August 2026.

This is a companion to [The state of AI-assisted data visualization
research](/reports/research-review/). It explains the
benchmarks and evaluations that appear throughout that review: what object the
system receives, what it must produce, how the result is judged, what human
context enters the evaluation, and which larger claims the result cannot
support.

It is not an exhaustive catalog of every chart dataset. It covers the
benchmarks that carry the main capability and technique claims in the current
evidence base, plus July and August 2026 work that materially expands the task
envelope. Scores are comparable only inside the named study and condition.

## Executive reading

Five conclusions survive the crosswalk.

1. **There is no single state-of-the-art visualization score.** Reproducing a
   chart, extracting its values, answering a question, detecting deception,
   comparing several charts, retrieving a chart from a document, reconstructing
   an interactive dashboard, and helping a reader are different capabilities.
2. **The benchmark frontier is expanding sideways.** Recent work adds real
   source data, professional charts, financial presentations, misleading
   designs, chart pairs, multi-chart scientific figures, chart-bearing
   documents, interaction state, and multiple languages. That expansion is as
   important as score gains on older tasks.
3. **The grader is part of the result.** Lexical overlap, code similarity,
   render success, model judges, expert ratings, browser replay, and human task
   performance can rank the same outputs differently. A benchmark should be
   read as a task-and-grader pair, not as a neutral bag of examples.
4. **Benchmark construction can make systems look more capable than human work
   does.** In the multi-chart PolyChartQA study, model accuracy was as much as
   27.4 percentage points lower on human-authored than model-generated
   questions. ChartDiff found that specialist and pipeline systems could win on
   ROUGE while losing badly on a human-aligned quality score.
5. **The largest remaining measurement gap sits after generation.** DashArena
   now measures task-grounded dashboard quality with browser replay, and small
   user studies measure parts of authoring experience. The field still rarely
   follows a generated artifact through acceptance, delivery, reader
   comprehension, later correction, and maintenance.

## A measurement ladder, not a leaderboard

The layers below are cumulative obligations, not interchangeable scores. A
system may pass an earlier layer and fail a later one.

| Layer | Question being measured | Typical evidence | What passing still does not prove |
| --- | --- | --- | --- |
| **1. Structure** | Did the system emit the requested fields, marks, layout, code, or specification? | exact match, structural similarity, code tokens, valid DSL | that the artifact runs, depicts the source faithfully, or communicates well |
| **2. Execution** | Does the code compile, render, and stay free of runtime errors? | execution rate, browser load, callback completion | that the rendered values or transformations are correct |
| **3. Data and semantic fidelity** | Are the values, labels, transformations, joins, calculations, and encodings faithful to the reference? | reference data, executable checks, task-specific rules, answer invariants | that a person can read the result or that the design is not misleading |
| **4. Visual perception and reasoning** | Can a model recover values, locate marks, answer questions, or compare charts from the rendered image? | QA accuracy, extraction error, grounding F1, comparative-summary ratings | that it can author a good chart, evaluate its source data, or serve as a reliable critic |
| **5. Integrity and quality** | Can a judge detect deceptive encodings, presentation defects, or expert-defined quality differences? | labeled misleaders, expert rubrics, paired preferences, calibrated model judges | that the judge knows the intended audience, domain truth, or reader outcome |
| **6. Interaction and state** | Do controls work, and is the resulting state correct after an intended action sequence? | browser replay, DOM inspection, callback checks, trajectory evidence | that the dashboard supports the right analysis or works across devices and assistive technology |
| **7. Human work and reader outcome** | Does the system improve authoring, comprehension, decisions, confidence calibration, or accessibility? | controlled user tasks, reader studies, delayed retention, field evidence | durable production value unless delivery, updates, ownership, and cost are also observed |

The main mistake this ladder prevents is treating an earlier layer as a proxy
for every later one. A clean render is not a correct analysis. Correct data is
not necessarily a readable or honest graphic. A model that reads a chart is not
automatically a good chart critic. A preferred authoring interface is not proof
that the resulting reader understands more.

## Generation and reconstruction benchmarks

These evaluations ask a system to create or recreate an artifact. They differ
substantially in how much of the desired result is specified and whether the
system sees source data, an image, a natural-language task, or an existing
artifact to revise.

| Evaluation | Input and requested output | Principal scoring evidence | Human context | Supported claim | Important nonclaim |
| --- | --- | --- | --- | --- | --- |
| [Plot2Code](https://arxiv.org/abs/2405.07990) (2024) | A scientific plot image; generate executable plotting code that recreates it. | code execution, text match, CLIP similarity, and multimodal quality ratings | no reader or production study | measures single-plot code reconstruction and visual/textual resemblance | does not measure whether the source data, analytical question, or design choice is appropriate |
| [MatPlotBench](https://arxiv.org/abs/2402.11453) inside MatPlotAgent (2024) | A natural-language plotting task; generate a Matplotlib figure. | a GPT-4V-based score over 100 author-built tasks, plus execution and ablations | limited human inspection; no reader outcome | isolates some benefit from execution, debugging, and rendered visual feedback | author benchmark and model judge do not establish general chart quality; some models regressed under the agent loop |
| [VisEval](https://arxiv.org/abs/2502.05036) inside nvAgent (2025) | A natural-language visualization query over one or more databases; generate and execute a visualization query. | heterogeneous checks over 2,524 queries and 146 databases, including execution and result correctness | no downstream user or reader study | measures structured database-to-visualization composition, especially multi-table cases | does not show that the resulting visual form aids interpretation or that organizational semantics are correct |
| [Text2Vis](https://aclanthology.org/2025.emnlp-main.1622/) (2025) | Data plus a question; generate an answer, chart code, and rendered chart. | pass criteria over 1,985 tasks combining answer, code, and annotated visual evidence | human-created annotations, but no authoring or reader outcome study | tests end-to-end answer-and-chart generation and cleanly ablates targeted feedback | its 26% to 42% matched gain supports answer-and-code feedback, not arbitrary extra prompting; visual feedback slightly reduced final pass to 41% |
| [RealChart2Code](https://arxiv.org/abs/2603.25804) (2026) | Real source data and realistic chart tasks, including multi-panel creation and iterative refinement. | execution, code, structure, visual and refinement measures | no production delivery or reader study | reveals a large transfer drop from simpler reconstruction tasks and exposes regressive editing | roughly 51% on this harder benchmark is not a prevalence estimate for ordinary chart work |
| [Raiven](https://arxiv.org/abs/2604.10008) (2026) | A fully specified scientific or information-visualization request; generate a restricted DSL compiled deterministically. | compile rate and VMPC over 100 prompts, plus seven-expert replication tasks | seven visualization experts; authors graded the main benchmark | strong bounded evidence that a DSL and compiler remove many code and linkage failures in stable scientific grammars | does not measure question discovery, deceptive design, raw-data truth, or general production use; its advantage narrows on ordinary 2D information visualization |
| [CharTide](https://aclanthology.org/2026.acl-long.1145/) (2026) | Chart-to-code tasks from ChartMimic, Plot2Code, and ChartX; train and compare chart-specialized models. | same-paper execution and fidelity comparisons, including answer-invariance training signals | no human authoring or reader outcomes | shows both baseline movement and gains from separating perception, code logic, and modality fusion | selected benchmark gains do not establish end-to-end data visualization competence |

### What this family establishes

Conventional chart generation has become more faithful, especially in visible
text and rendered resemblance. The remaining failures move toward semantics,
real data, multi-panel structure, and non-regressive revision. Constrained
representations and executable feedback help when they introduce information
or guarantees that a direct prompt lacks. They are not automatically useful on
every task or with every current model.

## Chart reading, extraction, integrity, and critique

These benchmarks observe a rendered chart or figure. Some ask for values or
answers; others ask for defects or quality ratings. Those tasks should not be
collapsed into a generic “vision quality” score.

| Evaluation | Object and task | Principal scoring evidence | Human context | Supported claim | Important nonclaim |
| --- | --- | --- | --- | --- | --- |
| [ChartQAPro](https://arxiv.org/abs/2504.05506) (2025) | 1,341 realistic charts from 157 sources; answer 1,948 human-written questions including conversations, hypotheticals, fact checks, and unanswerable cases. | question-answering accuracy by task category | one expert graduate student supplied a bounded human reference over sampled questions | older chart specialists transfer poorly to harder, more realistic distributions; the best reported model reached 55.81 versus an 85.02 human reference | the small human estimate is not a population norm; the benchmark is static and omits interaction |
| [Misviz](https://arxiv.org/abs/2508.21675) (2025) | Real and synthetic misleading charts; detect the misleading design. | F1 on separate real and synthetic distributions | labels derived from misleading-visualization taxonomies and curated examples | synthetic and real tests can reverse system rankings; frontier general models led on heterogeneous real charts while rules/classifiers remained competitive on controlled synthetic cases | one integrity classifier is not a universal visual-quality critic and does not measure reader harm |
| [VisJudge](https://arxiv.org/abs/2510.22373) (2025) | 3,090 single charts, multi-chart compositions, and dashboards; predict expert-adjudicated quality scores. | mean absolute error and correlation with the benchmark rubric | three crowd ratings per item followed by expert adjudication | a small task-specific critic can better match a defined quality rubric than larger general models | the rubric does not check source-data fidelity, interaction state, audience fit, or reader comprehension |
| [ChartAgent / ChartBench](https://aclanthology.org/2026.acl-long.843/) (2026) | Single charts and questions; a general model may call more than 40 chart-specific OCR, geometry, segmentation, and calculation tools. | overall QA accuracy plus matched tool ablations | 30 trajectories manually audited; no reader study | task-relevant executable tools raised the same GPT-4o orchestrator from 54.53% to 71.39%; chart-specific tools beat generic image tools by 30 points | half of audited trajectories needed recovery or failed, and single-chart QA is not authoring, critique, or publication |
| [FinChart-Bench](https://aclanthology.org/2026.acl-long.615/) (2026) | 1,200 financial charts from 2015–2024 corporate presentations; answer 7,016 true/false, multiple-choice, and numerical reasoning questions. | exact match for categorical answers and 5%-tolerant accuracy for numerical QA across 26 models | two stages of manual filtering and answer correction by the authors | real financial charts expose weak instruction following and spatial alignment; the top complex-QA scores remained around 60% | curated high-quality crops omit document retrieval and ordinary low-quality charts; financial QA is not financial advice or reader comprehension |
| [Chartography](https://arxiv.org/abs/2608.10677) (2026) | 100 deliberately difficult professional chart-reading tasks across 12 domain labels. | mean pass@1 over 20 runs per task and 30 model configurations | practitioner-authored tasks, three expert verifiers per task, acceptable answer ranges | professional chart reading remains a separate hard capability; the best tested configuration reached 45.0% and more reasoning usually helped only modestly | difficulty screening means the score is not the failure rate on ordinary professional charts |
| [Protecting MLLMs against misleading visualizations](https://aclanthology.org/2026.acl-long.377/) (2026) | 143 misleading and 124 non-misleading chart questions plus ChartQA; compare six inference-time correction methods. | exact or relaxed QA accuracy with bootstrapped intervals across 19 models | reuses human-validated visualization-literacy sets and adds 42 real-world misleading charts | average accuracy on misleading charts fell to 26.4%, near the 25.6% random baseline; table-based QA improved three tested models by 15.4–19.6 points | the best correction sometimes reduced ordinary-chart accuracy by up to 8.5 points; no intervention is a safe universal wrapper |

### What this family establishes

Reading values, reasoning, detecting deceptive encodings, and rating quality are
separable. The best technique depends on the failure. Converting a chart to a
table can protect against an inverted or truncated axis because the misleading
geometry disappears; it can also lose color, spatial emphasis, or a value
during extraction. Longer reasoning cannot repair a mark or axis the model
misperceived at the start. A critic should therefore be routed to a named
failure and checked for regression, not granted general authority over
“quality.”

## Multiple charts, documents, domains, and languages

This 2026 cluster matters because it replaces the clean isolated English chart
with more realistic units of reading: a pair, a multi-panel figure, a document
corpus, a professional domain, or a different language. It also makes dataset
construction effects unusually visible.

| Evaluation | Object and task | Principal scoring evidence | Human context | Supported claim | Important nonclaim |
| --- | --- | --- | --- | --- | --- |
| [ChartDiff](https://aclanthology.org/2026.alvr-main.19/) (2026) | 8,541 pairs of rendered charts from public time-series data; write a comparison summary. | ROUGE plus GPT-5.4 quality scores; the model score correlated .91 with ratings on 300 human-scored summaries | all accepted summaries manually verified; 300 outputs human-rated for judge calibration | comparative summarization is distinct from single-chart reading; multi-series pairs remain hard and lexical overlap can rank systems incorrectly | charts are synthetically rendered from selected public data, annotations are partly model-generated, and the main judge remains a model |
| [Beyond Single Plots / PolyChartQA](https://aclanthology.org/2026.findings-acl.1764/) (2026) | 534 multi-chart images containing 2,297 subcharts from 168 computer-science papers; answer 2,694 questions. | human accuracy, model-judge accuracy, BERTScore, and ablations by question source, difficulty, and figure structure | 519 human-authored questions; all generated questions manually verified; human/model judge agreement tested | multi-chart localization and reasoning cause large drops; human-authored questions were up to 27.4 points harder; explicit decomposition and verification improved some models by up to 5.39 points | evidence comes from 2024 computer-science figures, most with homogeneous subchart types; the name collides with a separate multilingual benchmark |
| [Chart-MRAG Bench](https://aclanthology.org/2026.acl-long.1164/) (2026) | 267 real documents containing 627 charts and 1,283 text passages; retrieve and answer 4,738 one- and two-hop questions across eight domains. | Recall@5/10 for retrieval; keypoint correctness and coverage for answers | 12 expert annotators, three reviews per sample, Fleiss kappa .82 | document retrieval is a major bottleneck; unified multimodal embeddings failed chart-intensive retrieval, and text-over-visual bias persisted | questions are semi-automatically generated from one publisher's documents; model-jury answer scoring and announced future release require caution |
| [POLYCHARTQA](https://aclanthology.org/2026.acl-long.2043/) (2026) | 22,606 rendered charts and 26,151 QA pairs across 10 languages and 16 chart types. | type-aware relaxed accuracy, OCR/value diagnostics, cross-lingual conditions, and fine-tuning ablations | automated validation plus targeted human review and language-level agreement checks | English benchmark competence does not transfer evenly; OCR and visual-language alignment dominate much of the multilingual gap, and dedicated training produces large gains | most items are translated and rerendered from existing English datasets; ten languages do not represent global use, cultural chart conventions, or real non-English work |
| [MM-JudgeBench](https://aclanthology.org/2026.findings-acl.1746/) (2026) | More than 60,000 pairwise multimodal preferences in 25 languages, including a chart-centric OpenCQA subset; choose the better answer. | pairwise accuracy, cross-language variance, position bias, length bias, and rationale quality | translation quality filtering; 300 chart-subset cases used to check GPT-5 reference judgments with three experts | automatic visual judges vary by language, task, and bias; model size and architecture alone do not guarantee judge reliability | the chart subset inherits English source content and GPT-5 preference labels, then translates text while holding images fixed; it is evidence about judges, not chart readers |

Two different 2026 datasets use the name **PolyChartQA**. One is a multilingual
single-chart QA benchmark; the other is a multi-chart scientific-figure QA
benchmark. A benchmark name is not a capability definition—always identify the
object, task, and source distribution.

### What this family establishes

The isolated chart is becoming an insufficient evaluation unit. Real work asks
systems to find the relevant chart, integrate it with surrounding prose,
compare views, track legends across panels, and operate outside English. These
benchmarks also weaken several easy assumptions: synthetic questions can be
easier than human questions, English can be a hidden pivot, a unified
multimodal retriever can underperform separate stores, and a judge that is
accurate on average can still exhibit material language or position bias.

## Interactive dashboards and human outcomes

This is the thinnest and most consequential part of the evidence base.

| Evaluation | Object and task | Principal scoring evidence | Human context | Supported claim | Important nonclaim |
| --- | --- | --- | --- | --- | --- |
| [DashboardQA](https://aclanthology.org/2026.findings-eacl.177/) (2026) | Navigate real interactive dashboards to answer questions requiring grounded actions. | task accuracy plus interaction traces and error analysis | benchmark tasks, not a reader or analyst study | chart reading plus GUI navigation remains difficult; the best reported agent reached 38.69% | answer accuracy does not measure dashboard generation, reader comprehension, or production reliability |
| [Dashboard2Code](https://arxiv.org/abs/2607.04727) (2026) | Reconstruct Plotly Dash applications from screenshots, optional DOM, and interaction over 180 dashboards and 450 tasks. | visual, code, dynamic-behavior, and callback checks; automatic score calibrated against 90 dashboards rated by three graduate evaluators | three visualization-experienced evaluators for metric calibration | DOM access improves exploration and static screenshots miss factually wrong hidden state | fixed 1920×1080 desktop, Plotly Dash only, no animation or popups, and no reader study |
| [DashArena](https://arxiv.org/abs/2608.10567) (2026) | Generate a single-file interactive dashboard and a two-turn intended-use trajectory for 234 open-ended tasks. | browser render and replay, schema and execution evidence, human-calibrated pairwise judging | six dashboard-experienced annotators; 100-pair calibration and failure audits | interaction evidence materially improves judging; current systems can attempt open-ended analytical dashboards but remain unreliable in render, replay, and semantics | model-authored trajectories are partial; Tableau-derived tasks and aggregate preference do not represent every audience, device, or decision |
| [Data Formulator 2](https://arxiv.org/abs/2408.16119) and [DashChat](https://arxiv.org/abs/2504.12865) user studies | Create, inspect, and revise charts or dashboard prototypes through mixed natural-language and direct controls. | task completion, observed strategies, interviews, and preference in small contextual studies | analysts, domain professionals, and designers in bounded authoring tasks | visible intermediate data, history, local edits, and direct manipulation support steering and verification | small studies of authoring or prototyping do not establish reader comprehension, governed production, or maintenance benefit |
| Reader and learning studies in the main review | Read or learn from charts with proactive dialogue, passive assistance, data stories, tactile graphics, or text. | immediate comprehension, preference, confidence, and qualitative mental-model evidence | includes a randomized 117-person educational study and a 12-person blind/low-vision study | delivered interaction and accessibility context can change the outcome; preference and spatial understanding need not improve measured accuracy | these studies do not yet evaluate professionally produced AI-generated charts through delivery and later use |

### What this family establishes

The field now has a credible middle layer between “the code ran” and “a human
benefited”: intended-use trajectories, browser replay, interaction state, and
human-calibrated artifact comparison. It still lacks a complete episode. No
captured benchmark follows the same task through authoritative data selection,
artifact acceptance, authenticated delivery, mobile and assistive use, reader
understanding, later data changes, correction, and ownership transfer.

## Where the evidence converges

- **Rendered and executable evidence beats prose-only self-review** when it
  exposes a genuinely new failure channel.
- **Structured intermediate representations help when they preserve the facts
  needed for the next check.** They become harmful when they discard relevant
  visual evidence or introduce extraction error.
- **Harder and more realistic distributions expose brittle benchmark fit.** A
  system that leads on clean or synthetic charts may transfer poorly to
  real-world, professional, multilingual, deceptive, or multi-chart cases.
- **Human calibration matters at every subjective layer.** Model judges can be
  useful, but their agreement, language behavior, position bias, and metric
  sensitivity need to be measured.
- **Interaction is data.** A final screenshot cannot establish callback logic,
  hidden state, or whether an intended analysis can actually be replayed.

## Where the evidence legitimately diverges

- **Image-first versus table-first reading.** Image-first models retain visual
  cues. Table-first systems make values inspectable and can neutralize some
  deceptive geometry. Both are valid when routed to the information each
  preserves.
- **General versus specialized models.** Current general models lead several
  transfer tests, while specialist training, grounding, and tools produce large
  gains on particular component failures. The useful boundary is task- and
  distribution-specific.
- **Direct code versus a DSL or declarative plan.** A compiler pays off in a
  stable grammar with recurring syntax and linkage failures. Direct code may be
  adequate and more expressive for common, reversible one-offs.
- **Automatic versus human grading.** Deterministic checks are essential for
  facts and behavior; human judgment is required for contextual quality and
  outcomes. Neither can substitute wholesale for the other.

## Demonstrated failures and net-negative techniques

The negative evidence is not a footnote. It constrains what the field should
try next.

- More reasoning can elaborate a mistaken visual premise; Chartography found
  only modest median gains from higher effort.
- Generic examples were null in Text2Vis, and adding visual feedback slightly
  reduced its final pass rate despite better visual subscores.
- Plot2Code found no clear general advantage for chain-of-thought or
  Plan-and-Solve; more detailed requirements could improve resemblance while
  lowering execution.
- A chart-processing component in nvAgent slightly hurt a stronger model on
  average while helping weaker and multi-table settings.
- Multi-stage deception mitigation helped some models and harmed another.
- Table-based misleading-chart QA added 15.4–19.6 points on the target cases
  but cost up to 8.5 points on non-misleading charts because extraction added
  errors.
- Model-generated questions materially overstated multi-chart performance
  relative to human-authored questions.
- Lexical overlap rewarded ChartDiff systems that a human-aligned judge rated
  much worse.
- A responsive dashboard can still contain wrong calculations, bindings, or
  state; render success and replay success remain incomplete gates.

## The missing benchmark

The next decisive benchmark is not merely larger. It should join the layers
that are currently separated:

1. a declared human purpose and visualization environment;
2. authoritative source data, semantics, permissions, and uncertainty;
3. an accepted artifact with inspectable specification and executable tests;
4. browser evidence across desktop, mobile, and relevant assistive technology;
5. a representative creator or analyst task with total time, repair, and
   abandonment recorded;
6. a representative reader task measuring comprehension, calibrated trust, or
   decision quality; and
7. a later update that tests whether another person can correct and maintain
   the artifact.

That is expensive by design. Smaller benchmarks should continue to isolate
component failures. The field needs both: focused diagnostics that explain what
broke, and a complete episode that reveals whether the assembled system
actually improved human work and communication.

## Reading rule

For any new visualization result, ask seven questions before using it in a
state-of-the-field claim:

1. What exact object did the system see?
2. What exact object did it have to produce or judge?
3. Was the source real, synthetic, translated, reconstructed, or curated?
4. Which metric or judge determined success, and how was that judge calibrated?
5. Which humans authored, verified, rated, used, or read the artifact?
6. What failure types and excluded contexts remain outside the test?
7. What practical decision can this result support without crossing those
   boundaries?

If a benchmark description does not make those answers legible, its score is
not yet interpretable.

## Update log

- **2026-08-14 — Initial public edition.** Crosswalked generation, execution,
  semantic, perceptual, integrity, interaction, and human-outcome evaluations
  to their supported claims and important nonclaims.