A practical evaluation

Measure the complete creator-to-artifact-to-reader episode.

A prettier first render is not enough evidence for a workflow change.

  1. 01

    Name the environment and audience.

    Exploration, governed BI, operations, public explanation, science, education, or a reusable application carry different obligations.

  2. 02

    Use representative work.

    Include self-owned data, awkward semantics, a correction request, and the actual delivery surface. Retain a simple task as a control.

  3. 03

    Record the current baseline.

    Measure human time, errors, correction path, and delivery effort before claiming a gain.

  4. 04

    Separate first draft from acceptance.

    Count prompting, waiting, cleanup, verification, publishing, and monetary cost.

  5. 05

    Inspect consequential choices.

    Make fields, filters, aggregation, transformations, code or semantic query, and generated interaction state visible.

  6. 06

    Test the delivered artifact.

    Recompute values, exercise controls, check exports, accessibility, responsive behavior, and future maintenance.

  7. 07

    Test readers separately.

    Ask defined readers for the claim, evidence, uncertainty, and next action. Measure correctness, time, confidence, and harmful misreadings.

The updated gap ledger

Correction, prepared change, three production afterlives, reader harm, accessibility, one handoff, and adjacent longitudinal governance now have bounded evidence. Seven leading cases still yield zero full release-validation chains.

“Partly answered” means one bounded study exists—not that the field can generalize.

Partly answered
  • Capability to human consequence3 of 10 controlled partial bridges; 0 accepted-delivery bridges
  • Correction and premature acceptance108 engineered-error episodes; no field distribution
  • Prepared repair and evolution50 Excel repairs + 80 changed visualizations; no field afterlife
  • Same-artifact production afterlifethree cases; the newest reaches maintainer recheck, not independent recovery
  • Exact release-validation join0 of 7 leading cases clears all six receipts
  • Longitudinal co-designthree returning blind co-designers across eight months; no formal evaluation or release recheck
  • Reader comprehension and harm117-person scaffolding study + 48-person misleading-chart study
  • Mobile and nonvisual outcomescontrolled non-AI delivery studies
  • AI-assisted BLV learningpreference and spatial model; no accuracy lift
  • Second maintenanceone direct episode; no rate or comparison
  • Longitudinal visualization governance21 projects + one patient deployment; not generative authoring
Still materially open
  • Independent production comparison and maintenance rates
  • Whole human + model cost
  • Second-person handoff on an inspectable artifact
  • Local semantic accuracy
  • Accessible delivery on readers’ own devices
  • Decision quality and trust calibration