Matched, replicated comparison · v2.5.0 · 90 Runs

One spec, 90 agentic builds of the same app

Every run received the identical real-time retrospective-board specification, was scored on a 14-criterion functional rubric (42-point scale), and was characterised for visual design treatment. The dataset spans model generations, two agent harnesses, reasoning-effort levels, a visual automation and testing tool (the Playwright MCP server), and a design-oriented prompt condition (a third-party prompt, not redistributed, and a one-paragraph paraphrase of its design directive, which is) — with six or seven repeated runs of identical configurations to quantify run-to-run variation. Conditions were chosen rather than randomised.

90
Graded runs
31
Perfect 42/42 on the first try
5
Model families
/42
14 criteria × 3
The Task

One spec, 90 runs

Each run built a self-hosted, real-time retrospective board (React/Vite + Node.js + Socket.io + SQLite) from the same OpenSpec. Requirements:

Claude Code runs were produced with Claude Code CLI v2.1.132, using the /opsx:apply command to have the agent implement the OpenSpec specification, zero shot. Antigravity runs used that agent's own harness and injected system prompt. The specification is identical in every run folder.

HarnessModel familyRunsMeanRangeCost
Claude CodeClaude Opus 4.74841.339–42$2.22–7.63
Claude CodeClaude Opus 4.62640.738–42$2.04–6.62
Claude CodeClaude Sonnet 4.6940.939–42$1.08–4.90
Claude CodeQwen230.524–37$41.41–178.88
AntigravityClaude Opus 4.6140.040not recorded
AntigravityClaude Sonnet 4.6142.042not recorded
AntigravityGemini 3.1339.338–40not recorded

Cost is Claude Code's /cost estimate from token counts at per-token model prices. The two Qwen models ran locally, so their figures are token-volume estimates at the hosted-model rate under which the harness reported the usage, not prices paid. The Antigravity harness does not report session cost.

Scoring Instrument

The 14-criterion rubric

Every run is scored on the same 14 functional criteria, each rated 3 / 2 / 1; the run's total is their sum (42 max). The numbered columns in the heatmap below correspond to these criteria.

#CriterionWhat it checks
1Local Dev EnvironmentDoes the local development environment come up without manual code changes?
2Docker DeploymentDoes the Docker image build and run without errors?
3Home PageDoes the landing page load correctly and show the board dashboard?
4Board CreationCan you create a new board with a custom title?
5User IdentificationCan users identify themselves for the board?
6Card InteractionCan you add cards to columns?
7Moving CardsCan cards be moved between different columns?
8CommentingCan users add comments to existing cards?
9Realtime UpdatesDoes a new card added to a board reflect in other browser windows instantly without refresh?
10Realtime UpdatesDoes a card moved to a different column reflect in other browser windows instantly without refresh?
11Realtime UpdatesDoes a new comment added to a card reflect in other browser windows instantly without refresh?
12Data PersistenceDoes data survive a server reboot?
13DocumentationIs there documentation for the API and running the app?
14ExportDoes the app export data to CSV?
3 — Pass

Worked on the first try, no changes needed.

2 — Fixed

Failed initially, fixed after one corrective prompt that described only the observed error.

1 — Failed

Never fully worked despite prompting.

Opus 4.7 Effort Grid

Effort buys what the tool didn't

A 2 × 4 design — High/xHigh effort across base, +Playwright, +full design prompt and +abridged design prompt, six replicates per cell. The abridged cells were added later as an ablation and are kept out of the pooled effort contrast. Pooling the six sweep cells, raising effort from High to xHigh moves first-try-perfect from 28% (5 of 18) to 89% (16 of 18); the testing tool moves cost, not score; either design prompt changes what is built and costs first-try reliability, not the total.

CellMean scoreFirst-try 42/42Cost (median)Visual rating (runs at 3 / 4 / 5)
High · base41.02/6$3.066 / 0 / 0
High · +Playwright41.53/6$4.346 / 0 / 0
High · +full design prompt40.50/6$4.280 / 1 / 5
High · +abridged design prompt40.81/6$4.140 / 0 / 6
xHigh · base42.06/6$3.336 / 0 / 0
xHigh · +Playwright42.06/6$5.596 / 0 / 0
xHigh · +full design prompt41.54/6$5.170 / 1 / 5
xHigh · +abridged design prompt41.01/6$4.730 / 0 / 6
Score Rankings

All 90 runs

Sorted by total score, then cost. Click any card for its screenshots. Repeated configurations appear multiple times by design — that spread is the point.

Sonnet 4.6
42
claude_sonnet_4.6_high_run_3
High
$1.08★★★☆☆
Sonnet 4.6
42
claude_sonnet_4.6_high_with_antigravity_prompt_run_3
HighDesign prompt
$1.82★★★★☆
Opus 4.6
42
claude_opus_4.6_high_abridged_prompt_run_3
HighAbridged design prompt
$2.10★★★★☆
Opus 4.6
42
claude_opus_4.6_high_run_5
High
$2.54★★★☆☆
Opus 4.7
42
claude_opus_4.7_xhigh_run_2
xHigh
$2.54★★★☆☆
Sonnet 4.6
42
claude_sonnet_4.6_with_playwright_high_run_2
HighPlaywright
$2.57★★★☆☆
Opus 4.7
42
claude_opus_4.7_xhigh_run_6
xHigh
$2.77★★★☆☆
Opus 4.7
42
claude_opus_4.7_high_run_6
High
$2.79★★★☆☆
Opus 4.6
42
claude_opus_4.6_high_abridged_prompt_run_2
HighAbridged design prompt
$2.94★★★★☆
Opus 4.7
42
claude_opus_4.7_high
High
$3.15★★★☆☆
Opus 4.7
42
claude_opus_4.7_with_playwright_high_run_2
HighPlaywright
$3.26★★★☆☆
Opus 4.7
42
claude_opus_4.7_xhigh
xHigh
$3.31★★★☆☆
Opus 4.7
42
claude_opus_4.7_xhigh_run_5
xHigh
$3.36★★★☆☆
Opus 4.6
42
claude_opus_4.6_high_abridged_prompt
HighAbridged design prompt
$3.45★★★★☆
Opus 4.7
42
claude_opus_4.7_with_playwright_high
HighPlaywright
$4.01★★★☆☆
Opus 4.7
42
claude_opus_4.7_xhigh_run_3
xHigh
$4.12★★★☆☆
Opus 4.7
42
claude_opus_4.7_xhigh_run_4
xHigh
$4.22★★★☆☆
Opus 4.7
42
claude_opus_4.7_with_playwright_high_run_3
HighPlaywright
$4.47★★★☆☆
Opus 4.7
42
claude_opus_4.7_with_playwright_xhigh_run_3
xHighPlaywright
$4.51★★★☆☆
Opus 4.7
42
claude_opus_4.7_with_playwright_xhigh_run_5
xHighPlaywright
$4.56★★★☆☆
Opus 4.7
42
claude_opus_4.7_xhigh_with_antigravity_prompt_run_2
xHighDesign prompt
$4.77★★★★★
Opus 4.7
42
claude_opus_4.7_xhigh_with_antigravity_prompt_run_4
xHighDesign prompt
$4.80★★★★★
Opus 4.7
42
claude_opus_4.7_xhigh_abridged_prompt
xHighAbridged design prompt
$4.83★★★★★
Opus 4.7
42
claude_opus_4.7_xhigh_with_antigravity_prompt_run_5
xHighDesign prompt
$4.99★★★★☆
Opus 4.7
42
claude_opus_4.7_with_playwright_xhigh
xHighPlaywright
$5.57★★★☆☆
Opus 4.7
42
claude_opus_4.7_xhigh_with_antigravity_prompt
xHighDesign prompt
$5.59★★★★★
Opus 4.7
42
claude_opus_4.7_with_playwright_xhigh_run_6
xHighPlaywright
$5.61★★★☆☆
Opus 4.7
42
claude_opus_4.7_with_playwright_xhigh_run_2
xHighPlaywright
$6.16★★★☆☆
Opus 4.7
42
claude_opus_4.7_high_abridged_prompt_run_2
HighAbridged design prompt
$7.21★★★★★
Opus 4.7
42
claude_opus_4.7_with_playwright_xhigh_run_4
xHighPlaywright
$7.63★★★☆☆
Sonnet 4.6
42
antigravity_sonnet_4.6
Antigravity—
—★★★★☆
Sonnet 4.6
41
claude_sonnet_4.6_high_run_2
High
$1.40★★★☆☆
Sonnet 4.6
41
claude_sonnet_4.6_high_with_antigravity_prompt
HighDesign prompt
$1.69★★★★☆
Opus 4.6
41
claude_opus_4.6_high_run_6
High
$2.04★★★☆☆
Opus 4.6
41
claude_opus_4.6_high_run_4
High
$2.09★★★☆☆
Opus 4.7
41
claude_opus_4.7_high_run_3
High
$2.22★★★☆☆
Sonnet 4.6
41
claude_sonnet_4.6_high_with_antigravity_prompt_run_2
HighDesign prompt
$2.27★★★★☆
Opus 4.6
41
claude_opus_4.6_high_abridged_prompt_run_4
HighAbridged design prompt
$2.31★★★★☆
Opus 4.6
41
claude_opus_4.6_high_with_antigravity_prompt_run_5
HighDesign prompt
$2.34★★★★★
Opus 4.6
41
claude_opus_4.6_high_abridged_prompt_run_6
HighAbridged design prompt
$2.60★★★★☆
Opus 4.6
41
claude_opus_4.6_high_with_antigravity_prompt_run_6
HighDesign prompt
$2.72★★★★☆
Opus 4.6
41
claude_opus_4.6_high_run_2
High
$2.77★★★☆☆
Sonnet 4.6
41
claude_sonnet_4.6_max_with_antigravity_prompt
MaxDesign prompt
$2.90★★★★★
Opus 4.7
41
claude_opus_4.7_high_run_4
High
$2.99★★★☆☆
Opus 4.6
41
claude_opus_4.6_high_with_antigravity_prompt
HighDesign prompt
$3.12★★★★☆
Opus 4.7
41
claude_opus_4.7_high_run_2
High
$3.13★★★☆☆
Opus 4.7
41
claude_opus_4.7_high_with_antigravity_prompt_run_6
HighDesign prompt
$3.15★★★★★
Opus 4.7
41
claude_opus_4.7_xhigh_abridged_prompt_run_4
xHighAbridged design prompt
$3.15★★★★★
Opus 4.6
41
claude_opus_4.6_with_playwright_high_run_6
HighPlaywright
$3.19★★★☆☆
Opus 4.6
41
claude_opus_4.6_with_playwright_high_run_5
HighPlaywright
$3.30★★★☆☆
Opus 4.7
41
claude_opus_4.7_high_abridged_prompt_run_5
HighAbridged design prompt
$3.33★★★★★
Opus 4.6
41
claude_opus_4.6_with_playwright_high_run_4
HighPlaywright
$3.47★★★☆☆
Opus 4.6
41
claude_opus_4.6_with_playwright_high_run_3
HighPlaywright
$3.56★★★☆☆
Opus 4.7
41
claude_opus_4.7_high_with_antigravity_prompt
HighDesign prompt
$3.64★★★★★
Opus 4.7
41
claude_opus_4.7_high_abridged_prompt_run_3
HighAbridged design prompt
$3.96★★★★★
Opus 4.7
41
claude_opus_4.7_high_with_antigravity_prompt_run_2
HighDesign prompt
$4.08★★★★★
Opus 4.7
41
claude_opus_4.7_with_playwright_high_run_4
HighPlaywright
$4.22★★★☆☆
Opus 4.7
41
claude_opus_4.7_high_abridged_prompt_run_4
HighAbridged design prompt
$4.32★★★★★
Opus 4.7
41
claude_opus_4.7_high_with_antigravity_prompt_run_3
HighDesign prompt
$4.48★★★★★
Opus 4.6
41
claude_opus_4.6_with_playwright_high_run_2
HighPlaywright
$4.49★★★☆☆
Opus 4.7
41
claude_opus_4.7_xhigh_abridged_prompt_run_6
xHighAbridged design prompt
$4.60★★★★★
Opus 4.7
41
claude_opus_4.7_with_playwright_high_run_6
HighPlaywright
$4.88★★★☆☆
Opus 4.7
41
claude_opus_4.7_xhigh_abridged_prompt_run_3
xHighAbridged design prompt
$4.89★★★★★
Opus 4.7
41
claude_opus_4.7_with_playwright_high_run_5
HighPlaywright
$5.03★★★☆☆
Opus 4.7
41
claude_opus_4.7_xhigh_with_antigravity_prompt_run_6
xHighDesign prompt
$5.34★★★★★
Opus 4.6
41
claude_opus_4.6_high
High
$5.61★★★☆☆
Opus 4.7
41
claude_opus_4.7_xhigh_abridged_prompt_run_2
xHighAbridged design prompt
$5.98★★★★★
Opus 4.6
40
claude_opus_4.6_high_run_3
High
$2.74★★★☆☆
Opus 4.6
40
claude_opus_4.6_high_with_antigravity_prompt_run_2
HighDesign prompt
$3.11★★★★☆
Opus 4.6
40
claude_opus_4.6_high_with_antigravity_prompt_run_3
HighDesign prompt
$3.51★★★★☆
Opus 4.7
40
claude_opus_4.7_high_abridged_prompt
HighAbridged design prompt
$3.55★★★★★
Opus 4.7
40
claude_opus_4.7_xhigh_abridged_prompt_run_5
xHighAbridged design prompt
$4.64★★★★★
Opus 4.7
40
claude_opus_4.7_high_abridged_prompt_run_6
HighAbridged design prompt
$4.66★★★★★
Opus 4.6
40
claude_opus_4.6_with_playwright_high
HighPlaywright
$5.36★★★☆☆
Opus 4.6
40
claude_opus_4.6_high_with_antigravity_prompt_run_4
HighDesign prompt
$6.08★★★★☆
Opus 4.7
40
claude_opus_4.7_high_with_antigravity_prompt_run_5
HighDesign prompt
$6.57★★★★★
Opus 4.7
40
claude_opus_4.7_xhigh_with_antigravity_prompt_run_3
xHighDesign prompt
$7.25★★★★★
Gemini 3.1 Pro high
40
antigravity_gemini_3.1_pro_high
Antigravity—
—★★☆☆☆
Gemini 3.1 Pro low
40
antigravity_gemini_3.1_pro_low
Antigravity—
—★★★☆☆
Opus 4.6
40
antigravity_opus_4.6
Antigravity—
—★★★☆☆
Sonnet 4.6
39
claude_sonnet_4.6_high
High
$1.58★★★☆☆
Opus 4.6
39
claude_opus_4.6_with_playwright_high_run_7
HighPlaywright
$2.08★★★☆☆
Opus 4.6
39
claude_opus_4.6_high_abridged_prompt_run_5
HighAbridged design prompt
$3.59★★★★☆
Opus 4.7
39
claude_opus_4.7_high_run_5
High
$3.96★★★☆☆
Opus 4.7
39
claude_opus_4.7_high_with_antigravity_prompt_run_4
HighDesign prompt
$4.71★★★★☆
Sonnet 4.6
39
claude_sonnet_4.6_with_playwright_high
HighPlaywright
$4.90★★★☆☆
Opus 4.6
38
claude_opus_4.6_high_run_1
High
$6.62★★★☆☆
Gemini 3.1 Flash
38
antigravity_gemini3_flash
Antigravity—
—★★★☆☆
Qwen 3.6
37
claude_qwen_3.6_high_with_playwright
HighPlaywright
$41.41★★★☆☆
Qwen Coder Next
24
claude_qwen_coder_next_high_with_playwright
HighPlaywright
$178.88★★★☆☆
Feature Heatmap

Runs × 14 criteria

Each cell is a criterion score (3 / 2 / 1); hover a column number for its definition. Rows sorted by total. Docker (col 2) and Local Dev (col 1) carry most of the misses.

Run · total1234567891011121314
claude_sonnet_4.6_high_run_3 · 4233333333333333
claude_sonnet_4.6_high_with_antigravity_prompt_run_3 · 4233333333333333
claude_opus_4.6_high_abridged_prompt_run_3 · 4233333333333333
claude_opus_4.6_high_run_5 · 4233333333333333
claude_opus_4.7_xhigh_run_2 · 4233333333333333
claude_sonnet_4.6_with_playwright_high_run_2 · 4233333333333333
claude_opus_4.7_xhigh_run_6 · 4233333333333333
claude_opus_4.7_high_run_6 · 4233333333333333
claude_opus_4.6_high_abridged_prompt_run_2 · 4233333333333333
claude_opus_4.7_high · 4233333333333333
claude_opus_4.7_with_playwright_high_run_2 · 4233333333333333
claude_opus_4.7_xhigh · 4233333333333333
claude_opus_4.7_xhigh_run_5 · 4233333333333333
claude_opus_4.6_high_abridged_prompt · 4233333333333333
claude_opus_4.7_with_playwright_high · 4233333333333333
claude_opus_4.7_xhigh_run_3 · 4233333333333333
claude_opus_4.7_xhigh_run_4 · 4233333333333333
claude_opus_4.7_with_playwright_high_run_3 · 4233333333333333
claude_opus_4.7_with_playwright_xhigh_run_3 · 4233333333333333
claude_opus_4.7_with_playwright_xhigh_run_5 · 4233333333333333
claude_opus_4.7_xhigh_with_antigravity_prompt_run_2 · 4233333333333333
claude_opus_4.7_xhigh_with_antigravity_prompt_run_4 · 4233333333333333
claude_opus_4.7_xhigh_abridged_prompt · 4233333333333333
claude_opus_4.7_xhigh_with_antigravity_prompt_run_5 · 4233333333333333
claude_opus_4.7_with_playwright_xhigh · 4233333333333333
claude_opus_4.7_xhigh_with_antigravity_prompt · 4233333333333333
claude_opus_4.7_with_playwright_xhigh_run_6 · 4233333333333333
claude_opus_4.7_with_playwright_xhigh_run_2 · 4233333333333333
claude_opus_4.7_high_abridged_prompt_run_2 · 4233333333333333
claude_opus_4.7_with_playwright_xhigh_run_4 · 4233333333333333
antigravity_sonnet_4.6 · 4233333333333333
claude_sonnet_4.6_high_run_2 · 4123333333333333
claude_sonnet_4.6_high_with_antigravity_prompt · 4132333333333333
claude_opus_4.6_high_run_6 · 4132333333333333
claude_opus_4.6_high_run_4 · 4132333333333333
claude_opus_4.7_high_run_3 · 4123333333333333
claude_sonnet_4.6_high_with_antigravity_prompt_run_2 · 4132333333333333
claude_opus_4.6_high_abridged_prompt_run_4 · 4132333333333333
claude_opus_4.6_high_with_antigravity_prompt_run_5 · 4132333333333333
claude_opus_4.6_high_abridged_prompt_run_6 · 4132333333333333
claude_opus_4.6_high_with_antigravity_prompt_run_6 · 4132333333333333
claude_opus_4.6_high_run_2 · 4132333333333333
claude_sonnet_4.6_max_with_antigravity_prompt · 4133333323333333
claude_opus_4.7_high_run_4 · 4123333333333333
claude_opus_4.6_high_with_antigravity_prompt · 4132333333333333
claude_opus_4.7_high_run_2 · 4123333333333333
claude_opus_4.7_high_with_antigravity_prompt_run_6 · 4123333333333333
claude_opus_4.7_xhigh_abridged_prompt_run_4 · 4133333323333333
claude_opus_4.6_with_playwright_high_run_6 · 4132333333333333
claude_opus_4.6_with_playwright_high_run_5 · 4133233333333333
claude_opus_4.7_high_abridged_prompt_run_5 · 4133333323333333
claude_opus_4.6_with_playwright_high_run_4 · 4132333333333333
claude_opus_4.6_with_playwright_high_run_3 · 4132333333333333
claude_opus_4.7_high_with_antigravity_prompt · 4123333333333333
claude_opus_4.7_high_abridged_prompt_run_3 · 4123333333333333
claude_opus_4.7_high_with_antigravity_prompt_run_2 · 4132333333333333
claude_opus_4.7_with_playwright_high_run_4 · 4132333333333333
claude_opus_4.7_high_abridged_prompt_run_4 · 4132333333333333
claude_opus_4.7_high_with_antigravity_prompt_run_3 · 4132333333333333
claude_opus_4.6_with_playwright_high_run_2 · 4132333333333333
claude_opus_4.7_xhigh_abridged_prompt_run_6 · 4132333333333333
claude_opus_4.7_with_playwright_high_run_6 · 4132333333333333
claude_opus_4.7_xhigh_abridged_prompt_run_3 · 4133333323333333
claude_opus_4.7_with_playwright_high_run_5 · 4132333333333333
claude_opus_4.7_xhigh_with_antigravity_prompt_run_6 · 4133333323333333
claude_opus_4.6_high · 4132333333333333
claude_opus_4.7_xhigh_abridged_prompt_run_2 · 4132333333333333
claude_opus_4.6_high_run_3 · 4022333333333333
claude_opus_4.6_high_with_antigravity_prompt_run_2 · 4032333332333333
claude_opus_4.6_high_with_antigravity_prompt_run_3 · 4032333323333333
claude_opus_4.7_high_abridged_prompt · 4033333322333333
claude_opus_4.7_xhigh_abridged_prompt_run_5 · 4022333333333333
claude_opus_4.7_high_abridged_prompt_run_6 · 4023333323333333
claude_opus_4.6_with_playwright_high · 4032333323333333
claude_opus_4.6_high_with_antigravity_prompt_run_4 · 4032333233333333
claude_opus_4.7_high_with_antigravity_prompt_run_5 · 4023323333333333
claude_opus_4.7_xhigh_with_antigravity_prompt_run_3 · 4032333323333333
antigravity_gemini_3.1_pro_high · 4032333333333323
antigravity_gemini_3.1_pro_low · 4032333333333332
antigravity_opus_4.6 · 4032333323333333
claude_sonnet_4.6_high · 3932333333223333
claude_opus_4.6_with_playwright_high_run_7 · 3932333313333333
claude_opus_4.6_high_abridged_prompt_run_5 · 3932333223333333
claude_opus_4.7_high_run_5 · 3923333333333133
claude_opus_4.7_high_with_antigravity_prompt_run_4 · 3923323323333333
claude_sonnet_4.6_with_playwright_high · 3932333331333333
claude_opus_4.6_high_run_1 · 3832333313233333
antigravity_gemini3_flash · 3823223333333323
claude_qwen_3.6_high_with_playwright · 3732322322333333
claude_qwen_coder_next_high_with_playwright · 2422321211111331
Key Findings

What the data shows

Capability tier dominates

Frontier models cluster near the 42 ceiling (family means ≈ 41); the two local Qwen models score 37 and 24, with token-volume estimates twelve to fifty times the frontier median. The tier gap dwarfs anything tools, prompt, or effort do within a tier (≤ 1–2 points on the total).

Totals conceal first-try reliability

Docker deployment failed first try in 40 of 90 runs (44%). Its first-try pass rate rose from 19% on Opus 4.6 High to 75% on Opus 4.7 High while family means moved less than a point; local-environment failures moved the other way (1 of 26 → 10 of 24).

The tool adds cost, not reliability

Playwright on vs off leaves the functional score flat while raising median cost +42% / +68% (Opus 4.7 High / xHigh; +27% on Opus 4.6). Every tool-enabled run invoked it; the premium is context re-reading (cache-read tokens 2.3M → 5.3M–7.0M), and tool-enabled runs still failed on Docker builds.

Effort buys first-try reliability

Opus 4.7 High→xHigh: first-try-perfect rises 28% → 89% (5 of 18 → 16 of 18), corrective prompts fall 16 → 3, for 9–29% more cost. The reliability the tool didn't deliver, effort did.

The design prompt changes what is built, not how well it works

None of the 45 unprompted Claude Code runs shipped a gradient, a keyframe animation or an imported display font; all 40 prompted runs shipped all of them, and every one was rated 4 or 5 against 3 for every unprompted run. Functional totals were unchanged, but first-try reliability fell, concentrated on drag-and-drop, and the cost persists at xHigh. A one-paragraph paraphrase reproduces the whole effect.

Variability is effort-sensitive

Seven identical Opus 4.6 High base runs scatter 38–42 functionally and 41–626 lines of CSS; on Opus 4.7 the structural spread narrows on its own while scores still scatter 39–42 at High, and xHigh compresses them to a uniform 42.

Docker & npm are the dominant failures

better-sqlite3 native builds in minimal containers and the Express 5 change that removed the app.get('*') wildcard break most first-run containers. Capability and effort catch them; a browser-testing tool doesn't.

Design Treatment

The design prompt, not compute, is what invokes design

Two independent measures over the 85 Claude Code runs. The source-based measures count, in each run's shipped code, the features the design directive names and technique-agnostic styling effort (analysis/design_markers.csv, no judge involved). The holistic rating (1–5) was produced by a large language model from the archived dashboard screenshots, reviewed by the author, and not blinded to condition (analysis/VISUAL_RATING_INSTRUMENT.md). The two agree in every Claude Code run. Neither depends on effort or the testing tool: every base and +Playwright run at either effort level sits at 3.

MeasureNo design prompt (45)Full design prompt (22)Abridged prompt (18)
CSS gradient (runs with ≥ 1)0 / 4522 / 2218 / 18
Keyframe animation (runs with ≥ 1)0 / 4522 / 2218 / 18
Google Fonts import (runs with ≥ 1)0 / 4522 / 2218 / 18
Named display font (runs with ≥ 1)0 / 4522 / 2218 / 18
CSS custom properties (median)103330
CSS lines (median)330967956
box-shadow rules (median)31215
transform rules (median)01418
Holistic visual rating 4 or 50 / 4522 / 2218 / 18
Holistic visual rating 3 or below45 / 450 / 220 / 18
With the full design prompt
Opus 4.7★★★★★
Opus 4.7★★★★★
Opus 4.6★★★★★
With the abridged (one-paragraph) design prompt
Opus 4.7★★★★★
Opus 4.7★★★★★
Opus 4.6★★★★☆
No design prompt (default)
Opus 4.7★★★☆☆
Opus 4.7★★★☆☆
Opus 4.7★★★☆☆
Cost Efficiency

Score vs cost

Final session cost (USD) after all fixes, low to high; bar scaled to the most expensive hosted-model run. The two off-scale Qwen runs (local models; their /cost figures are token-volume estimates, not prices) are omitted here, and the five Antigravity runs have no recorded cost.

claude_sonnet_4.6_high_run_3
$1.08 · 42
claude_sonnet_4.6_high_run_2
$1.40 · 41
claude_sonnet_4.6_high
$1.58 · 39
claude_sonnet_4.6_high_with_antigravity_prompt
$1.69 · 41
claude_sonnet_4.6_high_with_antigravity_prompt_run_3
$1.82 · 42
claude_opus_4.6_high_run_6
$2.04 · 41
claude_opus_4.6_with_playwright_high_run_7
$2.08 · 39
claude_opus_4.6_high_run_4
$2.09 · 41
claude_opus_4.6_high_abridged_prompt_run_3
$2.10 · 42
claude_opus_4.7_high_run_3
$2.22 · 41
claude_sonnet_4.6_high_with_antigravity_prompt_run_2
$2.27 · 41
claude_opus_4.6_high_abridged_prompt_run_4
$2.31 · 41
claude_opus_4.6_high_with_antigravity_prompt_run_5
$2.34 · 41
claude_opus_4.6_high_run_5
$2.54 · 42
claude_opus_4.7_xhigh_run_2
$2.54 · 42
claude_sonnet_4.6_with_playwright_high_run_2
$2.57 · 42
claude_opus_4.6_high_abridged_prompt_run_6
$2.60 · 41
claude_opus_4.6_high_with_antigravity_prompt_run_6
$2.72 · 41
claude_opus_4.6_high_run_3
$2.74 · 40
claude_opus_4.6_high_run_2
$2.77 · 41
claude_opus_4.7_xhigh_run_6
$2.77 · 42
claude_opus_4.7_high_run_6
$2.79 · 42
claude_sonnet_4.6_max_with_antigravity_prompt
$2.90 · 41
claude_opus_4.6_high_abridged_prompt_run_2
$2.94 · 42
claude_opus_4.7_high_run_4
$2.99 · 41
claude_opus_4.6_high_with_antigravity_prompt_run_2
$3.11 · 40
claude_opus_4.6_high_with_antigravity_prompt
$3.12 · 41
claude_opus_4.7_high_run_2
$3.13 · 41
claude_opus_4.7_high
$3.15 · 42
claude_opus_4.7_high_with_antigravity_prompt_run_6
$3.15 · 41
claude_opus_4.7_xhigh_abridged_prompt_run_4
$3.15 · 41
claude_opus_4.6_with_playwright_high_run_6
$3.19 · 41
claude_opus_4.7_with_playwright_high_run_2
$3.26 · 42
claude_opus_4.6_with_playwright_high_run_5
$3.30 · 41
claude_opus_4.7_xhigh
$3.31 · 42
claude_opus_4.7_high_abridged_prompt_run_5
$3.33 · 41
claude_opus_4.7_xhigh_run_5
$3.36 · 42
claude_opus_4.6_high_abridged_prompt
$3.45 · 42
claude_opus_4.6_with_playwright_high_run_4
$3.47 · 41
claude_opus_4.6_high_with_antigravity_prompt_run_3
$3.51 · 40
claude_opus_4.7_high_abridged_prompt
$3.55 · 40
claude_opus_4.6_with_playwright_high_run_3
$3.56 · 41
claude_opus_4.6_high_abridged_prompt_run_5
$3.59 · 39
claude_opus_4.7_high_with_antigravity_prompt
$3.64 · 41
claude_opus_4.7_high_abridged_prompt_run_3
$3.96 · 41
claude_opus_4.7_high_run_5
$3.96 · 39
claude_opus_4.7_with_playwright_high
$4.01 · 42
claude_opus_4.7_high_with_antigravity_prompt_run_2
$4.08 · 41
claude_opus_4.7_xhigh_run_3
$4.12 · 42
claude_opus_4.7_with_playwright_high_run_4
$4.22 · 41
claude_opus_4.7_xhigh_run_4
$4.22 · 42
claude_opus_4.7_high_abridged_prompt_run_4
$4.32 · 41
claude_opus_4.7_with_playwright_high_run_3
$4.47 · 42
claude_opus_4.7_high_with_antigravity_prompt_run_3
$4.48 · 41
claude_opus_4.6_with_playwright_high_run_2
$4.49 · 41
claude_opus_4.7_with_playwright_xhigh_run_3
$4.51 · 42
claude_opus_4.7_with_playwright_xhigh_run_5
$4.56 · 42
claude_opus_4.7_xhigh_abridged_prompt_run_6
$4.60 · 41
claude_opus_4.7_xhigh_abridged_prompt_run_5
$4.64 · 40
claude_opus_4.7_high_abridged_prompt_run_6
$4.66 · 40
claude_opus_4.7_high_with_antigravity_prompt_run_4
$4.71 · 39
claude_opus_4.7_xhigh_with_antigravity_prompt_run_2
$4.77 · 42
claude_opus_4.7_xhigh_with_antigravity_prompt_run_4
$4.80 · 42
claude_opus_4.7_xhigh_abridged_prompt
$4.83 · 42
claude_opus_4.7_with_playwright_high_run_6
$4.88 · 41
claude_opus_4.7_xhigh_abridged_prompt_run_3
$4.89 · 41
claude_sonnet_4.6_with_playwright_high
$4.90 · 39
claude_opus_4.7_xhigh_with_antigravity_prompt_run_5
$4.99 · 42
claude_opus_4.7_with_playwright_high_run_5
$5.03 · 41
claude_opus_4.7_xhigh_with_antigravity_prompt_run_6
$5.34 · 41
claude_opus_4.6_with_playwright_high
$5.36 · 40
claude_opus_4.7_with_playwright_xhigh
$5.57 · 42
claude_opus_4.7_xhigh_with_antigravity_prompt
$5.59 · 42
claude_opus_4.6_high
$5.61 · 41
claude_opus_4.7_with_playwright_xhigh_run_6
$5.61 · 42
claude_opus_4.7_xhigh_abridged_prompt_run_2
$5.98 · 41
claude_opus_4.6_high_with_antigravity_prompt_run_4
$6.08 · 40
claude_opus_4.7_with_playwright_xhigh_run_2
$6.16 · 42
claude_opus_4.7_high_with_antigravity_prompt_run_5
$6.57 · 40
claude_opus_4.6_high_run_1
$6.62 · 38
claude_opus_4.7_high_abridged_prompt_run_2
$7.21 · 42
claude_opus_4.7_xhigh_with_antigravity_prompt_run_3
$7.25 · 40
claude_opus_4.7_with_playwright_xhigh_run_4
$7.63 · 42
Recommendations

What to use

For reliable first-shot results

A frontier model at higher effort — effort, not a UI-testing tool, is what removes first-run failures.

Match resource to failure mode

Don't add a browser-testing tool expecting reliability; the dominant failures are build/environment faults it can't see. Spend on capability and effort, or on a tool aimed at the build.

Ask for visual quality explicitly

A paragraph of project instructions is enough to invoke it — and budget for the small first-try-reliability price it carries in the most interaction-heavy features.