Does a Frontend Skill Actually Make Codex Better?
Both builds passed 25/25 deterministic checks. The frontend skill changed Codex's architecture substantially, took about 29% longer, and did not produce the build a non-blinded human reviewer preferred.
Published
The task
Read the exact prompt
Replace the starter page with a responsive "Upgrade Team Plan" flow for a fictional SaaS product called Relayboard. Requirements: - A plan comparison step with at least three distinct tiers (e.g. Starter, Growth, Scale). Each tier must show its name, a one-line summary, a per-seat monthly price, and a recommended-seat range or a one-line "best for" callout. - A seat-count step with a working stepper (decrement, increment, and direct numeric input). The total monthly price must update live as the seat count changes. The total must be derived from the currently selected plan's per-seat price and the current seat count, not stored separately. - A review/summary step that shows the selected plan, the selected seat count, and the computed monthly total, with a clear primary "Confirm upgrade" action. - A confirmation state after confirm that shows the chosen plan, seat count, and total, and provides an explicit "Start over" or equivalent action that returns the user to the plan comparison step with no stale state. - The flow must be usable on desktop and mobile. On a 390-pixel-wide viewport, every screen of the flow must fit without horizontal scroll. - All interactive controls must be real buttons, inputs, selects, or labels. No divs that act as buttons. - Native HTML form attributes (required, min, max, type) for the first line of validation. Field-level error messages under invalid fields. - A single live region announces the seat count and total to assistive technology when they change. - One page route, one h1, one h2 per step, consistent spacing, and at least 4.5:1 text contrast against the page background. - Production build must succeed. TypeScript must typecheck. No material browser console errors. No new external dependencies. Do not ask questions. Make reasonable product, copy, and design decisions yourself. Use the existing Next.js and TypeScript setup.
What Codex Built With and Without the Frontend Implementation Skill
Both arms passed all 25 deterministic checks: production build, TypeScript, no console errors, no horizontal overflow at 390px, and the full plan-select-review-confirm-reset interaction sequence.
Without the skill, Codex implemented the entire Upgrade Team Plan flow in one largely monolithic 263-line app/page.tsx, touching only app/globals.css, app/layout.tsx, and app/page.tsx.
With the skill, Codex produced a 108-line app/page.tsx plus five dedicated step components (PlanStep, SeatStep, ReviewStep, ConfirmationStep, ProgressSteps), a typed app/_data/plans.ts, and a shared app/lib/format.ts currency helper — closely following the skill's prescribed component-decomposition and 'one helper for formatCurrency' guidance.
The no-skill run finished in about 7.7 minutes; the skill-treatment run took about 10.0 minutes, roughly 29% longer for the same passing outcome.
A non-blinded human visual review of both frozen builds preferred the no-skill output overall.
Architecture, Runtime, and Visual Differences Between the Two Codex Builds
| Difference | Observed |
|---|---|
| Deterministic correctness | No difference: both arms passed 25/25 identical checks with zero failures and zero console errors. |
| Code architecture | The skill changed Codex's implementation approach markedly — from one large page component to a decomposed set of step components, typed plan data, and a shared formatting helper, matching the skill's explicit guidance almost point for point. |
| Runtime | The skill-treatment run took about 29% longer (10.0 vs 7.7 minutes) to reach the same passing result. |
| Human visual preference | In non-blinded review, the reviewer preferred the no-skill build's visual output overall. This reflects editorial judgment made with knowledge of which arm had the skill, not a blinded score. |
When a Frontend Skill Is Worth the Extra Time on a Codex Build
On this one test, the frontend implementation skill did not produce a measurably more correct build: both arms passed all 25 automated checks. It did substantially change how Codex organized the code, moving it from one large page component toward the decomposed, typed, helper-based structure the skill prescribes. That architectural change cost about 29% more runtime and did not yield the build a human reviewer preferred in this non-blinded look. This is one paired run on one task with one skill and one Codex configuration; it does not show that skills generally help or hurt Codex, and it does not establish that the skill made Codex worse overall — only that it changed the tradeoff on this run.
Relayboard Upgrade-Flow Task, Skill Treatment, and Grading Setup
Setup
{
"participants": [
"Codex — no skill (baseline)",
"Codex — with frontend implementation skill (treatment)"
],
"model": "Codex CLI 0.150.0, model gpt-5.6-sol, reasoning effort high",
"held_constant": [
"identical starting fixture commit (969bd1ddbf725fc0583a2fe55cc71f992317a88a)",
"identical task prompt (sha256 42e63df2...)",
"fresh, non-resumed Codex sessions for both arms",
"identical sandbox flags: workspace-write, --ephemeral, --ignore-user-config, --ignore-rules",
"one contestant run per arm, no retries or repair passes",
"same deterministic grader run against both frozen source trees"
],
"treatment_variable": "Presence of the declared frontend-implementation skill (isolated CODEX_HOME containing only auth, built-in .system skills, and the one declared skill for the treatment arm; no skill for the baseline arm)",
"automated_validation": "Both arms passed 25/25 checks (build, typecheck, screenshots, overflow, console errors, and the full plan → seats → review → confirmation → reset interaction sequence)",
"post_run_repairs_to_source": 0,
"post_run_grader_screenshot_repair": true,
"cross_contamination": false
}Supporting evidence
{
"publication_quality_gate": "Standard case study: PASS",
"automated_checks": {
"no_skill": "25/25 passed",
"with_skill": "25/25 passed"
},
"runtime": {
"no_skill_minutes": 7.7,
"with_skill_minutes": 10,
"with_skill_pct_slower": 29
},
"architecture": {
"no_skill": "One largely monolithic app/page.tsx (263 lines); only app/globals.css, app/layout.tsx, app/page.tsx changed.",
"with_skill": "app/page.tsx reduced to 108 lines; five new step components under app/_components/, typed plan data in app/_data/plans.ts, and a shared app/lib/format.ts currency helper, closely matching the skill's prescribed decomposition."
},
"visual_review": "Non-blinded; reviewer preferred the no-skill build overall.",
"evidence_limitation": "Original per-step screenshots had a capture-timing bug and were recaptured post hoc from the unmodified frozen contestant source trees; no contestant was rerun. Full record: runs/2026-09-01-codex-frontend-skill-vs-no-skill/post-run-recapture/PROVENANCE.md.",
"notes": [
"This is one contestant run per arm, per Hafiz's preference for agent-experiment publications.",
"No invented product metrics were introduced by either arm beyond what the prompt specified."
]
}Provenance
{
"source_run": "runs/2026-09-01-codex-frontend-skill-vs-no-skill",
"source_results": "runs/2026-09-01-codex-frontend-skill-vs-no-skill/results.json",
"source_publication_analysis": "runs/2026-09-01-codex-frontend-skill-vs-no-skill/publication.json",
"source_results_sha256": "b7b5f2ba0f60ac499fb356334671b7226a10b47be5eefca4ccdc319dab0c3945"
}Documented limitations · 5
- One paired run: one task, one skill, one Codex model and configuration. Does not generalize to Codex skills broadly or to other tasks.
- The original per-step screenshot capture had a timing bug that made all step screenshots within an arm byte-identical (photographing a post-reset state). The intermediate step screenshots shown here were recaptured after the run from the same frozen, unmodified contestant source trees using a fixed grader. Neither contestant was rerun and neither source tree was modified; see runs/2026-09-01-codex-frontend-skill-vs-no-skill/post-run-recapture/PROVENANCE.md for the full repair record and before/after hashes.
- The visual comparison was not blinded: the reviewer knew which build came from which arm, so the preference is editorial judgment, not a blinded experimental score.
- Runtime figures (7.7 and 10.0 minutes) are rounded from the recorded run durations (464,132 ms and 599,809 ms) and reflect one invocation per arm, not an average over repeated runs.
- Deterministic checks cover build success, typechecking, absence of console errors, layout overflow, and the scripted interaction sequence; they do not measure code quality, maintainability, or design taste, which is why this run also includes non-blinded qualitative review.
See more experiments. Claude vs Codex on Case-Study Design →