Does CLAUDE.md actually help?

We ran the same Signalboard prompt twice from the same starting commit and the same model — once with no CLAUDE.md, once with a root CLAUDE.md file. Here is what came back both times.

Desktop capture of the Signalboard dashboard Claude Code produced without CLAUDE.md: dark sidebar, four summary cards, a signals-by-type bar chart, recent activity list, and a status breakdown. — Claude CodeWithout CLAUDE.mdOpen full capture

Desktop capture of the Signalboard dashboard Claude Code produced with a root CLAUDE.md: light sidebar, four summary cards, a signals-by-type bar chart, and a recent signals list of tagged cards. — Claude CodeWith CLAUDE.mdOpen full capture

What happened

Both conditions passed the same 10 deterministic checks and failed the same 7, out of 17 total. CLAUDE.md produced no measured improvement on this deterministic score in this single controlled run. The two applications look different — different sidebar treatment, different layout choices — but the grader could not tell them apart on outcome.

What this tells us

This is not evidence that CLAUDE.md never helps. It is one paired run, on one task, under one grader. What changed between the two runs is recorded below, not attributed — a single paired run cannot establish that CLAUDE.md caused the duration or token differences either.

What changed

One run per condition cannot attribute these to CLAUDE.md. They are recorded, not explained.

Metric	Without CLAUDE.md	With CLAUDE.md
Deterministic checks passed	10/17	10/17
Agent duration	478.8 seconds	409.2 seconds
Output tokens	33,769	30,142

Duration dropped 69.6s and output tokens dropped 3,627 in the CLAUDE.md run; the checks passed and failed did not change.

Without CLAUDE.md · 7 failed

customer_signal_records_exist
create_signal_works
derived_summary_statistic_updates
local_persistence_survives_reload
edit_signal_works
search_or_filter_works
delete_signal_works

With CLAUDE.md · 7 failed

customer_signal_records_exist
create_signal_works
derived_summary_statistic_updates
local_persistence_survives_reload
edit_signal_works
search_or_filter_works
delete_signal_works

The task

Read the exact prompt (770 characters, sent once per run)

Both runs used prompt SHA-256 139c5d30e5f71c11fc50594fea951423e1e86e5e1ef7ffbb1147d91919c57825 and started from commit dea92b54f8e10b629887361513c4384b92b82b7c.

Build a responsive SaaS-style Next.js application called Signalboard.

Signalboard is a simple workspace for tracking customer signals and feedback.

Requirements:

* Next.js
* TypeScript
* responsive desktop and mobile UI
* navigation
* dashboard
* list of customer-signal records
* create a new signal
* edit an existing signal
* delete a signal
* search or filter signals
* display at least one useful summary statistic derived from the current records
* local or mock persistence sufficient for the application to function
* accessible controls
* successful production build
* no material browser console errors
* do not use paid external APIs
* do not require production authentication

Do not ask questions.

Make reasonable product and design decisions yourself.

Evidence & setup

Held constant vs. changed

Task: Identical prompt bytes
Starting state: dea92b54f8e10b629887361513c4384b92b82b7c
Configured model: claude-sonnet-4-6
Only difference: A root CLAUDE.md file

Per-check comparison · 17 checks

Deterministic check	Without CLAUDE.md	With CLAUDE.md
Production Build Success	PASS	PASS
Typescript Success	PASS	PASS
Page Loads	PASS	PASS
Signalboard Product Identity	PASS	PASS
Navigation And Dashboard Exist	PASS	PASS
Customer Signal Records Exist	FAIL	FAIL
Accessible Controls	PASS	PASS
Desktop Renders	PASS	PASS
Create Signal Works	FAIL	FAIL
Derived Summary Statistic Updates	FAIL	FAIL
Local Persistence Survives Reload	FAIL	FAIL
Edit Signal Works	FAIL	FAIL
Search Or Filter Works	FAIL	FAIL
Delete Signal Works	FAIL	FAIL
Mobile No Material Horizontal Overflow	PASS	PASS
Mobile Renders	PASS	PASS
No Material Browser Console Errors	PASS	PASS

The exact CLAUDE.md

The complete treatment file, verbatim. This is the only declared difference between the two launches.

# Signalboard implementation guidance

- Follow the existing Next.js and TypeScript project conventions.
- Keep the architecture and dependencies simple; prefer a direct, maintainable implementation over unnecessary abstraction.
- Build a coherent responsive workspace experience for desktop and mobile.
- Use semantic HTML, labelled controls, visible focus states, and keyboard-usable interactions.
- Keep records and derived dashboard summaries in one consistent state model, with local persistence where appropriate.
- Before finishing, run the available typecheck and production build commands and resolve any failures.

Models, settings, and usage

Without CLAUDE.md

CLI: Claude Code 2.1.141
Configured model: claude-sonnet-4-6
CLI-exposed runtime model(s): claude-haiku-4-5-20251001, claude-sonnet-4-6
Runtime identity status: multiple runtime models confirmed by CLI output
Agent duration: 478.8 seconds
Exit code: 0

With CLAUDE.md

CLI: Claude Code 2.1.141
Configured model: claude-sonnet-4-6
CLI-exposed runtime model(s): claude-haiku-4-5-20251001, claude-sonnet-4-6
Runtime identity status: multiple runtime models confirmed by CLI output
Agent duration: 409.2 seconds
Exit code: 0

Usage comparison

Baseline output: 33,769 tokens
Treatment output: 30,142 tokens
Baseline cost: $1.0621
Treatment cost: $0.8693

Methodology

Two fresh nonresumed Claude Code executions began from the same fixture commit and received identical prompt bytes, exact model configuration, execution settings, and deterministic grader. Condition A is the one baseline capability run. Condition B is non-baseline; its only intended treatment variable was the preserved root CLAUDE.md.

Documented limitations · 4

This is one task with one execution per condition; observed differences cannot establish general or statistically reliable effects.
The deterministic grader tests specified functionality and structural accessibility heuristics, not overall visual quality or every valid implementation pattern.
Configured model identity and CLI-exposed runtime model identity are recorded separately; multiple runtime model identifiers, if exposed, are preserved without inventing a primary model.
Both processes ran under the same macOS user without OS-level filesystem isolation, although worktrees were sequential and no sibling workspace or run evidence path was supplied.

See the head-to-head next. See Claude vs Codex →