What was measured
raw: 10 vs 7 contradicted, of 360 sampled claims each · the rule needed story ≤ ½ × deepwiki
H1 was the core claim of the project: that a story whose every sentence links to pinned lines, with a proof per chapter, would contradict the code far less often than generated documentation. "Far less" was fixed in advance as half the rate or better. The measured rate came out higher, not lower. This explainer shows how the rate was produced, why it came out that way, and what it does and doesn't mean.
A "contradicted claim" is a sentence about the code that the code shows to be false. The rate is per 1,000 words of text, so a long wiki and a short story can be compared. Each piece of that definition is a design decision, and each gets a screen.
Turning prose into checkable claims
You can't check a paragraph. You can check "Response.ok is True if status_code is between 200 and 299." So the first step rewrites each text into atomic claims: one fact each, standing alone, with "it" and "this" resolved. The same prompt did this for both arms. It keeps behaviour, names, values, defaults, conditions and order; it drops narrative, advice, opinion, and anything about tests, CI or project history.
From the requests story
- In `merge_setting`, if either value is not a `Mapping`, the request value is returned as-is with no merging.
- In `merge_setting`, both mappings are flattened with `to_key_val_list` and inserted into `dict_class` with session items first and request items second.
- In `merge_setting`, any key whose merged value is `None` is deleted from the result.
From the requests DeepWiki
- `PreparedRequest.prepare_cookies` adds cookies to the request headers using `get_cookie_header`.
- `Response.json()` returns the JSON-decoded content using `complexjson`.
- `Response.ok` is True if `status_code` is between 200 and 299.
Two numbers come out of this step and both matter later: how many claims a text makes, and how many words it took. The story arm made about 26 claims per 1,000 words, DeepWiki about 28. Stories are denser in checkable statements: a concrete telling has more places to be right or wrong.
Forty per arm per repository
Judging every claim would have meant about 8,000 judgments. The preregistration said: sample up to 40 claims per repository per arm, with a fixed random seed, so the sample is the same on any rerun and nobody can pick it. That's 360 story claims and 360 DeepWiki claims across the nine repositories, 720 in all.
The rate is then scaled back up: if 2 of 40 sampled claims are contradicted and the text made 489 claims in 22,131 words, the estimate is (2 ÷ 40) × (489 ÷ 22,131 × 1,000) = 1.10 contradicted claims per 1,000 words. Screen 7 lets you work that formula.
Sampling has a cost that shows up in screen 8: with 40 draws, the difference between 0 and 2 contradicted is one or two sentences, and the interval around the final ratio is wide.
Who judged, and what it saw
The judge is claude-opus-5 through the API, same as every other role: no tools, no fine-tuning, fresh conversation per batch. It was given the repository's source at the commit the arm describes, and a numbered list of 20 claims, and asked to label each contradicted, supported or unverifiable, citing the lines that decide it.
Blind in two ways. The 80 sampled claims of a repository, both arms, were shuffled together before batching, so a batch mixes story and wiki claims and the judge is never told which is which. And the instruction was symmetric: "a claim that is true but imprecise is supported; a claim that is wrong in any specific is contradicted," whatever the source.
A second model, Jev (TypeSafe's, on Cloudflare: a different vendor), then re-scored each claim from only the claim and the lines Claude cited, as a probability of contradiction. Jev agreed with Claude's label on 703/720 (98%) of the claims it could score. Claude's label decides; Jev is the check on a shared blind spot.
The judge was tested before it was trusted
A judge that calls everything "supported" would make both arms look perfect. So before the comparison, it was tested on known answers: 10 true claims per arm from the in-sample repository, each paired with a false twin, the same sentence with one specific changed. The 40 were shuffled and judged blind. The passing bar was written down first: every lie caught, at most one truth wrongly flagged, per arm.
TWIN In `Serializer.dumps` the payload is wrapped as `self.dump_payload(want_bytes(obj))`. → contradicted
TWIN `URLSafeSerializerMixin.load_payload` delegates to `self.serializer.loads(json)`, bypassing the base `Serializer.load_payload`. → contradicted
TWIN `.pre-commit-config.yaml` enables the hooks `ruff` and `black`. → contradicted
Result: 20/20 lies caught, 0 truths wrongly flagged. That is the evidence that the judge can tell a lie from a truth when it can see the code. It is not evidence that the judge could see the code in every repository, which is the next screen.
The judge was run twice, and the first time it was half blind
The judge reads the repository as a block of files that has to fit a budget. Without a trace to say which files matter, that block led with the README and docs, and in the large repositories the budget ran out before the source: for rich the judge saw 1 of about 80 modules. It did the honest thing and said "unverifiable: that file was not shown." But unverifiable counts as not contradicted in the metric. The first pass found almost nothing wrong with the stories, because it couldn't see the code they were about, and the rule said ship.
The fix put the source files the claims name first, then the rest of the package, then docs; it was checked without the model by listing what each batch would see. The judge re-ran on the same 720 claims; nothing else was re-run. Both passes are published, the change is logged in the preregistration's deviations, and the change can only move claims out of "unverifiable", so it could only make the story look worse or the same. It made it look worse.
Why the story loses on the rate with nearly the same raw count
deepwiki: (7 ÷ 360) × (5,938 ÷ 215,717 × 1000) = 0.54
Ten versus seven wrong sentences is close. The gap in the rate comes from the second factor: the story packs more checkable claims into each thousand words, so the same proportion of wrong claims is more wrong claims per word. The metric was preregistered that way on purpose, as a rate per word of what a reader reads, and it stands. But it is worth seeing what it rewards. Move the slider to change how many claims per 1,000 words the story makes, holding its proportion of wrong claims fixed:
This slider is an illustration of the formula, not a result. The result is the top line.
Every contradicted claim, in full
Here is the whole of H1's evidence: 10 story claims and 7 DeepWiki claims that the second judging pass labelled contradicted, with the judge's reason and Jev's independent probability. Read them; they are the finding.
story 10 contradicted
attrs · Setting the annotations before the build would have made `_transform_attrs` read them as auto-attribs declarations.
judge: `make_class` passes `these=cls_dict`, and `_transform_attrs` takes the `these is not None` branch so annotations are never interpreted as auto-attribs declarations; the complaint would instead come from `from_counting_attr`'s "Type annotation and type argument cannot both be present" check.
lines: src/attr/_make.py:399-403, src/attr/_make.py:441-451, src/attr/_make.py:2600-2605 · Jev: 0.30
attrs · `_generate_unique_filename` appends a counter when the candidate filename is already taken by different source.
judge: `_generate_unique_filename` just formats a string; the collision counter logic lives in `_linecache_and_compile`.
lines: src/attr/_make.py:1634-1641, src/attr/_make.py:245-259 · Jev: 0.42
click · `Option.resolve_envvar_value` returns `None` immediately when no `envvar` is declared.
judge: `Option.resolve_envvar_value` falls back to the `auto_envvar_prefix`-derived variable before returning `None`, so it does not return `None` immediately.
lines: src/click/core.py:3664-3676 · Jev: 0.43
httpx · `Request.__init__` re-wraps the URL with a `copy_with()` call
judge: The URL is constructed via `URL(url)`/`URL(url, params=params)`, not with a `copy_with()` call.
lines: httpx/_models.py:399-399 · Jev: 0.89
markupsafe · The three branch tests in `EscapeFormatter.format_field` are `hasattr` checks.
judge: There are only two `hasattr` tests (`__html_format__` and `__html__`); the third branch is an unconditional `else`.
lines: src/markupsafe/__init__.py:339-353 · Jev: 0.48
markupsafe · The C implementation in `_speedups.c` writes each replacement directly to an output buffer in a single pass.
judge: The C code makes two passes: GET_DELTA first scans the input to compute the size, then DO_ESCAPE writes to the new buffer.
lines: src/markupsafe/_speedups.c:3-16, src/markupsafe/_speedups.c:83-96 · Jev: 0.33
rich · In `Text.render`, the combined active styles are cached by the tuple of style ids.
judge: The cache key is a tuple of resolved `Style` objects (`styles = tuple(style_map[_style_id] ...)`), not a tuple of style ids.
lines: rich/text.py:754-766 · Jev: 0.76
rich · A zero-width joiner sequence counts as one cell under `cell_len`.
judge: In _cell_len a ZWJ merely causes the following character to be skipped, so the joined sequence takes the width of the leading character (2 cells for a wide emoji), not one cell.
lines: rich/cells.py:144-158 · Jev: 0.45
rich · `Text.wrap` calls `divide_line(str(line), width, fold=True)` for each line.
judge: wrap calls divide_line(str(line), width, fold=wrap_overflow == "fold") and only in the non-no_wrap branch, so fold is not always True.
lines: rich/text.py:1233-1240 · Jev: 0.73
tqdm · The first thing `tqdm.__new__` does is `with cls.get_lock():`.
judge: The first statement is `instance = object.__new__(cls)`; the lock is acquired only afterwards.
lines: tqdm/std.py:664-666 · Jev: 0.94
deepwiki 7 contradicted
click · The `Abort` exception is a subclass of `ClickException` and exits with status code 1.
judge: `Abort` subclasses `RuntimeError`, not `ClickException`, and carries no exit code.
lines: src/click/exceptions.py:362-363 · Jev: 0.84
jinja · `PackageLoader` loads templates from a directory within a Python package using `importlib.resources`.
judge: PackageLoader uses importlib.util.find_spec, import_module, zipimport and plain os file access, not importlib.resources.
lines: src/jinja2/loaders.py:302-401 · Jev: 0.77
markupsafe · The publish workflow is triggered on push of tags and on workflow_dispatch.
judge: The publish workflow triggers only on push of tags; there is no workflow_dispatch trigger.
lines: .github/workflows/publish.yaml:2-4 · Jev: 0.80
markupsafe · `Markup.format_map()` is defined at lines 317-323 of `src/markupsafe/__init__.py`.
judge: format_map is defined at lines 327-333, not 317-323.
lines: src/markupsafe/__init__.py:327-333 · Jev: 0.86
markupsafe · The publish workflow artifacts are named `build-sdist` and `build-wheels-{os}`.
judge: The sdist artifact is named "sdist" and wheel artifacts are named "{platform}-{arch|all}", not "build-sdist"/"build-wheels-{os}".
lines: .github/workflows/publish.yaml:26-30, .github/workflows/publish.yaml:77-81 · Jev: 0.94
requests · `Response.ok` is True if `status_code` is between 200 and 299.
judge: `ok` returns True whenever `raise_for_status()` doesn't raise, i.e. for any status code below 400 (including 3xx), not only 200-299.
lines: src/requests/models.py:861-874, src/requests/models.py:1160-1171 · Jev: 0.68
requests · `_parse_content_type_header` sets the value to `True` for parameters without values.
judge: The loop only records parameters containing '=', and never assigns True for value-less parameters.
lines: src/requests/utils.py:560-566 · Jev: 0.79
Two things stand out. The story's errors are small misstatements of mechanism in otherwise accurate paragraphs, and every one sits in a chapter whose citation and proof checks passed: those checks pin lines and values, they don't read the sentence. DeepWiki's errors include two about CI workflow files and one about line numbers, kinds of claim a story never makes.
Could it be the sample?
Ten versus seven, out of 360 each, is a difference of three sentences. The preregistered bootstrap resamples whole repositories (some twice, some not at all), recomputes both rates and their ratio, and repeats. Draw and watch.
The preregistered run (10,000 draws, fixed seed) gave a 95% interval for the ratio of 0.55 to 4.86. It crosses ½ (where the story would have passed) and 1 (where the two arms are equal). Statistically, nine repositories and 40 claims each cannot tell these two arms apart on accuracy. The rule was written to use the point estimate, and the point estimate is 1.3: on the wrong side of ½ by a wide margin. So the rule says kill, and the interval says the honest sentence is "no better," not "worse."
What you can say, and what you can't
The sentence that survives: on nine small Python libraries, a blind model judge reading the code found CodeStories no more accurate than DeepWiki's pages (0.71 vs 0.54 contradicted claims per 1,000 words; 10 vs 7 of 360 sampled; 95% interval on the ratio 0.55–4.86). The checks a story carries, that its lines exist, haven't changed, and its proofs run, did not make its sentences more accurate than generated documentation. The preregistered rule required half the rate; it said kill.
Everything on this page comes from eval/ in the repository: both judging passes, every claim, every verdict, and the judge's test.