CodeStories vs DeepWiki · Omoju Miller

A repository told as a story that follows its data through one real run, every claim linked to pinned lines and every chapter backed by a proof that runs, against DeepWiki's generated wiki for the same ten Python libraries. The rule for deciding was written and committed before any result existed.

the preregistered rule (§8)

kill

the accountability claim did not hold.

✗ H1: story density <= 1/2 DeepWiki
✓ general: story >= DeepWiki - 5
✓ path: story >= DeepWiki + 10

Omoju, after reading and before any metric (§9)

ship

One GET, End to End, and it made sense to me.

What Actually Reaches the Terminal, this was something I wasn't farmiliar with at all. I was able to follow it, and understand how it works. What made it good was being able to click on each thing in the control flow diagram, and then jump into where that was in the code base. In my mind, I could follow that data through the system. SHIP

What "it helps" would feel like, written before the runs: I get a reasonable understanding of the code base, I know what the critical paths are, I know where the security boundaries are.

the measures

Nine repositories out of sample (itsdangerous was used to build the pipeline and is reported but not counted). H1 is accountability: contradicted claims per 1,000 words, from a blind judge reading 40 sampled claims per arm per repository against the code at the commit each arm describes. H2 is comprehension: a reader model given only one arm's text answers questions written from the code and the run, never from either arm, with keys checked by execution.

contradicted claims / 1k words

0.71story

0.54deepwiki

ratio story/deepwiki, 95% CI 0.55–4.86 · rule: ≤ 0.5

path questions (this run)

98%story

63%deepwiki

difference, 95% CI 23 to 47 points · rule: ≥ +10

general questions (whole repo)

76%story

56%deepwiki

difference, 95% CI 5 to 33 points · rule: ≥ −5

Intervals resample whole repositories (10,000 draws, fixed seed). The decision uses the point estimates; where an interval crosses a threshold, it says so here. How each measure was produced, step by step: the accuracy measure and the comprehension measure.

Per repository

repo	arm	words	claims	contradicted	unverifiable	per 1k words	path	general	Jev agrees
attrs	story	9,375	237	2/40	0	1.26	100%	71%	38/40
attrs	deepwiki	14,140	363	0/40	8	0.00	56%	36%	39/40
click	story	8,601	248	1/40	1	0.72	94%	71%	39/40
click	deepwiki	33,482	558	1/40	8	0.42	75%	36%	37/40
flask	story	7,261	192	0/40	2	0.00	94%	42%	40/40
flask	deepwiki	28,535	687	0/40	7	0.00	75%	58%	39/40
httpx	story	8,108	227	1/40	0	0.70	94%	79%	40/40
httpx	deepwiki	27,567	725	0/40	5	0.00	69%	50%	40/40
itsdangerous*	story	7,707	123	0/40	0	0.00	94%	92%	39/40
itsdangerous*	deepwiki	12,089	291	2/40	0	1.20	94%	100%	37/40
jinja	story	8,842	210	0/40	0	0.00	100%	57%	40/40
jinja	deepwiki	15,506	473	1/40	5	0.76	50%	79%	40/40
markupsafe	story	5,961	97	2/40	1	0.81	100%	92%	38/40
markupsafe	deepwiki	31,855	312	3/40	1	0.73	94%	75%	40/40
requests	story	10,347	277	0/40	0	0.00	100%	100%	39/40
requests	deepwiki	22,131	489	2/40	6	1.10	64%	71%	38/40
rich	story	9,560	300	3/40	6	2.35	100%	93%	37/40
rich	deepwiki	25,426	1876	0/40	7	0.00	38%	64%	40/40
tqdm	story	8,751	177	1/40	0	0.51	100%	79%	40/40
tqdm	deepwiki	17,075	455	0/40	2	0.00	44%	36%	39/40

* in sample. "Jev agrees": a second, cheaper model's verdict on the same claim and the lines the judge cited.

robustness

Every story was generated by the same pipeline with no hand edits. A chapter that failed a check (a citation to lines that don't exist, a proof that doesn't pass) went back to the model with the verdict, at most twice. "Refs resolve" is the share of DeepWiki's own [path:lines]() references that point at lines existing at the commit its pages are pinned to.

repo	chapters	citations	repairs	still failing	cost	time	DeepWiki words	refs resolve
attrs	9	212	1	0	$5.20	20 min	17,699	94%
click	9	227	1	0	$4.44	16 min	52,198	90%
flask	8	138	2	0	$3.86	13 min	36,447	95%
httpx	8	210	0	0	$3.34	9 min	38,220	94%
itsdangerous*	8	129	—	—	$1.85	7 min	17,655	79%
jinja	8	190	1	0	$4.42	17 min	19,978	96%
markupsafe	6	104	2	0	$2.31	11 min	45,504	91%
requests	10	241	0	0	$4.30	14 min	29,267	98%
rich	9	261	1	0	$5.29	20 min	62,473	96%
tqdm	9	177	1	0	$4.78	16 min	21,545	99%

* in sample: the itsdangerous story was written while the pipeline was being built, before the repair loop existed, and two of its proofs were fixed by hand.

The nine out-of-sample stories: $37.94 in model calls. The evaluation itself (questions, extraction, judging, reading, grading) is in eval/ with every model response saved.

was the judge trustworthy?

Before any comparison ran, the judge was tested on planted lies: 10 true claims per arm from the in-sample repository, each with a false twin that changes one specific, judged blind among the truths. Story claims: 10/10 lies caught, 0/10 truths wrongly flagged. DeepWiki claims: 10/10 caught, 0/10 wrongly flagged. The passing bar (every lie caught, at most one false alarm per arm) was written into the test before it ran.

Omoju hand-checked 20 judged claims with the arm hidden: agreed with the judge on 10/10 (story) and 8/10 (deepwiki).

the judge was run twice

The first judging pass could not see most of the source in the large repositories (the repository block led with docs and ran out of room), and said so: it called 243 of 360 story claims and 137 of 360 DeepWiki claims unverifiable. Since unverifiable counts as not contradicted in the preregistered metric, that pass did not measure H1. The judge was given the source the claims name and re-run on the same sampled claims; nothing else was re-run. Both passes are published.

pass	story unverifiable	deepwiki unverifiable	story / 1k	deepwiki / 1k	rule said
first (source mostly unseen)	243/360	137/360	0.07	0.69	ship
second (source in view)	10/360	49/360	0.71	0.54	kill

what changed after preregistration

2026-10-05 · Reruns after infrastructure failures (§10). The first launch of the nine-repository run finished the question stage for six repositories, then every claim-extraction call on a DeepWiki text timed out in the API client's stream reader (nine long streams at once). No result had been produced or seen. The client timeout was raised (3600 s, 3 retries) and the run resumed from the finished stages; nothing already written was redone.

2026-10-05 · Claim extraction in chunks for long texts. For rich, the DeepWiki text (62,473 words) yielded more claims than fit in one model output (64k tokens); the JSON was cut off. Extraction now splits a text over 25,000 words at DeepWiki page boundaries and concatenates the claims, with the same prompt. Only rich/DeepWiki is affected; the eight other repositories' extractions were single calls and are unchanged. Decided before any metric was computed or read.

2026-10-05 · The accuracy judge re-run with the source in view (decided after the first results were read). The judge (§5.1.3) reads the repository as the pipeline's cached block, which without a trace leads with the README and docs; for the large repositories the context budget ran out before the source (rich: 1 of ~80 modules shown; attrs without `_make.py`; requests without `models.py` and `sessions.py`). The first run's "unverifiable" verdicts say so in their reasons. Both arms were judged the same way. Because "unverifiable" counts as not contradicted in the §5.1.4 metric, H1 was not properly measured. Fix: the judge's block now puts the source files the batch's claims name first, then the rest of the package source, then docs. Only the judging stage is re-run, on the same 40 sampled claims per repository per arm; extraction, sampling, questions, answers and grades are untouched. The first run's judgments are kept in `eval/judgments-first-run/` and its results in `eval/results-first-run.md`; both sets of numbers are published. Omoju's §9 verdict was recorded before either. The change can only move claims from unverifiable to supported or contradicted.

read one

The reading page keeps the title, the real-world premise, a control-flow chart of the run drawn from executed lines, and the code beside the narrative. Each chapter's proof is a file you can run. All of them are listed in the gallery.

Preregistration, pipeline and every intermediate file are in the repository. The decision rule was committed before the first result and is quoted here unchanged. Stories are written by Claude; this page is generated from eval/results.json.