Omoju Miller

Is 76% against 56% a real result, and what exactly does it say?

codestories · the comprehension measure (H2) · how it was done

    What was measured

    76%story general questions
    56%deepwiki general questions
    98%story path questions
    63%deepwiki path questions

    Those are percentage scores on a quiz. A reader that has seen only one document about a repository, and never the code, is asked questions about the code. Each answer is marked correct (1), partly (½) or wrong (0). The percentage is the average over all questions, pooled across nine repositories.

    So the claim behind "76% vs 56%" is: a reader given a CodeStory answered owner questions about the repository better than a reader given DeepWiki's wiki for the same repository. "Document" here means one of two texts: the whole CodeStory of a repository, or the whole DeepWiki wiki for it. Whether that is a fair sentence depends on three things: who the reader was, where the questions came from, and how the marking was done. The next screens show each.

    The two arms, same code

    Story

    Written by the CodeStories pipeline: one real run of the repository's intended use is traced, and the story follows the data through it. 6–10 chapters, about 6,000–10,000 words. Every claim links to lines at a pinned commit; each chapter has a proof that runs.

    DeepWiki

    Cognition's generated wiki for the same repository, every page, fetched through its public server on 2 October. Organised by the code's structure. 14,000–33,000 words per repository, with [file:lines]() references.

    Nine repositories count: requests, flask, click, httpx, rich, attrs, jinja, markupsafe, tqdm. A tenth, itsdangerous, was used to build the pipeline and is reported but not counted. For eight of the nine, DeepWiki's pages describe the very same commit the story cites; the two that differ (attrs, markupsafe) are each judged against their own commit.

    The reader for each arm gets the whole document. DeepWiki's is two to four times longer; nothing was trimmed.

    The part that makes it fair: who saw what

    Every role here is the same off-the-shelf model, Anthropic's claude-opus-5, called through the API with adaptive thinking: no fine-tuning, no tools, no agents, and no memory between calls. Each role is a fresh conversation that knows nothing of the others. Click a role to see what it was given and what it was kept from. The rule throughout: nobody who writes a question, answers one, or marks one knows which arm is which, or sees anything that would let them favour one.

    Click a role above.

    Where the questions came from

    A model that never saw either document was given the repository, the intended-use script, and the trace of that run. It wrote 8 path questions (what happens to the data on this run) and 7 general questions (what an owner would ask: guarantees, failure modes, risky defaults), each with a key.

    PATH · requests p8
    Nearby wrong input: if the scenario had passed auth=('user', 'pass', 'extra') instead of a 2-tuple, what would happen during preparation?
    GENERAL · requests g3
    Under what exact conditions does requests drop the Authorization header when following a redirect, and which cross-scheme case is deliberately allowed?
    GENERAL · requests g5
    What retry behaviour does a default HTTPAdapter provide, and what is the difference between max_retries=0 and max_retries=3?

    Keys are checked before anyone reads an arm. A path question's key comes with a small program that must run against the repository and assert the keyed answer; a general question's key names the source lines that settle it, which must exist. Keys that fail are dropped, not fixed: 3 of 135 were. The writer can be wrong; it cannot be wrong in a way that survives.

    from requests.models import PreparedRequest
p = PreparedRequest()
try:
    p.prepare(method='GET', url='http://127.0.0.1:54897/x', auth=('user', 'pass', 'extra'))
    raised = None
except TypeError as e:
    raised = e
assert raised is not None and 'not callable' in str(raised)

    Reading and marking

    The reader is a model given one text in full, the whole story or the whole DeepWiki wiki for that repository, and the fifteen questions. No repository, no tools, no web. It may answer "the text does not say", which is marked wrong.

    The grader is a model given the question, the key, and the answer, and nothing else: not the arm, not the document. It marks correct, partly or wrong, with a one-line reason. Here is one it marked wrong:

    requests g1 · deepwiki · wrong
    Question: What does prepare_cookies guarantee about being called more than once on the same PreparedRequest?
    Answer given: "The documentation describes prepare_cookies(cookies) only as converting cookies into the Cookie header via get_cookie_header…; it does not state any guarantee about calling it more than once…"
    Grader: Denies the once-only contract and misses both the cookielib has_header mechanism and the fact that resolve_redirects pops the Cookie header before re-preparing.

    Same questions, same keys, same grader for both arms. The only thing that differs between the two scores is the document the reader was handed.

    The nine repositories, side by side

    show:

    Could nine repositories give this by chance?

    Nine is not many. The preregistration's answer is a bootstrap: pretend the nine are the whole world, draw nine of them with replacement (so some appear twice, some not at all), recompute the difference, and repeat. The spread of those differences is the uncertainty. Draw some.

    The preregistered run used 10,000 draws with a fixed seed: the 95% interval for the general-question difference was +5 to +33 points; for path questions, +23 to +47. An interval that stays above zero is what "not luck" means here. The general one clears zero, by less than the path one does.

    What you can say, and what you can't

    The reader was a model, and the same family as the authorClaude Opus 5, reading the text with no tools. It is also the model that wrote the stories, so a reader that happened to prefer its own prose would favour the story; blind grading does not remove that. The accuracy measure has a second judge from another vendor (Jev); this one has only the blinding. No human read for this measure. The one human signal is Omoju's blind read of two stories, recorded before the numbers, and the 20-claim hand check on the accuracy judge.
    "General" meant the core modules, not the whole repositoryChecked after the decision: all 61 general questions are keyed to files the traced run passes through. The question writer had the trace in front of it, and in these libraries the intended use runs through the core. The stories visit those files; they don't visit docs/, the CI, or the far corners. Nothing here measures questions about those.
    Nine small, pure-Python librariesNothing here says anything about large, multi-language systems, and the interval for general questions starts at +5 points: real, not huge.
    The other measure went the other wayThe same study found the stories no more accurate than DeepWiki (0.71 vs 0.54 contradicted claims per 1,000 words; 10 vs 7 of 360), which is why the preregistered rule said kill. Comprehension and accuracy are separate findings; quote them together.

    The sentence that survives all of that: on nine small Python libraries, a model reader given a CodeStory answered owner-style questions about the core modules of the repository better than one given DeepWiki (76% vs 56%; 95% interval on the difference +5 to +33 points, resampling repositories), with questions written from the code and one traced run by a model that saw neither document, keys checked by execution, and blind grading. On questions about the traced run itself the gap was +35 points.

    Everything on this page comes from eval/ in the repository: the questions, every answer, every grade, and every model response.