MityaNamed after Mitya, Dmitri Karamazov.Named after Mitya, Dmitri Karamazov.

Five novels are parsed into 231 chapters, chunked at 500 tokens with 100 overlap, and indexed as 3,284 points that carry book, part and chapter as payload rather than as text. A question is routed by a pure function with no model in it: name a resolvable chapter and ask for a summary and it takes a structural path, otherwise the default one. That path expands the query with canonical names and aliases from a human gated entity map, runs a dense leg and a BM25 leg in a single Qdrant round trip fused server side by RRF, then reranks down to the three spans that are served.

Two things went wrong, and both were in the measurement rather than in the stack.

The published recall number was the recall of a pipeline production does not run. The harness called the router, asserted the route, logged that the route was fine, then ran the default hybrid path for every row regardless. The structural retrieval function was imported by production, imported by the quality harness, and imported by nothing in the retrieval harness. The assertion passed the entire time.

Then recall turned out to be bimodal across repeated runs of one unchanged commit. The entity map's scan returned a set, so any query touching more than one character got its alias blocks back in a different order depending on the process hash seed. Different order, different expanded query, different embedding, different ranking. That is not only a measurement problem: two users could ask the same question and get different chunks.

Both fixed, and the baseline re-derived: recall@5 of 0.7105, 27 of 38 rows on the primary family.^[1][1]0.7105 = 27/38, denominator sequence 26 / 25 / 38; MRR 0.5039.[1]0.7105 = 27/38, denominator sequence 26 / 25 / 38; MRR 0.5039. v1 had published 0.720, and the new number is not a regression from it, because the denominator moved three times in between.^[2][2]v1 0.720 = 18/25, superseded.[2]v1 0.720 = 18/25, superseded. The two were never measuring the same set. Knowing which configuration produced which number is a separate problem, and the system did not have an answer to it yet.

One large defect was found and deliberately left alone. Retrieval never sees the conversation. In the session corpus, 68 of 108 turns are contentless follow-ups, so 63 percent of multi turn requests retrieve against a string with nothing in it to retrieve on.^[3][3]63% = 68 of 108 turns.[3]63% = 68 of 108 turns. Documented, unrepaired, and the biggest known gap in the system.

The prompt is assembled in blocks: a system message, the persona head, a precedence marker that names each block and declares it data, the memory block, the conversation, and a reinforcement tail. The user turn carries the question and nothing else. Generation runs on hosted APIs behind a gateway speaking to two provider families, picked for how much they diverge in streaming shape, error vocabulary and rate limit surface, and each reached through its own native API rather than a compatibility endpoint. The first token is pulled before the response body opens, so a transport failure, a refusal or a size rejection arrives as a typed status instead of a hung 200.

The first adversarial evaluation this project ever ran found that the prompt's injection fence was doing nothing. Against a control differing in exactly one respect, the shipped prompt was set-identical: the same attempts landed, with an empty symmetric difference in both directions. Its replacement, composition C, then nearly lost to a broken metric. An assistant-tell detector scored C four times worse for breaking character, and all 26 of 26 matches turned out to be the bare literal "I cannot" inside in-character narration, which is the persona correctly refusing. The metric was punishing the arm that obeys its own grounding instruction, and published unread it would have said the safer prompt breaks character four times as often.

Composition C shipped, and it is the only change the entire optimizer phase promoted. It cut jailbreak violations from 0.5625 to 0.1250 across 16 attempts, a 78 percent reduction in that family,^[4][4]0.5625 → 0.1250 over 16 attempts, jailbreak family only.[4]0.5625 → 0.1250 over 16 attempts, jailbreak family only. and on the quality leg it came back superior on groundedness rather than merely non-inferior, +0.724 against a detection floor of 0.343.^[5][5]+0.724 against a measured MDE (minimum detectable effect) of 0.343 at n = 58, judge scale 1 to 5: superior, not merely non-inferior.[5]+0.724 against a measured MDE (minimum detectable effect) of 0.343 at n = 58, judge scale 1 to 5: superior, not merely non-inferior. Prompt extraction did not move, holding at 1 in 8 on every arm tested, which is where the result stops.^[6][6]1/8 on all three arms: control, shipped, promoted.[6]1/8 on all three arms: control, shipped, promoted. The two legs ran on different models, so they are two results rather than one.

The output ceiling from section 2 was raised from 400 to 1024, and the change was recorded as a measurement break enforced in code: ask the gate to compare a number from before it against one from after and it returns an error naming the break instead of a delta.

/chat takes a session cookie, resolves it server side on every request, derives the tenant, and streams SSE events: tokens, then sources, then meta. Sessions are opaque server side rows rather than JWTs, so a session can be revoked; the cookie is HttpOnly, Secure and SameSite Lax, and it rotates on login. The inbound tenant header was removed rather than supplemented, because an authenticated tenant that an unauthenticated header can override is not authenticated. In front of that sits pure ASGI admission middleware holding one in-flight slot per request, bounded per tenant and in total, releasing on all six exits including client disconnect.

The admission layer exists because of what the first load test found. The system had never rejected anything: at every concurrency level, on both arms, zero local 429s, zero provider 502s, not one erroring request, and a 62 second p99 at a 100 percent success rate. It was queueing silently and without bound: every request succeeded, and the only symptom was a tail nobody was watching. Bounding it at two per tenant and six in total, with the inequality chosen so the total binds before every tenant reaches its own cap, took p99 on the same arm at the same level from 64,405 ms with zero rejections to 4,429 ms with thirteen.^[7][7]p99 64,405 ms / 0 rejections to 4,429 ms / 13.[7]p99 64,405 ms / 0 rejections to 4,429 ms / 13.

The five rung degradation ladder was written but had never been wired into the request path, and the readiness probe it depended on was querying llama-server, which no longer generates anything. Both were fixed together. Readiness is now deliberately config-only, configured and credentialed rather than reachable, because the front end polls it every few seconds against a free tier that allows very few requests per minute.

The concurrency knee moved from one to four after the lock was narrowed to cover routing through context assembly,^[8][8]knee 1 -> 4 (first measurement, v1 published no concurrency number).[8]knee 1 -> 4 (first measurement, v1 published no concurrency number). and authentication went in at no measurable latency cost.^[9][9]auth 2,045 ms p50 against 2,194 ms pre-auth.[9]auth 2,045 ms p50 against 2,194 ms pre-auth. What the admission bound buys is a bounded wait, not more throughput: retrieval is still mutually exclusive, so the lock was narrowed rather than parallelized.

One span per stage boundary feeds both the trace and the metric from a single set of instrumentation calls, so a timing and its distribution cannot disagree. A configuration version is the content hash of a frozen validated model, twelve hex characters of a canonical serialization, and every span, every metric and every evaluation row carries it. Two identical configurations are the same version by construction. Runs are stamped with the config version, the corpus version, the git SHA, whether the tree was dirty, the serving model and the prompt composition. On top of that sits a promotion gate that compares the promoted configuration against a candidate in three layers, with no hand set threshold anywhere: every bar is derived from the run being judged.

That design came out of finding what happens without it. A configuration version that did not pin the generation model meant the version did not churn between two provider arms, so arm B was served arm A's cached answers, contaminating quality, then latency, then cost, in that order. Setting an environment variable to select an arm turned out not to switch arms at all: both runs carried the same config version, labelled nothing, and produced scores plausible enough to have been believed. The rule that came out of it is short. To measure an arm, promote a configuration for it.

Then the gate was asked what it could actually see. Strip every retrieved passage from the prompt, which is a total and certain regression, and the two mechanical metrics move by 0.0000. Identical zeros. The LLM judge catches it at 5.4 times its own detection floor,^[10][10]mechanical 0.0000 / 0.0000 vs judge groundedness 4.022 -> 1.000 at 5.4x MDE.[10]mechanical 0.0000 / 0.0000 vs judge groundedness 4.022 -> 1.000 at 5.4x MDE. but the mechanical layer does not respond at all, and no sample size fixes a metric that does not respond. Worse, the ungrounded configuration is 32 percent cheaper and passes the gate clean,^[11][11]ungrounded arm 32% cheaper.[11]ungrounded arm 32% cheaper. so anything optimizing against cost under that gate is being steered toward it. The gate's report now carries a permanent caption excluding grounding from what it verifies, enforced by the report validator rather than by convention, and the schema rejects any overall, score or composite key outright: a single score would hide a weighting nobody has chosen.

Across the phase, thirteen or more defects were found in the instruments, and two in the product.^[12][12]instrument defects 13+ vs product defects 2.[12]instrument defects 13+ vs product defects 2. Not one of the thirteen was caught by a run that failed. They were caught by runs that passed and looked fine.

Three cost numbers exist because they answer different questions. A reconstructed estimate computable from a configuration and a manifest alone, which the promotion gate blocks on. An assembled estimate that requires retrieval and the memory read to have actually run, which the optimizer ranks on. And a realised figure from provider reported usage. Free tier calls are priced at published list rates and flagged unmetered, so any aggregate containing one renders as an estimate rather than a bill.

v1 had never measured where the time went. Its design document justified the reranker at roughly a fifth of the retrieval path. One trace put it at the overwhelming majority of it. Retrieval proper is flat at about 256 ms regardless of depth, so every millisecond above that is reranker work. Cutting the fused pool from 25 to 10 took retrieval wall clock from 2,121 ms to 917 ms, a 2.3 times reduction, and recall@5 did not move at any depth tested between 25 and 5.^[13][13]fused 25 -> 10, 2,121 ms -> 917 ms, recall@5 flat at 0.7105.[13]fused 25 -> 10, 2,121 ms -> 917 ms, recall@5 flat at 0.7105. It is a latency axis, not a cost axis: the same change is worth about 0.04 percent of the money. Generation stopped being the bottleneck, and retrieval became it.

Context assembly gained the ability to merge adjacent retrieved spans and drop the text they duplicate. Over 97 retrieval rows it cut mean served span tokens from 1,535 to 1,502, with recall, coverage and multi hit rate identical, 28 rows cheaper and none more expensive.^[14][14]merge-only 1,535 -> 1,502 tokens, -2.19% over 97 rows.[14]merge-only 1,535 -> 1,502 tokens, -2.19% over 97 rows. It ships as the default collision policy, and enabling it in the request path is a pointer write that is planned rather than done.

The cost model is exact and the spend is nothing. A production answer costs $0.003982 at list rate, the cheapest measured arm $0.001325, and actual spend across the phase was $0.00, because every rate row is unmetered.^[15][15]$0.003982 production / $0.001325 cheapest / $0.00 actual.[15]$0.003982 production / $0.001325 cheapest / $0.00 actual. Routing between arms was rejected before it was built: the best routed family came out 74 percent more expensive than always sending traffic to the cheapest arm, and the hierarchy is inverted, since the cheaper arm is the larger model. That result is bounded by what a free tier could measure.