When a token-generation test uses the wrong golden

Published October 16, 2026 · Updated October 17, 2026

The release test failed when the model chose Poland.

I was adding four-bit support for Qwen3.8-27B to Doppler, my model-execution runtime. The model loaded and generated text, but the coding agent reported a failed qualification. A test required every generated token to match a saved transcript from a higher-precision BF16 model.[1]

That saved transcript was the golden: the sequence the test treated as correct. The two runs had reached different countries in an open-ended list. Before fixing the model, we needed to understand what that disagreement actually proved.

Two correct capitals, one failed test

The prompt was The capital of France is. Both versions supplied Paris, then kept listing countries and capitals. They agreed through twelve pairs, ending with Austria and Vienna. The BF16 reference continued toward Denmark and Copenhagen. The four-bit version, Q4, chose Poland and Warsaw.[1]

The harness reported First source-token mismatch at generation index 97. Its generation check passed; its token-parity check failed. Both capital pairs were correct, and the prompt had not specified which country should follow Austria.

The agent had correctly read a failed test. The mistake was letting that test settle a larger question: whether the Q4 implementation was broken.

Deterministic does not mean identical

Both runs used greedy decoding: choose the highest-scoring next token. Doppler recorded temperature: 0 and topK: 1; the reference used do_sample: false.[1] No random sampling or LLM judge was involved.

But the models were calculating with different weights. Quantization stores most weight matrices using fewer bits, changing their values. When token scores are close, those changes can swap the winner. After the first different token, each model also receives a different history, so later differences become harder to interpret.

That mechanism explains how the paths can split; the transcript alone does not establish the score margin at this branch. The related floating-point issue appears in Ask Gemma 4 again and it may answer the opposite.

Poland being plausible was no proof of runtime correctness, either. Incorrect weight decoding, a wrong operator, a tokenizer mismatch or a broken cache could also change the answer. We needed a reference for the computation Q4 was supposed to perform.

Compare the same model at the same step

For an implementation check, the independent reference must use the same retained Q4 weights. Comparing Q4 with BF16 can reveal the effect of quantization, but cannot by itself separate that effect from a runtime bug.

Fix the prompt tokens, template, model artifacts and generation settings. Then feed both implementations the same token history and compare their next-token scores. This keeps the first numerical disagreement separate from the later consequences of different choices.

Set numerical limits from independent controls before examining holdout results. If the candidate fails, trace the first broken computation. Generating a new golden from the candidate’s own output would only teach the test to accept whatever the candidate does.

The follow-up Q4 qualification used an independent reference that decoded the actual retained quantized bytes. It did not quantize the original weights again and assume they matched. Its limits were calibrated, then frozen before new holdout prompts ran. Those scoped checks passed.[2]

A separate task found a real wrong answer

Correctly executing Q4 did not settle whether Q4 answered well. Both precisions passed an initial six-task quality check. Separate browser checks covered follow-up, cancellation, reuse and history through a native whole-model provider.[3][4] Neither result established general answer quality.

A later code-tracing task made that limit concrete. The model was asked to evaluate this Python calculation and return only the number:

values = [2, 5, 8]
result = sum(
    x * 2 for x in values
    if x > 3
)

Only 5 and 8 qualify. Double them and add: 10 + 16 = 26. Q4 answered 22.

The independent reference reproduced the wrong answer with Q4. With matching F16 weights, it returned 26. The model revision, prompt, tokenizer, greedy generation and stopping rules were held constant; both paths stopped after the two digits.[5]

This implicates quantization in this particular failure. It does not establish general Q4 or F16 quality, and it does not qualify Doppler’s F16 GPU execution: these controls used independent CPU computations. Release eligibility still depends on the frozen quality suite and its declared thresholds.

The country-list mismatch had shown a different continuation. The code-tracing control showed a wrong answer that the higher-precision weights avoided. We needed different tests to tell those stories apart.

Why the distinction matters

There was a practical reason to keep investigating Q4. Historical managed buffer-pool peaks were 50.6 GiB for F16 and 16.3 GiB for Q4; a later chunked Q4 provider peaked at 16.9 GiB.[1][2] Those separate runs are not a matched speed benchmark, and the counters do not measure all physical memory used by the process and driver.

Lower memory use made Q4 worth testing. It did not excuse wrong answers. Equally, choosing Poland did not justify rejecting it for the wrong reason.

The original failure stays in the record. A useful test makes clear what a failure means: the runtime disagreed with a matching reference, the answer was wrong, or the application could not complete the request. That gives the agent a specific problem to fix.