A public draft model lost 167 tokens per second to the one built into Qwen3.6

[u/Ctbhatia]

I work at CoralBricks, the inference provider behind the writeup at the end, so this is a builder's comparison rather than a neutral review. We tuned the serving stack for Qwen3.6-35B-A3B and swapped in a public drafter, a second model that guesses tokens for the main one to check. We expected it to beat the draft head Qwen3.6 ships with. On the same stack it ran at 525 tokens per second against 692.

The gap was the checkpoint, not the architecture. The public drafter accepted 4.33 tokens per verification step against 4.67 for the model's own head on the same stack. It had been trained for short general benchmarks and attends over a 2,048-token window, while agent turns replay tens of thousands of tokens of file paths and tool output.

Fine-tuning the drafter on software-engineering traces generated by Qwen3.6 itself and widening its window to 8K took acceptance to 6.46 tokens per step. The target weights never changed.

With that drafter the stack ran at 1,138 tokens per second, 1.93x stock SGLang. Conditions were thinking off, one H100 SXM per request, tensor parallel 1, batch size 1, greedy decoding and max tokens 1024, a median across all 15,777 runnable turns.

The step got cheaper too. Stock SGLang's MTP-4 step takes 7.72 ms and verifies five positions. Our stack verifies eight and takes 5.71 ms, with the kernels and the fine-tuned drafter both in it. GSM8K stayed at 96.8%, because the target model still decides every token.

The pair to log is accepted tokens per verification step and the time that step took. Acceptance alone can pick a drafter that loses once each verification costs more. A tokens-per-second headline measured on short prompts says little about a 90K-token agent turn. On our workload the head the model ships with was the stronger baseline, since it was trained alongside the model.

The ablations, including the ideas that lost, are in the writeup https://www.coralbricks.ai/blog/speculative-decoding-qwen36?utm_source=reddit&utm_campaign=content

[u/NeuralNomad87]

the acceptance plus step time pairing is a good callout.

how many SWE traces did the fine-tune need before acceptance stopped climbing? and did the 8K window cost you anything on short turns?

[u/Ctbhatia → u/NeuralNomad87]

the trace count isnt in my notes, i'll dig it out of the run logs. on short turns nothing should change until the drafter reads past the old 2k window, we never isolated it tho

[u/NeuralNomad87 → u/Ctbhatia]

makes sense, short turns shouldn't change if it never reads that far. if the trace count turns up I'd still be curious, was it closer to hundreds or thousands?