Comparing ISTA-DASLab's DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF 2.25bpw against the official API

[u/PhysicalIncrease3]

I've been running ISTA-DASLab's 2.25bpw quant of DSv4.1-1017 for a few days, and it's holding up well. But the fact that it's only 2.25bpw average did make me wonder how good the quant actually is against the full model.

Local setup: Project-maya v1.1.27. ISTA-DASLab's 2.25bpw quant. Quality-relevant settings: "DS4_INDEX_TC": "1". "MAYA_DS4_PROMPT_SKIP": "2".

Test: 100 question text-only subset of Humanities Last Exam.

Benchmarking models isn't easy, and these results are nothing but a smoke test. There are lots of issues with these results:

1) I ran the questions with a 32768 token limit per question, because anything larger becomes prohibitively time consuming. It already took my local instance around 12 hours to complete the test, and that's at an average of 40.4 toks/s.

2) When approaching the token limit, Deepseek's API and project-maya work differently. Project-maya will reduce thinking_budget to around 3/4 of the token limit so it always gives an answer, but as a result it never used more than around 24768 tokens on a question. The Deepseek API will instead just use all 32768 tokens thinking if it needs and then output nothing.

This really fucks with the results on a lot of questions unfortunately. There were some questions local answered correctly because it was forced to give a result. There were others that the API answered correctly while using between 24k-32k tokens, where the local model was wrong while being capped at 24k.

3) I ran on thinking "high", not "max" for both, because Deepseek's API was thinking too much and failing to output anything for most questions on "max". Even on thinking "high", the API failed to answer 26 questions and thinking was cut short to 24k on a similar number of local questions.

4) This is only a 100 question subset, and I saw significant run-to-run variance on both the official API and my local model too. There are some questions that both sides will sometimes pass, sometimes fail using the settings as tested. To get proper statistically relevant comparisons you'd need to run this test 10 times or probably more.

The results themselves:

Local mode: 13 correct "strict" answers, 20 semantically correct answers. API: 13 correct "strict" answers, 21 semantically correct answers.

On the 74 questions the official API actually completed:

• Local semantic: 19/74
• Official semantic: 21/74
• Both correct: 17
• Local only: 2
• Official only: 4
• Neither: 51

It's also worth noting that when both were wrong, the wrong answer was identical 23 times, and semantically the same in 11 more. 7 more questions were wrong "simiarly" IE both picked fungi but chose different species.

Conclusion:

Given the flaws in my testing, it's impossible to say specifically how close the quant is to the full fat model. You'd need to burn hundreds of millions of tokens to really know with certainty and that's just on Humanities Last Exam. However what I can say fairly confidently now is that the 2.25bpw quant isn't completely lobotomized. Anything the API could answer, the local model was there or there abouts. For example I'm pretty confident that if I took any question the API could answer, and asked my local model with a 256k token limit it'd probably answer the same. Both sides are capable of a much better score with unlimited token budget and max reasoning.

[u/More-Revenue8609]

Im using their quant for Qwen Flash Next. Very impressive!

Did try Deepseek v4.1 Flash using strata but unfortunately too slow on my pc (15-20tok/s). Does someone gets better speeds using similar setup to mine:

Xeon e5 2667 v4
Dual 3080 20gb (both pcie 3 x16). So total 40gb vram
128gb ddr4 2400mhz
pcie3 nvme 1tb

Just curious if Im missing something or is this speed expected.

[u/PhysicalIncrease3 → u/More-Revenue8609]

I'd try project-maya, it's quite a bit faster for me. I get around 1100 prefill on prompt 64K<, 40 tok/s average on decode. Running 1x3090, 4x3060, 128GB DDR5, 7900X CPU. The GPU unlock helped quite a lot, despite only applying successfully to the 3090 and 2 of the 3060s.

The 2bpw quant will likely run a lot faster than the 2.25 as you should be able to pin it entirely in ram. Project-maya doesn't support the ISTA-DASLab quants in setup.py but you can just edit the .json and point project-maya at the strata-built pack and it works fine. It's also worth asking your AI to graft the 2bpw dflash dfrafter from the Maya-XS quant onto the ISTA-DASLab quant. Saves about 4GB vram, runs faster and doesn't hurt acceptance rates. You can use the vision.bin from the project-maya quant also to get vision working.

[u/PcChip → u/PhysicalIncrease3]

i'm running maya with the exl3bit quant right now, i can only get up to about 35 tok/s decode and prefill is terribly slow

let me know if you try dsx and it's faster, i might try switching

[u/Glittering-Call8746 → u/PhysicalIncrease3]

Gpu unlock for 3090 vs 3060 ? Which is worth it

[u/whiteh4cker → u/More-Revenue8609]

You can give this a try: https://github.com/lxe/dsx

[u/More-Revenue8609 → u/whiteh4cker]

Thanks, didnt see that.
Seems promising since its very similar to my setup. If I could get 30-35 tok/s ngl I would be very pleased.

[u/VotZeFuk]

"MAYA_DS4_PROMPT_SKIP": "2"

I wouldn't go farther than "1" on that.

[u/PhysicalIncrease3 → u/VotZeFuk]

Funnily enough I was thinking of testing that. I enabled it on the basis that Deepseek does the same in DeepInfra. What's your experience?

[u/VotZeFuk → u/PhysicalIncrease3]

Nothing particularly negative about that, but from what I've gathered, "2" is not exactly lossless? Anyway, there shouldn't be that much of a difference but it shifts the answer somewhat.

[u/Intraluminal]

That's a very useful result despite its limitations. Thank you!

[u/EmilPi]

Hi! Nice to see someone actually trying to verify things. I create and evaluate a lot of benchmarks, but my local setup is bottleneck to run it fast.

[u/Primary_Note_118 → u/EmilPi]

kudos for actually running the smoke test, those token limit quirks sound like a nightmare to work around. 2.25bpw hanging that close to the API on semantic matches is wild, was half expecting it to fall off a cliff

[u/GaelOffMySoul]

Would you have the time ( and HW) to try https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw ? Or maybe share the 100 questions to i can test it :) I have a few deepseek api tokens left to try

[u/musicurgy]

I found ISTA-DASLab's GSQ-RCO and nazeshinjite DeepSeek-V4.1-Flash-1017-ds4-GGUF very comparable in my benchmarking

[u/Careful-Report6526]

Props for actually running a structured evaluation instead of a 3-prompt vibe check.
One thing we learned the hard way while running automated local benchmarks: always log wall-clock duration and total generated reasoning tokens alongside the accuracy score. With aggressive quants (and especially with Thinking enabled), a quant sometimes loses only 1–2% accuracy on paper, but takes 2x as many reasoning tokens looping before it settles on an answer. Tracking time-efficiency next to the raw score makes those regressions jump right out.

[u/vogelvogelvogelvogel]

great comparison, thanks for sharing!

[u/And-Bee]

Thanks for doing this. What was the subject matter of the questions?

[u/indrasmirror]

I was wondering if you might try my qaunt out. https://huggingface.co/IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored

In my testing it help up very close to the API on my custom benchmark :) I was quite shocked by that tbh.

[u/Steuern_Runter]

Interesting! Could you also test how quantized Qwen 3.8 Flash Next holds up against DS 4.1 on the same tasks?

[u/SnooPaintings8639]

Good work! We all hope you'll keep on pushing this, we need such tests.

Let's upvote OP before the "Q2 is brain-dead/lobotomized no matter whaa" crowd gets here.

[u/PhysicalIncrease3 → u/SnooPaintings8639]

Thanks, I'd love to do more testing but to get to the point where you have statistically relevant results you'd have to burn a huge number of tokens. The run to run variance is massive. On this run alone, the API correctly answered 4 questions the local model got wrong, and the local model answered 3 correctly than the API did not. There are others questions I've seen both sides answer correctly/incorrectly across different runs as well.

For me, if the quant is close enough that you're having to faff around finding statistical variance over many runs, then it's good enough!

[u/SnooPaintings8639 → u/PhysicalIncrease3]

I have the same stance as your second paragraph. If the objective difference between quants is no larger than the model inherent variance, but the speed is 2 x higher on one... it is a no-brainer which one to choose.

[u/TheRealJesus2 → u/PhysicalIncrease3]

I feel like not enough people get this point lol. 

This is why my vibes matter the most when it’s me driving the model. Unfortunately that also takes a lot of tokens to figure out but at least I’m doing work while it happens :) 

Appreciate the work, OP!

[u/someguyinr → u/SnooPaintings8639]

Turns out lobotomizing the model just stopped it from overthinking.

[u/enternoescape → u/SnooPaintings8639]

DeepSeek perhaps due to parameter count and the fact it's starting with 4bit experts doesn't seem to get as upset at Q2. I'm running EXL3 Q2.9 because that's the max I can fit between VRAM and RAM and it's doing a great job on items Qwen 3.8 Flash Next at a NVFP4 struggles with. I'm getting 2000 tok/s prefill and 54-100 tok/s decode running Project Maya.