r/dotnet • u/fuzhongkai • 20h ago
Article I benchmarked a native .NET LLM inference engine against llama.cpp, vLLM, SGLang, NInfer, and velox on an RTX 5090 — here are the results
TL;DR
I benchmarked TensorSharp, an open-source native .NET inference engine, against five other inference engines running Qwen3.8 27B on a single NVIDIA RTX 5090 (32GB).
The benchmark covers real-world workloads: coding, long-context processing, multi-turn conversations, JSON generation, function calling, agentic workflows, vision, and concurrent requests.
Key findings:
- TTFT: TensorSharp achieved the fastest screenshot-processing TTFT at 125 ms, ahead of vLLM (127 ms), SGLang (153 ms), and llama.cpp (309 ms).
- Prefill: TensorSharp processed approximately 8,000–9,300 tokens/sec, compared with 8,900–12,400 for NInfer/SGLang and 6,300–7,600 for llama.cpp.
- Decode: TensorSharp reached 80 tokens/sec without speculative decoding and 234 tokens/sec with DFlash2, a 2.9× improvement.
- Coding: TensorSharp generated code at 339 tokens/sec, compared with 321 for llama.cpp, 383 for vLLM, and 415 for NInfer.
- Parallel requests: TensorSharp achieved 445 aggregate tokens/sec with 4 concurrent requests, ranking second behind NInfer (488).
- Agentic workflows: TensorSharp completed a four-turn agentic workflow in 0.748 seconds, the fastest of all six engines.
- Overall winner: NInfer led most raw throughput measurements. TensorSharp was particularly competitive in agentic workflows, screenshot TTFT, and concurrent inference.
For a native .NET inference engine, these results are encouraging. They show that .NET can be a viable foundation for high-performance local AI inference, not just an API client for Python-based inference servers.
Project: https://github.com/zhongkaifu/TensorSharp
1. Benchmark setup
All six engines ran sequentially on the same machine.
| Component | Configuration |
|---|---|
| GPU | NVIDIA RTX 5090, 32GB |
| CPU | AMD EPYC 9354 |
| System memory | 93GB limit |
| OS | Ubuntu 24.04 |
| Model | Qwen3.8 27B |
| Context | 40,960 tokens per sequence |
| Concurrency | Up to 4 requests |
| Speculative decoding | DFlash2, up to 7 drafts per step |
| Inference mode | Greedy, thinking disabled |
| Repetitions | 3 per scenario |
Engines tested:
- TensorSharp — native .NET, direct CUDA backend
- llama.cpp — C/C++ inference engine
- vLLM — Python/PyTorch inference framework
- SGLang — Python/PyTorch serving framework
- velox-inferer — Rust inference engine
- NInfer — native inference engine
An important qualification: the engines used different quantized checkpoints and drafter formats. TensorSharp and llama.cpp used the same NVFP4 GGUF weights, while vLLM/SGLang and NInfer/velox used their respective model formats.
Therefore, these are end-to-end engine + checkpoint comparisons, not isolated comparisons of programming languages or inference kernels.
2. Time to First Token (TTFT)
TTFT matters for interactive applications because it determines how quickly users see the first response.
Without speculative decoding
Lower is better. All values are in seconds.
| Task | TensorSharp | llama.cpp | vLLM | SGLang | velox | NInfer |
|---|---|---|---|---|---|---|
| Short prompt (~50 tokens) | 0.050 | 0.027 | 0.061 | 0.042 | 0.088 | 0.051 |
| 4K prompt | 0.430 | 0.524 | 0.368 | 0.320 | 0.533 | 0.322 |
| 16K prompt | 1.64 | 2.05 | 1.45 | 1.34 | 1.80 | 1.24 |
| 32K prompt | 3.96 | 5.01 | 3.73 | 3.58 | 4.21 | 3.02 |
| Multi-turn, later turns | 0.052 | 0.038 | 0.145 | 0.058 | 0.081 | 0.046 |
| Screenshot (1344×768) | 0.125 | 0.309 | 0.127 | 0.153 | 0.250 | 0.162 |
Observations:
TensorSharp’s screenshot TTFT was approximately 2.47× faster than llama.cpp and 2× faster than velox.
For short text prompts, llama.cpp was fastest, while TensorSharp responded in 50 ms.
For long-context workloads, NInfer led. TensorSharp processed a 32K prompt in 3.96 seconds, compared with 5.01 seconds for llama.cpp.
3. Prefill performance: processing long prompts
Prefill throughput becomes particularly important for RAG, document analysis, codebase understanding, and long-context agents.
The benchmark measured the following prefill throughput ranges:
| Engine | Prefill throughput |
|---|---|
| TensorSharp | 8,000–9,300 tok/s |
| NInfer / SGLang | 8,900–12,400 tok/s |
| llama.cpp | 6,300–7,600 tok/s |
These are reported ranges, not identical per-engine averages.
A more concrete comparison is the 32K-prompt TTFT:
| Engine | 32K TTFT |
|---|---|
| NInfer | 3.02 s |
| SGLang | 3.58 s |
| vLLM | 3.73 s |
| TensorSharp | 3.96 s |
| velox | 4.21 s |
| llama.cpp | 5.01 s |
TensorSharp was approximately 21% faster than llama.cpp in 32K-prompt TTFT, although it remained behind NInfer, SGLang, and vLLM.
For .NET developers building document-processing pipelines or retrieval-augmented agents, this is an important result: a native .NET engine can process substantial contexts at competitive speeds.
4. Decode performance: plain vs. DFlash2
Decode throughput determines how quickly the model generates its response after prefill.
Single-request decode throughput
| Engine | Plain decode | DFlash2 decode | Speedup |
|---|---|---|---|
| TensorSharp | 80 | 234 | 2.9× |
| llama.cpp | 84 | 213 | 2.5× |
| vLLM | 69 | 248 | 3.6× |
| SGLang | 79 | 235 | 3.0× |
| velox | 69 | 235 | 3.4× |
| NInfer | 77 | 281 | 3.6× |
All throughput values are tokens/sec.
Without speculative decoding, TensorSharp ranked second, just behind llama.cpp.
With DFlash2, TensorSharp reached 234 tok/s, effectively tied with SGLang and velox at 235 tok/s, but behind vLLM and NInfer.
The 234 tok/s result is the mean across code, prose, and passage-copy workloads.
Interestingly, all engines reporting DFlash2 acceptance achieved approximately 4.4–4.6 emitted tokens per verification step.
That suggests the remaining performance differences are primarily in draft-and-verify execution costs rather than drafter acceptance.
5. Performance by generation task
A single decode number doesn’t tell the whole story.
Different workloads have different generation characteristics. We separately measured coding, prose generation, and passage copying with DFlash2 enabled.
Higher is better, tokens/sec.
| Engine | Code generation | Prose generation | Passage copying |
|---|---|---|---|
| TensorSharp | 339 | 174 | 190 |
| llama.cpp | 321 | 160 | 158 |
| vLLM | 383 | 189 | 171 |
| SGLang | 368 | 184 | 152 |
| velox | 346 | 184 | 175 |
| NInfer | 415 | 209 | 217 |
Coding assistants
TensorSharp generated code at 339 tok/s:
- 5.6% faster than llama.cpp
- 11.5% slower than vLLM
- 18.3% slower than NInfer
This makes TensorSharp a competitive option for local code generation, while leaving meaningful room to optimize speculative verification.
Prose generation
TensorSharp achieved 174 tok/s, compared with 160 for llama.cpp and 209 for NInfer.
Passage copying
TensorSharp reached 190 tok/s, ranking second behind NInfer.
These results show why it is useful to benchmark different output patterns rather than relying on a single synthetic generation test.
6. Parallel request performance
Single-request speed is important, but inference servers also need to handle multiple simultaneous users or agent tasks.
Without speculative decoding
Aggregate output throughput, tokens/sec:
| Engine | 1 request | 2 requests | 4 requests |
|---|---|---|---|
| TensorSharp | 80 | 137 | 215 |
| llama.cpp | 84 | 154 | 251 |
| vLLM | 69 | 116 | 226 |
| SGLang | 79 | 140 | 148 |
| velox | 69 | 67 | 67 |
| NInfer | 77 | 144 | 257 |
TensorSharp scaled from 80 tok/s for one request to 215 aggregate tok/s for four requests, a 2.69× throughput increase.
With DFlash2
| Engine | 4-request aggregate throughput |
|---|---|
| TensorSharp | 445 tok/s |
| llama.cpp | 259 tok/s |
| vLLM | 304 tok/s |
| SGLang | 251 tok/s |
| velox | 303 tok/s |
| NInfer | 488 tok/s |
TensorSharp ranked second, achieving:
- 72% higher throughput than llama.cpp
- 46% higher throughput than vLLM
- 47% higher throughput than velox
- Approximately 91% of NInfer’s throughput
This is particularly relevant for .NET applications serving multiple simultaneous chat sessions, coding agents, or background AI tasks.
One caveat: SGLang could only run two active DFlash2 lanes within the 32GB GPU memory limit, so its four-request result includes queued requests.
7. Agentic workflows and function calling
This is one of the most interesting parts of the benchmark for .NET developers.
Rather than only measuring token generation, we tested complete agent-style tasks involving tool calls, structured output, and multiple conversation turns.
All results below use DFlash2.
Lower is better. End-to-end latency in seconds.
| Task | TensorSharp | llama.cpp | vLLM | SGLang | velox | NInfer |
|---|---|---|---|---|---|---|
| JSON object | 0.281 | 0.314 | 0.391 | 0.304 | 0.369 | 0.274 |
| JSON schema | 0.309 | 0.473 | 0.445 | 0.360 | Failed | 0.258 |
| Single function call | 0.199 | 0.218 | 0.354 | 0.171 | 0.383 | 0.180 |
| 3 parallel function calls | 0.427 | 0.484 | 0.539 | 0.457 | 0.680 | 0.378 |
| Agentic loop (4 turns) | 0.748 | 0.817 | 1.45 | 0.791 | 1.42 | 0.817 |
Agentic workflow: TensorSharp finished first
The agentic benchmark simulated a small workflow involving file listing, file reading, and calculator operations.
The engine had to complete the tool interactions and return the correct final result.
TensorSharp finished in 748 ms, compared with:
- 817 ms for llama.cpp
- 1,450 ms for vLLM
- 791 ms for SGLang
- 1,420 ms for velox
- 817 ms for NInfer
This is approximately 1.94× faster than vLLM and 1.90× faster than velox in this particular workflow.
For developers building .NET-based AI agents, this is arguably more meaningful than raw tokens/sec.
An agent’s responsiveness depends on the entire tool-execution loop, not just model decoding.
Structured output correctness
The benchmark also validated output correctness.
Most checks passed, with two notable exceptions:
- velox ignored the strict JSON-schema response format in all three repetitions.
- NInfer with DFlash2 produced an empty JSON key in one of three JSON-object replies.
One additional detail: NInfer’s JSON responses were shorter than those of the other engines, which explains part of its end-to-end latency advantage.
This is why benchmarking structured outputs should include correctness checks, not just latency.
8. Why this matters for .NET
TensorSharp is an open-source project exploring whether we can build a complete local AI inference engine and agent runtime around the .NET ecosystem.
The goal isn’t to wrap a Python inference server behind an HTTP client.
It’s to make .NET a first-class environment for model inference, with native execution backends and agent integration.
The benchmark used TensorSharp’s direct-CUDA backend, so these numbers should not be interpreted as pure managed-C# kernel performance. GPU execution and native kernels remain critical.
But the results demonstrate something useful:
A .NET-based inference stack can compete with established inference frameworks on real workloads, including speculative decoding, concurrent serving, and agentic tool execution.
There is still substantial optimization work to do.
In particular, TensorSharp’s DFlash2 draft-and-verify execution is slower than NInfer’s despite similar draft acceptance rates.
That is an actionable optimization target rather than a fundamental limitation of .NET.
9. Limitations and reproducibility
A few things worth emphasizing:
- All measurements were performed on a single RTX 5090.
- Different engines used different quantized checkpoints and drafter formats.
- The benchmark used greedy decoding with thinking disabled.
- Each scenario had three measured repetitions.
- Differences below approximately 5% should be treated as ties.
- The benchmark did not evaluate more than four concurrent requests, model load time, video, or broader answer quality.
- These are operational performance comparisons, not proof that one programming language is inherently faster than another.
The benchmark uses streamed OpenAI-compatible requests and validates the responses.
The TensorSharp repository includes scripts for running the comparison and generating the performance reports:
- eng/qwen38-engine-servers.sh
- eng/qwen38-engine-bench.py
- eng/qwen38-engine-report.py
GitHub: https://github.com/zhongkaifu/TensorSharp
10. Discussion
I’m the author of TensorSharp, and I’d be interested in feedback from the .NET community.
In particular:
- Would you consider running local LLM inference directly inside a .NET application instead of depending on a separate Python-based inference service?
- For your workloads, which matters most: TTFT, prefill throughput, decode speed, concurrent throughput, or end-to-end agent latency?
- What would make a native .NET inference engine genuinely useful in production?
I’m also interested in suggestions for improving DFlash2 verification performance and scaling beyond four concurrent requests.
The project is open source, and contributions, benchmark reproductions, and technical feedback are very welcome.