TL;DR

I benchmarked TensorSharp, an open-source native .NET inference engine, against five other inference engines running Qwen3.8 27B on a single NVIDIA RTX 5090 (32GB).

The benchmark covers real-world workloads: coding, long-context processing, multi-turn conversations, JSON generation, function calling, agentic workflows, vision, and concurrent requests.

Key findings:

• TTFT: TensorSharp achieved the fastest screenshot-processing TTFT at 125 ms, ahead of vLLM (127 ms), SGLang (153 ms), and llama.cpp (309 ms).
• Prefill: TensorSharp processed approximately 8,000–9,300 tokens/sec, compared with 8,900–12,400 for NInfer/SGLang and 6,300–7,600 for llama.cpp.
• Decode: TensorSharp reached 80 tokens/sec without speculative decoding and 234 tokens/sec with DFlash2, a 2.9× improvement.
• Coding: TensorSharp generated code at 339 tokens/sec, compared with 321 for llama.cpp, 383 for vLLM, and 415 for NInfer.
• Parallel requests: TensorSharp achieved 445 aggregate tokens/sec with 4 concurrent requests, ranking second behind NInfer (488).
• Agentic workflows: TensorSharp completed a four-turn agentic workflow in 0.748 seconds, the fastest of all six engines.
• Overall winner: NInfer led most raw throughput measurements. TensorSharp was particularly competitive in agentic workflows, screenshot TTFT, and concurrent inference.

For a native .NET inference engine, these results are encouraging. They show that .NET can be a viable foundation for high-performance local AI inference, not just an API client for Python-based inference servers.

Project: https://github.com/zhongkaifu/TensorSharp

1. Benchmark setup

All six engines ran sequentially on the same machine.

Component	Configuration
GPU	NVIDIA RTX 5090, 32GB
CPU	AMD EPYC 9354
System memory	93GB limit
OS	Ubuntu 24.04
Model	Qwen3.8 27B
Context	40,960 tokens per sequence
Concurrency	Up to 4 requests
Speculative decoding	DFlash2, up to 7 drafts per step
Inference mode	Greedy, thinking disabled
Repetitions	3 per scenario

Engines tested:

• TensorSharp — native .NET, direct CUDA backend
• llama.cpp — C/C++ inference engine
• vLLM — Python/PyTorch inference framework
• SGLang — Python/PyTorch serving framework
• velox-inferer — Rust inference engine
• NInfer — native inference engine

An important qualification: the engines used different quantized checkpoints and drafter formats. TensorSharp and llama.cpp used the same NVFP4 GGUF weights, while vLLM/SGLang and NInfer/velox used their respective model formats.

Therefore, these are end-to-end engine + checkpoint comparisons, not isolated comparisons of programming languages or inference kernels.

2. Time to First Token (TTFT)

TTFT matters for interactive applications because it determines how quickly users see the first response.

Without speculative decoding

Lower is better. All values are in seconds.

Task	TensorSharp	llama.cpp	vLLM	SGLang	velox	NInfer
Short prompt (~50 tokens)	0.050	0.027	0.061	0.042	0.088	0.051
4K prompt	0.430	0.524	0.368	0.320	0.533	0.322
16K prompt	1.64	2.05	1.45	1.34	1.80	1.24
32K prompt	3.96	5.01	3.73	3.58	4.21	3.02
Multi-turn, later turns	0.052	0.038	0.145	0.058	0.081	0.046
Screenshot (1344×768)	0.125	0.309	0.127	0.153	0.250	0.162

Observations:

TensorSharp’s screenshot TTFT was approximately 2.47× faster than llama.cpp and 2× faster than velox.

For short text prompts, llama.cpp was fastest, while TensorSharp responded in 50 ms.

For long-context workloads, NInfer led. TensorSharp processed a 32K prompt in 3.96 seconds, compared with 5.01 seconds for llama.cpp.

3. Prefill performance: processing long prompts

Prefill throughput becomes particularly important for RAG, document analysis, codebase understanding, and long-context agents.

The benchmark measured the following prefill throughput ranges:

Engine	Prefill throughput
TensorSharp	8,000–9,300 tok/s
NInfer / SGLang	8,900–12,400 tok/s
llama.cpp	6,300–7,600 tok/s

These are reported ranges, not identical per-engine averages.

A more concrete comparison is the 32K-prompt TTFT:

Engine	32K TTFT
NInfer	3.02 s
SGLang	3.58 s
vLLM	3.73 s
TensorSharp	3.96 s
velox	4.21 s
llama.cpp	5.01 s

TensorSharp was approximately 21% faster than llama.cpp in 32K-prompt TTFT, although it remained behind NInfer, SGLang, and vLLM.

For .NET developers building document-processing pipelines or retrieval-augmented agents, this is an important result: a native .NET engine can process substantial contexts at competitive speeds.

4. Decode performance: plain vs. DFlash2

Decode throughput determines how quickly the model generates its response after prefill.

Single-request decode throughput

Engine	Plain decode	DFlash2 decode	Speedup
TensorSharp	80	234	2.9×
llama.cpp	84	213	2.5×
vLLM	69	248	3.6×
SGLang	79	235	3.0×
velox	69	235	3.4×
NInfer	77	281	3.6×

All throughput values are tokens/sec.

Without speculative decoding, TensorSharp ranked second, just behind llama.cpp.

With DFlash2, TensorSharp reached 234 tok/s, effectively tied with SGLang and velox at 235 tok/s, but behind vLLM and NInfer.

The 234 tok/s result is the mean across code, prose, and passage-copy workloads.

Interestingly, all engines reporting DFlash2 acceptance achieved approximately 4.4–4.6 emitted tokens per verification step.

That suggests the remaining performance differences are primarily in draft-and-verify execution costs rather than drafter acceptance.

5. Performance by generation task

A single decode number doesn’t tell the whole story.

Different workloads have different generation characteristics. We separately measured coding, prose generation, and passage copying with DFlash2 enabled.

Higher is better, tokens/sec.

Engine	Code generation	Prose generation	Passage copying
TensorSharp	339	174	190
llama.cpp	321	160	158
vLLM	383	189	171
SGLang	368	184	152
velox	346	184	175
NInfer	415	209	217

Coding assistants

TensorSharp generated code at 339 tok/s:

• 5.6% faster than llama.cpp
• 11.5% slower than vLLM
• 18.3% slower than NInfer

This makes TensorSharp a competitive option for local code generation, while leaving meaningful room to optimize speculative verification.

Prose generation

TensorSharp achieved 174 tok/s, compared with 160 for llama.cpp and 209 for NInfer.

Passage copying

TensorSharp reached 190 tok/s, ranking second behind NInfer.

These results show why it is useful to benchmark different output patterns rather than relying on a single synthetic generation test.

6. Parallel request performance

Single-request speed is important, but inference servers also need to handle multiple simultaneous users or agent tasks.

Without speculative decoding

Aggregate output throughput, tokens/sec:

Engine	1 request	2 requests	4 requests
TensorSharp	80	137	215
llama.cpp	84	154	251
vLLM	69	116	226
SGLang	79	140	148
velox	69	67	67
NInfer	77	144	257

TensorSharp scaled from 80 tok/s for one request to 215 aggregate tok/s for four requests, a 2.69× throughput increase.

With DFlash2

Engine	4-request aggregate throughput
TensorSharp	445 tok/s
llama.cpp	259 tok/s
vLLM	304 tok/s
SGLang	251 tok/s
velox	303 tok/s
NInfer	488 tok/s

TensorSharp ranked second, achieving:

• 72% higher throughput than llama.cpp
• 46% higher throughput than vLLM
• 47% higher throughput than velox
• Approximately 91% of NInfer’s throughput

This is particularly relevant for .NET applications serving multiple simultaneous chat sessions, coding agents, or background AI tasks.

One caveat: SGLang could only run two active DFlash2 lanes within the 32GB GPU memory limit, so its four-request result includes queued requests.

7. Agentic workflows and function calling

This is one of the most interesting parts of the benchmark for .NET developers.

Rather than only measuring token generation, we tested complete agent-style tasks involving tool calls, structured output, and multiple conversation turns.

All results below use DFlash2.

Lower is better. End-to-end latency in seconds.

Task	TensorSharp	llama.cpp	vLLM	SGLang	velox	NInfer
JSON object	0.281	0.314	0.391	0.304	0.369	0.274
JSON schema	0.309	0.473	0.445	0.360	Failed	0.258
Single function call	0.199	0.218	0.354	0.171	0.383	0.180
3 parallel function calls	0.427	0.484	0.539	0.457	0.680	0.378
Agentic loop (4 turns)	0.748	0.817	1.45	0.791	1.42	0.817

Agentic workflow: TensorSharp finished first

The agentic benchmark simulated a small workflow involving file listing, file reading, and calculator operations.

The engine had to complete the tool interactions and return the correct final result.

TensorSharp finished in 748 ms, compared with:

• 817 ms for llama.cpp
• 1,450 ms for vLLM
• 791 ms for SGLang
• 1,420 ms for velox
• 817 ms for NInfer

This is approximately 1.94× faster than vLLM and 1.90× faster than velox in this particular workflow.

For developers building .NET-based AI agents, this is arguably more meaningful than raw tokens/sec.

An agent’s responsiveness depends on the entire tool-execution loop, not just model decoding.

Structured output correctness

The benchmark also validated output correctness.

Most checks passed, with two notable exceptions:

• velox ignored the strict JSON-schema response format in all three repetitions.
• NInfer with DFlash2 produced an empty JSON key in one of three JSON-object replies.

One additional detail: NInfer’s JSON responses were shorter than those of the other engines, which explains part of its end-to-end latency advantage.

This is why benchmarking structured outputs should include correctness checks, not just latency.

8. Why this matters for .NET

TensorSharp is an open-source project exploring whether we can build a complete local AI inference engine and agent runtime around the .NET ecosystem.

The goal isn’t to wrap a Python inference server behind an HTTP client.

It’s to make .NET a first-class environment for model inference, with native execution backends and agent integration.

The benchmark used TensorSharp’s direct-CUDA backend, so these numbers should not be interpreted as pure managed-C# kernel performance. GPU execution and native kernels remain critical.

But the results demonstrate something useful:

A .NET-based inference stack can compete with established inference frameworks on real workloads, including speculative decoding, concurrent serving, and agentic tool execution.

There is still substantial optimization work to do.

In particular, TensorSharp’s DFlash2 draft-and-verify execution is slower than NInfer’s despite similar draft acceptance rates.

That is an actionable optimization target rather than a fundamental limitation of .NET.

9. Limitations and reproducibility

A few things worth emphasizing:

• All measurements were performed on a single RTX 5090.
• Different engines used different quantized checkpoints and drafter formats.
• The benchmark used greedy decoding with thinking disabled.
• Each scenario had three measured repetitions.
• Differences below approximately 5% should be treated as ties.
• The benchmark did not evaluate more than four concurrent requests, model load time, video, or broader answer quality.
• These are operational performance comparisons, not proof that one programming language is inherently faster than another.

The benchmark uses streamed OpenAI-compatible requests and validates the responses.

The TensorSharp repository includes scripts for running the comparison and generating the performance reports:

• eng/qwen38-engine-servers.sh
• eng/qwen38-engine-bench.py
• eng/qwen38-engine-report.py

GitHub: https://github.com/zhongkaifu/TensorSharp

10. Discussion

I’m the author of TensorSharp, and I’d be interested in feedback from the .NET community.

In particular:

• Would you consider running local LLM inference directly inside a .NET application instead of depending on a separate Python-based inference service?
• For your workloads, which matters most: TTFT, prefill throughput, decode speed, concurrent throughput, or end-to-end agent latency?
• What would make a native .NET inference engine genuinely useful in production?

I’m also interested in suggestions for improving DFlash2 verification performance and scaling beyond four concurrent requests.

The project is open source, and contributions, benchmark reproductions, and technical feedback are very welcome.

https://github.com/zhongkaifu/TensorSharp

I recently published some updates to HttpResponseTransformer, which is a NuGet package I originally published over a year ago. I’ve been meaning to share it ever since, and with the update, this seems like a good time.

The library overall lets you define targeted transform rules for outgoing HTTP responses, like manipulating raw bytes, text, and HTML. For example: csharp services.AddResponseTransformer(builder => builder .TransformDocument(document => document .When(ctx => ctx.Request.Path == "/") .InjectScript(script => script .FromUrl("/js/custom.js") .At("//head")))); The new version adds support for image transforms and caching.

The original impetus was to support a set of Jellyfin plugins called Jellyfin PowerToys. Although Jellyfin is extendable, the support for extending the UI is pretty poor. There are some community hacks, but I wanted a simple, reliable way to do it. I've found more uses for it since then; for instance, I used it to build the UI extension framework in Magpie.

I’m really happy with how the API turned out in particular. Hopefully someone else will find it useful too, let me know what you think.

With .NET 8 and .NET 9 both ending support on November 10, a lot of teams are doing the LTS-to-LTS jump right now. I went through the official breaking-change lists and wrote up the ones most likely to hit a typical Web API.

The build-time ones are annoying but obvious. These are the ones that aren't:

• Exception diagnostics are suppressed when IExceptionHandler.TryHandleAsync returns true. No ExceptionHandlerMiddleware log line, no error.type tag on http.server.request.duration. If an alert counts those logs, it goes quiet while the 500s continue. Log inside your handler, or set ExceptionHandlerOptions.SuppressDiagnosticsCallback = _ => false.
• Cookie auth no longer redirects known API endpoints ([ApiController], JSON minimal APIs, TypedResults, SignalR). You get 401/403 with the Location header still set. [AllowCookieRedirect] or .AllowCookieRedirect() opts an endpoint back in.
• EF Core 10 sends collections as multiple parameters (IN (@ids1, @ids2, ...) instead of OPENJSON), and parameter names changed (@status instead of @__status_0). New SQL text means every cached plan recompiles once after deploy. UseParameterizedCollectionMode or EF.Parameter() brings back the old shape for the app or for one query.
• Default container images are Ubuntu now, and .NET 10 ships no Debian images. aspnet:10.0-bookworm-slim simply doesn't exist.
• **dotnet restore audits transitive packages**, so NU1903 can show up for packages you never referenced. With TreatWarningsAsErrors, that fails the build.

The loud ones, for completeness: KnownNetworks → KnownIPNetworks (ASPDEPR005), Swashbuckle 10 on Microsoft.OpenApi 2.x, ExecuteUpdateAsync taking a plain lambda, and NU1510.

Two tips from the comments when I shared this elsewhere: if you'd rather keep the old exception telemetry, SuppressDiagnosticsCallback restores it everywhere; and make your pipeline read the SDK from global.json (UseDotNet@2 with useGlobalJson: true in Azure DevOps, global-json-file in GitHub Actions) so CI can't drift from local builds.

I wrote it up as a case study with before/after code and a checklist. Full disclosure: it's my blog, and the scenario is a composite built from Microsoft's documented changes, not a single real incident.

https://www.softinbit.com/dotnet-field-guide/upgrading-dotnet-8-to-10

What else bit you on the way to 10?