Debunking "Faster than Rust": Malicious Benchmarks in Tov

[u/madonuko]

https://blog.fyralabs.com/debunking-faster-than-rust-malicious-benchmarks-in-tov/

I have debunked (some of the) benchmarks of this new and suspicious "Tov" programming language.

Slight background: My friend (who is pretty interested in new stuff generally) found this thing and sent its main site to me saying things like "looks pretty cool", but my intuition told me this sounds like "a scam", esp. when it claimed it performs faster than C and Rust.

I personally think apart from changing the code, they probably tried to set up some specific environments that make their programs run faster than C / Rust, or maybe they just made up the numbers. Who knows.

[u/a__nice__tnetennba]

How is something that is C faster than C? That's a bizarre claim regardless of how it actually compares to Rust directly.

[u/madonuko → u/a__nice__tnetennba]

I think the saddest part is they managed to create this software that can transpile JS/TS to C, and they decided that promoting this new technology in such a deceiving way was a good idea. If you look at the nature of the program (transpiler) itself, it's probably not the worst thing out there.

[u/chucker23n → u/madonuko]

they decided that promoting this new technology in such a deceiving way was a good idea

Engagement bait. Seems it's been effective, given that we're discussing them.

[u/taw → u/a__nice__tnetennba]

Very easily. Human-written C that's required to follow C ABIs everywhere, and compiled-to-C are nothing alike.

Cross compiler can make tons of assumptions that C cannot, and aggressive optimizations based on those assumptions. For example you can skip a lot of memory management code and just bump allocate half the stuff on stack. C compiler can't do that, as C source code doesn't have necessary information, but your source language could do that.

You can hand write some abomination in C that would match cross compiler in speed, but it really wouldn't look like C.

[u/gnufan → u/taw]

This was very obvious when I did performance computing, and you needed to avoid anything that made optimisation harder for the compiler, usually any unnecessary indirection, any type weirdness, and any alignment issues in memory, no that stuff is going in an array aligned with word boundaries until it is output in whatever bizarre format the client requires. Compilers have gotten smarter but the same types of things still make optimisation harder.

[u/knome → u/taw]

if you're only talking about idiomatic code layout, fine, but to say a C compiler can't do what compiling into C and then using a C compiler can do is a very odd way to phrase it.

[u/cdb_11 → u/taw]

For example you can skip a lot of memory management code and just bump allocate half the stuff on stack.

I don't understand this. I can use the stack or even write a separate bump allocator in C just fine? This is exactly why manual memory management is useful. Not having to rely on fragile compiler optimizations is a good thing, as far as I'm concerned. Because, as you said, "source code doesn't have necessary information". You instead provide this information by actually writing the code that does what you want.

but it really wouldn't look like C.

It'd be a perfectly normal C. Again, I don't get it. Is people idea of C having a malloc on every third line or something? By the way, not that you should be relying on this, but a fun fact is that C compilers can optimize out malloc and free.

[u/simspelaaja → u/a__nice__tnetennba]

Well, it depends on what you're actually comparing. In a real world scenario machine generated C can beat hand-written C, because C makes it very hard to build efficient reusable abstractions due to lack of generics / templates.

It's common in C to use linked lists (which are almost always really inefficient compared to vectors / resizable arrays), or to use linear scans over arrays instead of hashmaps and sets. C's standard library doesn't include any containers, while Rust and C++ have a wide selection of containers for different use cases. And Rust and C++ will usually generate near-optimal machine code for each container instance, while C code is usually more pointer & dynamic allocation heavy by necessity. The qsort function in C calls the comparator function via a function pointer, while in C++ std::sort usually just inlines it.

The compiler can still in some situations optimize the C code to be near-equivalent, and there are tricks that can be done with macros and compiler extensions to emulate generics, but in general reaching Rust/C++ level of performance in C requires a lot of repetition. A lot of the time people just won't do it.

[u/Rechenplaner → u/a__nice__tnetennba]

Tov compiles to C and counts references instead of collecting garbage, so there are no collector pauses and the binaries stay small.

from the "Toc" page. I consider that extremely bold. Swift also employs reference counting, and it is often slower than garbage collection (GC) due to the overhead which becomes a major disadvantage when there are many allocations unlike highly optimized GCs, which handle mass allocations pretty well.

[u/KaiAusBerlin → u/a__nice__tnetennba]

At the end it's not the programming language that matters. All that matters is the compiler. The proceccor doesn't run a language. It runs bytecode.

A system near language just makes it easier to write a more optimised compiler.

[u/cdb_11]

All language benchmarks should be taken with a huge grain of salt, just in general.

[u/the3gs]

I wouldn't think to use ahash over std::collections::HashMap unless I very specifically was dealing with a hashmap related bottleneck, and even then I would have to research to find ahash, so I think it is unfair to say "they deliberately used the standard library instead of a library to slow it down".

[u/moefh → u/the3gs]

You wouldn't think you're hitting a hashmap-related bottleneck in a benchmark named "map_insert"?

[u/Ok_Mycologist_6604]

basically with AI we got rebranded crypto bros into the industry and I am sick and tired of it, there is no solution for it. they will take ur job with the slop until a plane crashes and people maybe re-evaluate.

[u/my-new-new-account → u/Ok_Mycologist_6604]

This is funny, because most of Rust jobs are in crypto

[u/stumpychubbins]

From their pitch, I would not be surprised if what they’re actually benchmarking between languages is the code produced by default by an AI coding agent if you ask it to solve the stated problem in the given language. An AI will not use the ahash library, so they don’t use the ahash library in their benchmarks, and for them that constitutes an accurate benchmark in their eyes.

[u/YaBoyMax → u/stumpychubbins]

I see where you're coming from, but I think that's overly charitable. I fed the benchmark description verbatim to free ChatGPT with the prompt:

Implement the following program in Rust. Optimize for maximum performance.

It immediately used ahash without further prompting. I'm not discounting that using HashMap could be an honest mistake for that particular case, but I don't think they designed the benchmarks explicitly with LLMs in mind for other languages. Even if they had, that should have been clearly disclosed.

[u/stumpychubbins → u/YaBoyMax]

Well, that’s certainly damning. I wasn’t really giving them credit, I still think it’s a terrible way to handle benchmarks, I just thought there was a possibility of it not being malicious. It sounds like even with that bad methodology they still should’ve ended up with better results than they did, though

[u/CrasseMaximum]

Remind me of V

[u/bwainfweeze]

If I’m recalling the anecdote correctly, Brian Cantrell tried to solve a problem in Rust and found that it was much faster than C++ in a very surprising manner.

When he dug into the code he found that the Rust solution was using a B-tree under the hood. He said that made sense because it was easier to write a correct implementation in Rust and C++ would use something more conservative for correctness.

Moral of the story is, when a new language shows up, the test is not if you can do a thing that language does in the old language. It’s whether you will do that thing. The same way the first person I recall predicting the future success of smart phones over laptops. You can take a laptop with you, but you will take a pocket computer with you.

So what I’m looking for in a new language is if it supports better semantics and/or data structures that nobody attempts in my status quo language. Such benchmarks are “unfair” but still completely apropos. It is in fact the entire point of writing something new instead of using what exists.

[u/cdb_11 → u/bwainfweeze]

When he dug into the code he found that the Rust solution was using a B-tree under the hood. He said that made sense because it was easier to write a correct implementation in Rust and C++ would use something more conservative for correctness.

The ordered set in Rust's standard library is just implemented as a B-tree, using unsafe. C++ uses red-black trees. Not because they are particularly easier to implement, but because the standard requires iterator stability. And this rules out using a B-tree. Standardizing these mostly useless guarantees was just a historic mistake, the same way STL's hash map was. Keep in mind that this is just what the standard library does. People do use B-trees in C++.

My alternative moral of the story is that the interface can lock you out of optimizations, and that once the interface gets adopted it can be hard or impossible to change it.

[u/HatProud9045]

Worth noting they probably disabled CPU throttling too. Classic benchmark trick.

[u/carsncode → u/HatProud9045]

That's not a trick - unless throttling is the thing you're measuring, leaving it on for a benchmark only makes results less informative

[u/Gio_Cri]

at a first glance their benchmarks actually seem comparable for small toy projects with naive implementations. would be interesting to see a comparison on what it's like on larger projects, personally i expect the performance andvantage and code ease to be negated quickly with project size

[u/bwainfweeze]

It is very hard to get clean benchmarks and confirmation bias is everywhere. If the first run supports your claim people tend to declare victory and move on.

In some benchmarking system it can be whoever goes first who runs faster. Or sometimes whoever goes second, particularly in JITs. It’s not an always thing but when you do something repeatedly and rely on it, it doesn’t take that many false positives to look like a numpty.