How do you make a programming language memory safe without paying a lot in overhead?

[u/AeroForger]

I am designing a language. I want memory safety without using a tracing garbage collector or having heavy runtime costs. I have been looking at the options and I would like to know how others have dealt with these tradeoffs.

The options I have looked at:

- Ownership and borrow checking (Rust-style): Zero runtime cost but the compiler must prove things and the learning curve and syntax friction are real. I am not a fan of Rusts syntax so I wonder if the same guarantees can be provided with a surface.

- Reference counting (Swift/Nim-style): Predictable and simple but you pay for increments and decrements on every copy and cycles need either weak references or a cycle collector.

- Tracing GC: Easy for the user. Pause times and memory overhead are the usual complaints. It is a fit if the language is meant for games or system work.

- Regions/arenas with escape analysis: Allocate in scopes and let the compiler prove that nothing escapes. Very cheap,. I am not sure how far escape analysis can go before it becomes too restrictive.

- Bounds checks and tagged or hardware‑assisted checks (like ARM MTE): Catch many errors at low cost but they are mitigations, not guarantees.

My current thinking is a mix arenas by default an ownership model for long‑lived values and bounds checks that can be elided when the compiler can prove them safe. I would like to know whether that is a direction or whether I am missing a well‑known approach.

Specific questions:

has anyone implemented escape analysis that stays useful without forcing annotations?

Is there a middle ground between full borrow checking and reference counting that does not reintroduce cycle problems?

for those who have shipped a memory‑safe language with overhead what would you do differently?

Links, to papers, languages or post‑mortems are very welcome :)

[u/rafaelRiv15]

That is a whole field of research

[u/rantingpug]

Algebraic effects, fbip, perceus... Just a few things to add to that list. Check out koka.

But yeah, that question is a bit of a thorny one

[u/matthieum]

Bounds checks and tagged or hardware‑assisted checks (like ARM MTE): Catch many errors at low cost but they are mitigations, not guarantees.

I just want to note that bounds checks are a different beast altogether.

That is, all the above options are about time (use after free...) whereas bounds-check are about space. As such, they are completely orthogonal with the rest.

In practice, I would argue that you should not worry overmuch about bounds-checks. Rust developers are extremely performance conscious, and yet Rust has mandatory bounds-checks (in safe code), which the compilers mostly manage to eliminate during optimizations.

For compile-time guarantees of absence of bounds-checks...

A possibility is branded indexes/keys. Brands are the idea of linking a variable to another. For example, something like let vec = Vec::new(); let len: 'vec usize = vec.len(); where 'vec is a "brand" which ties len to vec. You can then require that vec[] only accept indexes branded with 'vec and no other. But you still need to develop a whole way of deriving valid indexes, etc... This is related to dependent types, and with immutable values it should be feasible? NOT too simple, though.

I think I would prefer a (seemingly?) lower-key yet generic solution: pre-conditions, post-conditions, and invariants. You'd annotate Vec::at with @requires(i < self.len()).

What I like most about pre-conditions, post-conditions, and invariants is that you can flexibly enforce them at either compile-time or run-time:

• Performance sensitive function? Slap @comptime(invariants) on it, and be assured that if it compiles, there's no run-time check.

• Otherwise? Just generate the code smartly:

  • Pre-conditions insert an assert in the caller (before the call), which has the context on how those arguments were generated, and an assume in the callee (at the top).

  • Post-conditions insert an assert in the callee (before return), which has the context on how this result was generated, and an assume in the caller (after the call).

  • The above give the optimizer: maximum ability to eliminate the asserts, and maximum ability to take advantage of the assumes.

Also, your front-end may attempt to optimize the assert out by itself. After all, it's already capable of proving those conditions when required to, so it might as well try -- with a bounded budget -- even when not ask to, at high enough optimization levels. It may, after all, have more context than the backend it hands the code over to at the end.

And as a bonus, those pre-conditions, post-conditions, and invariants can also be used by static analysis tools, AI, and whatever to spot bugs. It just keeps on giving!

[u/KATO-Kanryu]

The mistake many language designers make is searching for a single silver bullet (e.g., forcing a 100% borrow-checker or a heavy tracing GC onto every single byte).

In my own compiled systems language (Hike), I took a pragmatic approach by providing distinct, orthogonal memory primitives matched to specific workloads:

1. Short-lived bulk allocations (The Arena Pattern): Inspired by Odin, we use explicit area(...) { ... } blocks. Everything allocated inside is bump-allocated and bulk-freed in O(1) upon exiting the block. The compiler strictly forbids referencing internal variables outside the block; anything that must outlive the area must be explicitly moved via a built-in deepcopy(...). This completely avoids Use-After-Free without complex lifetime annotations.

2. Transient slice/string data: Substrings and subslices share backing buffers. Instead of per-variable tracking, reference counting is performed per buffer. Combined with Copy-on-Write, this eliminates redundant allocations while ensuring safe, predictable cleanup.

3. Cross-thread long-lived state: We provide threadable (thread-local isolated storage) and concurrent (heap variables with compiler-enforced mutual exclusion/atomic access) to prevent data races at the language level.

4. Zero-cost multi-gigabyte structures (mmap): Variables can be directly backed by memory-mapped files. Large datasets or binary models are flushed to a single binary and mapped straight from the OS filesystem, giving near-instant initialization for multi-gigabyte heaps with zero parsing overhead.

5. Foreign FFI boundaries (cbuffer): To handle pointers from external C/Rust libraries safely, we provide an owned cbuffer type. It holds the native pointer, size, and custom release callback, and remains strictly valid until the user explicitly invokes .Close().

By separating scoped scratchpad memory (area) from owned buffer types (cbuffer) and buffer-level RC, you don't need a garbage collector, and you don't need to fight a borrow checker.

[u/Moogly2021 → u/KATO-Kanryu]

I went with the arena pattern and also function annotations that can force the behavior per function, so you can go as crazy as you want when you need it.

[u/TemperOfficial]

In principle, it's very easy. Every time you access memory you check if that access is valid.

Easiest way to solve that is with indirect memory access. i.e. all memory access goes through a handle that is checked.

Simple way is have code blocks that allow dereferencing of pointers versus ones that don't. In the ones that don't, you have indirect access via a handle, and the process of getting memory back from that handle involves a check.

This is what everyone does in Rust to get around the borrow checker.

This is what old C llibs do. OpenGL for example, although the API isn't great.

The primary issue is contextually what is valid memory?

[u/DrShocker → u/TemperOfficial]

What I do is simply map the entire address space to one byte and use it for everything.

[u/TemperOfficial → u/DrShocker]

what if you need to use two bytes?

[u/brat3108 → u/TemperOfficial]

In principle, it's very easy. Every time you access memory you check if that access is valid.

And if it isn't, then what? You can't really have such failures in production code. It would be a bug that needs to be fixed.

The primary issue is contextually what is valid memory?

Or what does 'memory safety' actually mean?

[u/DrShocker → u/brat3108]

I've seen different arguments for returning an error/throwing an error vs returning a nil. It's just an engineering decision what to do. My personal preference is likely to just crash because if they're trying to access invalid memory there's not much to do that's valid, and crashes get fixed whereas errors logged often don't in my experience.

[u/TemperOfficial → u/brat3108]

> And if it isn't, then what? You can't really have such failures in production code. It would be a bug that needs to be fixed.

Depends on the callers needs. Either, abort or return some error value.

'memory safety' yes this is a good question because ultimately its subjective what constitutes "valid" memory.

[u/pthierry]

A copying GC can also gave great performance, take a look at the recent research on the subject.

I know about GHC's low latency GC, for example: Using GHC low-latency garbage collection in production

[u/flatfinger → u/pthierry]

One thing I'd like to see in a GC framework would be a category of semantically immutable objects that have a telescoping "superior equivalent" pointer, which for each object would be initially set to itself, and may at any arbitrary time have references to an object become references to the superior equivalent. If two semantically immutable objects are compared and found to be transitively equivalent, and there exists a way of ranking references, then the "superior equivalent" for both objects could be set to whichever of the object's superior equivalent ranked above the other. The net effect would be that comparisons would build cliques of objects that were known to be equivalent, and the GC would tend to change references to all such objects into references to one common superior object.

[u/pthierry → u/flatfinger]

Do you have an example of what a "superior equivalent" could be?

[u/sunxore]

Check out goose programming language. Super fast and memory safe. Early days yet though so remains to be seen if it is a viable model.

[u/Business-Weather-217]

Two big components here:

1. Checks are only in contract (eg. C++ profiles)

2. Static anaylsis - this is the hard one and this is what rust does, for you basic version of it would suffice but it goes such deep that it might have occure a struggle.

[u/ParsingError]

Ref counting is also difficult to deal with in multithreaded situations because accessing the pointer and changing the reference count are not a single atomic operation, so if you allow contended access to memory containing an object reference, you can wind up with situations where one thread is trying to overwrite the last reference to an object while another is reading it, and the overwriting thread decrements the refcount to zero and destroys it before the reading thread can increment the count.

I'm not sure if there's really a good way of dealing with that other than either preventing read/write contention through access restrictions, or something gnarly like deferred deletion checkpoints.

[u/Y_mc]

I think there needs to be a compromise between memory safety and overhead.

[u/leosmi_ajutar]

You going to support this language for longer than 2 months like your last one?

I stopped maintaining my language, Sydrogen. The code and book are still up if you want to look around : r/Compilers

[u/MitchAlsup]

Thre are some things better left to the OS ...

For example, I was spending too much time new-ing and ~destructing in an application. After spending way to much time trying to optimize those things, I came across a different solution. And that was: To package up a whole request, fork off a thread/process to perform that unit of work new-ing only. Then returning the vastly shortened result to spawner and letting OS clean up the memory allocation problem in the background ...

This works because OS clean up by pages, while application clean up by allocation AND there can never be a use-after-free problem.