How a single contiguous allocation — and a type system that won't let you feed strings to a scaler — is the real reason datarust fits in 2.3 megabytes.
In the last post I showed you the whole datarust workflow: impute, scale, one-hot, train a logistic regression, evaluate, and save it as JSON — all without a Python runtime in sight.
The Docker image shrank from ~900 MB to ~8 MB, and the binary was 2.3 MB.
But I skimmed over something important.
I kept saying "the flat memory layout" as if it were a detail.
It isn't.
It's the whole bet.
Every scaler, every encoder, every model, every metric in datarust runs on top of one data structure.
If you understand that structure — why it looks the way it does and what it refuses to let you do — the rest of the library stops being magic.
So let's zoom in.
Meet .
Two containers, on purpose Real data is mixed.
Numbers in one column, strings in the next.
In Python, everything flows through one giant or a , and the type system just... shrugs.
A string column next to a float column gets coerced into dtype.
You'll find out at training time, in the form of an error message three frames deep. datarust does the opposite.
It splits your data into two types at the source: is only. is strings only.
They are different types, and the compiler will refuse to compile a program that hands a string column to a scaler.
Not at runtime — at compile time.
In the last post I called this "putting on glasses for the first time." Let me show you what it actually buys you.
The API is built on that split: A expects strings.
The type system makes that a compile error on a numeric column. sklearn can't do this — every column name is just a string, and will happily run on floats if you don't read the docs carefully.
Construction is a lie detector Here's the thing that surprised me most when I actually read the source: doesn't just store your data.
It validates it.
This returns an error: Not a panic.
Not silently appearing in column
3.
Not a jagged array that flows downstream and poisons your model.
A precise, recoverable error, at the moment of construction.
In pandas, a ragged column silently becomes dtype.
In numpy, you get a cryptic error deep in some array-conversion path.
In datarust, the constructor is the bouncer, and it's the only place the check needs to happen.
From then on, every function that takes a can trust the shape without re-checking.
There's a matching for when you already have a contiguous buffer, and it validates the element count matches the declared shape: Even the allocation is checked: — so you can't ask for a matrix so large that overflows and wraps around into a buffer too small.
I know that sounds paranoid.
It's also the difference between a library that panics in production and one that returns .
The one allocation Now for the part I waved my hands at last time.
Here's the internal layout: That's it.
Not — no allocation per row, no pointers-to-pointers.
Element lives at .
Your 50,000×200 matrix is one contiguous of ten million floats, in one heap allocation.
Why does that matter?
Three reasons: Cache locality.
When a scaler walks a row, every element is adjacent in memory.
With , each row is a separate allocation, and walking rows means jumping between unrelated memory pages.
Auto-vectorization. hands the compiler a plain .
Modern CPUs get to use SIMD, and LLVM gets to prove the loop has no aliasing.
The Python boundary doesn't exist.
This is the quiet killer.
In a numpy pipeline, every crosses from Python into C and back, materializing Python float objects for the results.
In datarust, is a Rust function operating on a Rust buffer.
The whole pipeline never leaves native code.
This wasn't always the case.
Pre-0.3, was .
When we switched to the flat layout, the same workloads got dramatically faster with no algorithm changes: Workload (50,000 × 200) (v0.2) flat (v0.3, default) 115 ms 8.4 ms 81 ms 12.2 ms 459 ms 137 ms Pipeline (3 scalers) 662 ms 152 ms PCA 1056 ms 1008 ms 88 ms 98 ms got 3.4× faster, 4.4×.
Same math, same data — only the memory layout changed.
Flat beats fancy.
The hot loops are designed around that layout.
You almost never call in an inner loop; you grab a row slice and iterate: returns a slice, and exposes the whole buffer for the tightest loops. exists for the times you need it — and it's bounds-checked in release mode, with a precise panic message.
If you want to be defensive, returns instead.
NaN is a type problem, so we treat it like one Python has a lovely tradition of letting flow through a pipeline until it silently infects your coefficients.
You train, your model returns , and you spend an afternoon bisecting which column did it. datarust's answer is three validators, each with a distinct contract: The subtle one is the middle: .
Imputers like and are allowed to see — that's the missing-value marker they exist to fix.
But an infinity is not a missing value, it's a broken number.
So imputers call , not .
The distinction is baked into the API, and it means the error message tells you which kind of dirty data you're dealing with.
When the validation fails, the error tells you exactly where: Row 2, column
0.
Not "somewhere." You go straight to the bad cell.
And this is exactly why the preprocessing order matters.
The in the previous post had a missing tenure value in row 2 — a that a scaler would refuse to touch.
You can't skip the imputer and hope the scaler tolerates it: Without the imputer, that last line fails loudly — — instead of silently training a model on broken numbers.
The validation isn't there to annoy you.
It's there to force the pipeline to be honest about what it handles.
The sparse sibling One-hot encoding a high-cardinality column produces a lot of zeros.
Storing all of them as is waste.
So there's a third container: , a Compressed Sparse Row matrix mirroring .
Notice something: I passed twice. datarust sums duplicate coordinates (), drops entries that sum to zero, and sorts each row by column index — so the invariants a reader expects are always true.
And if you hand it a malformed CSR array, it rejects it with a specific message instead of indexing out of bounds later.
The constructor validates , column ranges, and per-row ordering.
Bad data is rejected here, not at the model.
What it doesn't do (yet) Honesty section, because the tone of these posts is "here's what I learned," not "buy my perfect library." is not flat.
It's still , one heap allocation per string.
It's fine for categorical columns (which are wide and short), but it's a deliberate asymmetry, and it shows: string-heavy workloads don't get the same cache wins as numeric ones.
No views. and copy.
In numpy you'd get a view with fancy indexing; here, correctness and simplicity win over aliasing.
No .
It's everywhere.
An mode would halve memory for big datasets, but it's a sweeping refactor across every estimator.
It's on the roadmap, not in the crate.
No NumPy interop.
The feature preserves a nested-JSON wire format for , so fitted pipelines round-trip through / — but there's no reader yet.
That's explicitly listed as under consideration.
Why this is the core feature Every number in every example in the first article passed through .
The 179–620× speedups against sklearn's ?
Partly the flat buffer, partly never crossing the Python/C boundary.
The type-safety that caught a on a numeric column before it could run?
That's vs being different types.
The JSON model that loads in a WASM module or an ARM embedded target?
That's a buffer you can serialize and a shape you can trust.
In other words: the 400× Docker shrink isn't the story.
The one that makes it possible — and the constructor that refuses to let you put garbage in it — is the story.
If you want to see the data structure that everything else is built on, the full source is in — 1,400 lines including tests, and the tests are where the invariants are documented.
Or just run the pipeline from the last post and know that every number in it started as a flat buff