Keeping five dataframes...: we made one of them the reference and pinned every disagreement with a test

[u/coldoven]

I maintain mloda-registry, an open-source set of data operations (aggregations, windows, ranks, resampling) that run on PyArrow, pandas, Polars, DuckDB and SQLite. The hardest part is not writing five implementations. It is getting five engines to agree on edge cases.


Same tiny inputs, native defaults:


    engine    median     sum        max        sum           groups for
              [1, NaN]   [1, NaN]   [1, NaN]   [null, null]  keys 0.0, -0.0
    PyArrow   1.0        NaN        1.0        null          2
    pandas    1.0        1.0        1.0        0.0           1
    Polars    NaN        NaN        1.0        0.0           1
    DuckDB    NaN        NaN        NaN        null          1
    SQLite    no median  1.0        1.0        null          1


What we settled on:


- One engine is the reference (PyArrow), instead of a written spec that nobody can run.
- Every backend runs the same tests and has to match the reference.
- Where an engine can match, the implementation is fixed. Polars' all-null sum returns null in our code, not 0.
- Where it can't (pandas and SQLite can't keep NaN apart from null), the difference goes on a list, each entry with a test that pins it. A contributor who changes the behavior gets a failing test.
- When the reference is the odd one out (only PyArrow splits 0.0 and -0.0), it stays the reference. Picking a new reference per edge case brings the drift back.


The list and the tests are in the first comment. If you maintain something with several backends: how do you keep them in line?I maintain mloda-registry, an open-source set of data operations (aggregations, windows, ranks, resampling) that run on PyArrow, pandas, Polars, DuckDB and SQLite. The hardest part is not writing five implementations. It is getting five engines to agree on edge cases.


Same tiny inputs, native defaults:


    engine    median     sum        max        sum           groups for
              [1, NaN]   [1, NaN]   [1, NaN]   [null, null]  keys 0.0, -0.0
    PyArrow   1.0        NaN        1.0        null          2
    pandas    1.0        1.0        1.0        0.0           1
    Polars    NaN        NaN        1.0        0.0           1
    DuckDB    NaN        NaN        NaN        null          1
    SQLite    no median  1.0        1.0        null          1


What we settled on:


- One engine is the reference (PyArrow), instead of a written spec that nobody can run.
- Every backend runs the same tests and has to match the reference.
- Where an engine can match, the implementation is fixed. Polars' all-null sum returns null in our code, not 0.
- Where it can't (pandas and SQLite can't keep NaN apart from null), the difference goes on a list, each entry with a test that pins it. A contributor who changes the behavior gets a failing test.
- When the reference is the odd one out (only PyArrow splits 0.0 and -0.0), it stays the reference. Picking a new reference per edge case brings the drift back.


The list and the tests are in the first comment. If you maintain something with several backends: how do you keep them in line?

[u/coldoven]

The list, each case with its test: https://github.com/mloda-ai/mloda-registry/blob/main/docs/guides/data-operation-patterns/known-divergences.md

Why PyArrow is the reference: https://github.com/mloda-ai/mloda-registry/blob/main/docs/guides/data-operation-patterns/03-reference-implementation.md

[u/garros-yu]

the reference implementation is what makes this tractable at all. in my testing the disagreements split into real bugs and engines just defining edge cases differently, null ordering and float tolerance mostly, and the suite stays readable only when each divergence gets pinned as intended. generate the ugly cases on purpose too, empty frames, all null columns, duplicate join keys, that's where five engines give five answers fastest imo