IndexScalarQuantizer.merge_from() accepts incompatible trained ranges and silently decodes merged vectors incorrectly

Author: leemeiiCreated Sep 7, 2026Updated Sep 7, 2026

Environment:

- FAISS version: 1.15.0
- Python package: faiss-cpu==1.15.0

Description:

IndexScalarQuantizer.merge_from() accepts two Scalar Quantizer indexes whose trained quantization ranges are incompatible.

The destination index is trained on values in [0, 1], while the source index is trained on values in [100, 101]. Each index stores its midpoint.

Calling:

    destination.merge_from(source, 0)

succeeds and empties the source index. However, the moved source vector is subsequently decoded using the destination index's quantization state.

The source vector reconstructs as:

    Before merge: [100.5, 100.5]
    After merge:  [0.5, 0.5]

A control case using indexes trained on the same range preserves the reconstruction correctly.

This is not ordinary quantization error:

- The source index reconstructs correctly and consistently before the merge.
- The raw encoded bytes are unchanged.
- The reconstruction changes only after the code is moved into an index with incompatible trained state.
- The source emptiness and destination cardinality invariants remain correct.

Minimal reproduction:

python

    import faiss
    import numpy as np

    d = 2

    destination = faiss.IndexScalarQuantizer(
        d, faiss.ScalarQuantizer.QT_8bit
    )
    source = faiss.IndexScalarQuantizer(
        d, faiss.ScalarQuantizer.QT_8bit
    )

    destination.train(
        np.array([[0.0, 0.0], [1.0, 1.0]], dtype="float32")
    )
    source.train(
        np.array([[100.0, 100.0], [101.0, 101.0]], dtype="float32")
    )

    destination.add(
        np.array([[0.5, 0.5]], dtype="float32")
    )
    source.add(
        np.array([[100.5, 100.5]], dtype="float32")
    )

    before = source.reconstruct(0)

    destination.merge_from(source, 0)

    after = destination.reconstruct(1)

    print("before:", before)
    print("after:", after)
    print("source.ntotal:", source.ntotal)
    print("destination.ntotal:", destination.ntotal)

Observed output:

    before: [100.5, 100.5]
    after: [0.5, 0.5]
    source.ntotal: 0
    destination.ntotal: 2

Expected behavior:

`merge_from()` should reject indexes with incompatible trained quantization state by raising a clear exception instead of silently merging them.

Compatibility should include at least:

- dimension;
- metric;
- quantizer type;
- code size;
- Scalar Quantizer configuration;
- trained scale, range, offset, and other decoder state.

Alternatively, IndexScalarQuantizer should provide a stricter override of check_compatible_for_merge().

Root cause:

The compatibility check inherited from IndexFlatCodes verifies structural properties such as the encoded type, dimension, and code size.

IndexScalarQuantizer does not perform a stricter comparison of the trained quantization state. Scalar Quantizer codes are meaningful only together with the scale/offset state learned during training. Appending source code bytes to the destination index causes those bytes to be decoded using the destination's incompatible quantization parameters, producing incorrect reconstructions.

Impact:

This can cause:

- silently corrupted merged indexes;
- incorrect reconstructed vectors;
- incorrect search distances and rankings;
- corruption that is difficult for downstream applications to detect;
- incorrect results after shard merging, persistence, or reload.

Suggested fix:

Implement a stricter IndexScalarQuantizer::check_compatible_for_merge() that compares the complete Scalar Quantizer configuration and trained decoder state.

If incompatible indexes are detected, merge_from() should fail before modifying either index.

Source: facebookresearch/faiss

View original on GitHub View discussion on GitHub