#5590·faiss

IndexFastScan::merge_from silently ignores nonzero add_id

Author: leemeiiCreated Sep 7, 2026Updated Sep 7, 2026

Summary

A public IndexPQFastScan merge accepts add_id=100, successfully moves the source entry, and empties the source index. However, the moved vector is returned under label 1 instead of the required label 100.

This is a silent ID-mapping correctness bug. The vector data and ntotal are correct, but the logical ID of every moved entry can be wrong.

Reproduction

Run:

bash
python3 repro.py --trials 10
python
#!/usr/bin/env python3
"""Fresh-process reproduction for IndexFastScan::merge_from ignoring add_id."""

import argparse
import datetime as dt
import json
from pathlib import Path
import subprocess
import sys

from oracle import check_bug, check_control, expected_bug_contract, expected_control

HERE = Path(__file__).resolve().parent


def find_root():
    for parent in HERE.parents:
        if (parent / "VDBMS/faiss/faiss/IndexFastScan.cpp").is_file():
            return parent
    raise RuntimeError("cannot locate checked-out VDBMSFuzz root")


def now():
    return dt.datetime.now(dt.timezone.utc).replace(microsecond=0).isoformat()


def run_process(binary, mode):
    try:
        p = subprocess.run([str(binary), mode], cwd=str(HERE), capture_output=True,
                           text=True, timeout=30, check=False)
        return {"mode": mode, "returncode": p.returncode, "stdout": p.stdout,
                "stderr": p.stderr, "timed_out": False}
    except subprocess.TimeoutExpired as exc:
        return {"mode": mode, "returncode": None, "stdout": exc.stdout or "",
                "stderr": exc.stderr or "", "timed_out": True}


def write_json(path, value):
    path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n")


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--trials", type=int, default=10)
    args = ap.parse_args()
    if args.trials < 10:
        raise SystemExit("--trials must be at least 10")
    root = find_root()
    build = root / "VDBMS/bug-finder-agent/work/faiss/build-make"
    static_lib = build / "faiss/libfaiss.a"
    binary = HERE / "runtime/repro_helper"
    binary.parent.mkdir(parents=True, exist_ok=True)
    (HERE / "logs").mkdir(parents=True, exist_ok=True)
    compile_cmd = ["g++", "-std=c++17", "-O2", "-I", str(root / "VDBMS/faiss"),
                   "-I", str(build), "-o", str(binary), str(HERE / "repro_helper.cpp"),
                   str(static_lib), "-fopenmp", "/lib/x86_64-linux-gnu/libblas.so.3",
                   "/lib/x86_64-linux-gnu/liblapack.so.3"]
    if not static_lib.is_file():
        print("missing audited FAISS static library", file=sys.stderr)
        return 2
    c = subprocess.run(compile_cmd, cwd=str(root), capture_output=True,
                       text=True, check=False)
    write_json(HERE / "runtime/compile.json", {"command": compile_cmd,
        "returncode": c.returncode, "stdout": c.stdout, "stderr": c.stderr})
    if c.returncode != 0:
        return 2
    started = now()
    control = run_process(binary, "control")
    trials = [run_process(binary, "bug") for _ in range(args.trials)]
    control_ok = check_control(control["returncode"], control["stdout"])
    checks = [{**t, "oracle_match": check_bug(t["returncode"], t["stdout"]),
               "fresh_process": True, "fresh_in_memory_graph": True} for t in trials]
    bug_ok = all(t["oracle_match"] for t in checks)
    finished = now()
    raw = {"started_at": started, "finished_at": finished, "control": control,
           "trials": checks}
    normalized = {"control_pass": control_ok, "trial_count": len(checks),
                  "trial_pass_count": sum(t["oracle_match"] for t in checks),
                  "all_trials_match_bug_oracle": bug_ok,
                  "expected_control": expected_control(),
                  "expected_bug_contract": expected_bug_contract(),
                  "fresh_process_per_trial": True,
                  "fresh_in_memory_graph_per_trial": True}
    schedule = {"kind": "deterministic_fresh_process_stability", "required_trials": 10,
                "actual_trials": len(checks), "control_runs": 1,
                "data_directories": "none; each process creates fresh in-memory indexes",
                "started_at": started, "finished_at": finished}
    write_json(HERE / "raw-responses.json", raw)
    write_json(HERE / "normalized-results.json", normalized)
    write_json(HERE / "trial-results.json", checks)
    write_json(HERE / "schedule.json", schedule)
    (HERE / "logs/stability.log").write_text(
        "started_at=" + started + "\ncompile_returncode=" + str(c.returncode) + "\n" +
        "control=" + json.dumps(control, sort_keys=True) + "\n" +
        "\n".join("trial_%d=" % (i + 1) + json.dumps(x, sort_keys=True)
                   for i, x in enumerate(checks)) + "\nfinished_at=" + finished + "\n")
    if not control_ok or not bug_ok:
        print("oracle failed", file=sys.stderr)
        return 3
    print(json.dumps(normalized, sort_keys=True))
    return 10


if __name__ == "__main__":
    raise SystemExit(main())

The destination index initially contains one vector. The source index contains one vector. After calling:

cpp
destination.merge_from(source, 100);

the query exactly matches the moved source vector.

Observed control result:

add_id=0 ntotal=2 labels=1,0 source=0

Expected nonzero-offset result:

add_id=100 ntotal=2 labels=100,0 source=0

Actual result:

add_id=100 ntotal=2 labels=1,0 source=0

All ten fresh-process trials reproduce the incorrect label.

Root Cause

IndexFastScan::merge_from declares the parameter as unused:

cpp
void IndexFastScan::merge_from(Index& otherIndex, idx_t /*add_id*/)

It copies each source code into the destination position ntotal + i, increments ntotal, and resets the source index. It never stores or applies the requested ID offset.

Because FastScan uses implicit sequential labels, the physical destination position 1 is returned as label 1. The required logical label should be:

source_id + add_id = 0 + 100 = 100

The public Index::merge_from contract states that add_id is added to all moved IDs:

FAISS Index contract

The IndexFastScan API also documents add_id as the ID offset for merged vectors:

FAISS IndexFastScan API

Why This Is a Bug

This is not an approximate-search recall difference. The query matches the moved vector, and the failure is the returned identifier.

If nonzero offsets cannot be represented by this sequential storage format, the method should reject them explicitly, as IndexFlatCodes::merge_from does. Silently accepting and ignoring the argument violates the public API contract.

[IndexFlatCodes implementation](https://github.com/facebookresearch/faiss/blob/main/faiss/IndexFlatCodes.cpp#L99-L108)

Impact

Applications that merge shards using disjoint ID ranges may retrieve the wrong document or metadata record for every moved FastScan entry. The merge reports success, making the problem difficult to detect.

Scope

No metadata, payload, predicate, filter, wrapper, or where behavior is involved.

Deduplication

As of 2026-09-07, no exact public FAISS Issue or PR was found for IndexFastScan::merge_from ignoring a nonzero add_id.

Related but distinct reports include:

The current upstream IndexFastScan.cpp implementation still contains the unused add_id parameter and the same positional copy behavior.