IndexFastScan::merge_from silently ignores nonzero add_id
Summary
A public IndexPQFastScan merge accepts add_id=100, successfully moves the source entry, and empties the source index. However, the moved vector is returned under label 1 instead of the required label 100.
This is a silent ID-mapping correctness bug. The vector data and ntotal are correct, but the logical ID of every moved entry can be wrong.
Reproduction
Run:
python3 repro.py --trials 10#!/usr/bin/env python3
"""Fresh-process reproduction for IndexFastScan::merge_from ignoring add_id."""
import argparse
import datetime as dt
import json
from pathlib import Path
import subprocess
import sys
from oracle import check_bug, check_control, expected_bug_contract, expected_control
HERE = Path(__file__).resolve().parent
def find_root():
for parent in HERE.parents:
if (parent / "VDBMS/faiss/faiss/IndexFastScan.cpp").is_file():
return parent
raise RuntimeError("cannot locate checked-out VDBMSFuzz root")
def now():
return dt.datetime.now(dt.timezone.utc).replace(microsecond=0).isoformat()
def run_process(binary, mode):
try:
p = subprocess.run([str(binary), mode], cwd=str(HERE), capture_output=True,
text=True, timeout=30, check=False)
return {"mode": mode, "returncode": p.returncode, "stdout": p.stdout,
"stderr": p.stderr, "timed_out": False}
except subprocess.TimeoutExpired as exc:
return {"mode": mode, "returncode": None, "stdout": exc.stdout or "",
"stderr": exc.stderr or "", "timed_out": True}
def write_json(path, value):
path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n")
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--trials", type=int, default=10)
args = ap.parse_args()
if args.trials < 10:
raise SystemExit("--trials must be at least 10")
root = find_root()
build = root / "VDBMS/bug-finder-agent/work/faiss/build-make"
static_lib = build / "faiss/libfaiss.a"
binary = HERE / "runtime/repro_helper"
binary.parent.mkdir(parents=True, exist_ok=True)
(HERE / "logs").mkdir(parents=True, exist_ok=True)
compile_cmd = ["g++", "-std=c++17", "-O2", "-I", str(root / "VDBMS/faiss"),
"-I", str(build), "-o", str(binary), str(HERE / "repro_helper.cpp"),
str(static_lib), "-fopenmp", "/lib/x86_64-linux-gnu/libblas.so.3",
"/lib/x86_64-linux-gnu/liblapack.so.3"]
if not static_lib.is_file():
print("missing audited FAISS static library", file=sys.stderr)
return 2
c = subprocess.run(compile_cmd, cwd=str(root), capture_output=True,
text=True, check=False)
write_json(HERE / "runtime/compile.json", {"command": compile_cmd,
"returncode": c.returncode, "stdout": c.stdout, "stderr": c.stderr})
if c.returncode != 0:
return 2
started = now()
control = run_process(binary, "control")
trials = [run_process(binary, "bug") for _ in range(args.trials)]
control_ok = check_control(control["returncode"], control["stdout"])
checks = [{**t, "oracle_match": check_bug(t["returncode"], t["stdout"]),
"fresh_process": True, "fresh_in_memory_graph": True} for t in trials]
bug_ok = all(t["oracle_match"] for t in checks)
finished = now()
raw = {"started_at": started, "finished_at": finished, "control": control,
"trials": checks}
normalized = {"control_pass": control_ok, "trial_count": len(checks),
"trial_pass_count": sum(t["oracle_match"] for t in checks),
"all_trials_match_bug_oracle": bug_ok,
"expected_control": expected_control(),
"expected_bug_contract": expected_bug_contract(),
"fresh_process_per_trial": True,
"fresh_in_memory_graph_per_trial": True}
schedule = {"kind": "deterministic_fresh_process_stability", "required_trials": 10,
"actual_trials": len(checks), "control_runs": 1,
"data_directories": "none; each process creates fresh in-memory indexes",
"started_at": started, "finished_at": finished}
write_json(HERE / "raw-responses.json", raw)
write_json(HERE / "normalized-results.json", normalized)
write_json(HERE / "trial-results.json", checks)
write_json(HERE / "schedule.json", schedule)
(HERE / "logs/stability.log").write_text(
"started_at=" + started + "\ncompile_returncode=" + str(c.returncode) + "\n" +
"control=" + json.dumps(control, sort_keys=True) + "\n" +
"\n".join("trial_%d=" % (i + 1) + json.dumps(x, sort_keys=True)
for i, x in enumerate(checks)) + "\nfinished_at=" + finished + "\n")
if not control_ok or not bug_ok:
print("oracle failed", file=sys.stderr)
return 3
print(json.dumps(normalized, sort_keys=True))
return 10
if __name__ == "__main__":
raise SystemExit(main())
The destination index initially contains one vector. The source index contains one vector. After calling:
destination.merge_from(source, 100);the query exactly matches the moved source vector.
Observed control result:
add_id=0 ntotal=2 labels=1,0 source=0Expected nonzero-offset result:
add_id=100 ntotal=2 labels=100,0 source=0Actual result:
add_id=100 ntotal=2 labels=1,0 source=0All ten fresh-process trials reproduce the incorrect label.
Root Cause
IndexFastScan::merge_from declares the parameter as unused:
void IndexFastScan::merge_from(Index& otherIndex, idx_t /*add_id*/)It copies each source code into the destination position ntotal + i, increments ntotal, and resets the source index. It never stores or applies the requested ID offset.
Because FastScan uses implicit sequential labels, the physical destination position 1 is returned as label 1. The required logical label should be:
source_id + add_id = 0 + 100 = 100The public Index::merge_from contract states that add_id is added to all moved IDs:
The IndexFastScan API also documents add_id as the ID offset for merged vectors:
Why This Is a Bug
This is not an approximate-search recall difference. The query matches the moved vector, and the failure is the returned identifier.
If nonzero offsets cannot be represented by this sequential storage format, the method should reject them explicitly, as IndexFlatCodes::merge_from does. Silently accepting and ignoring the argument violates the public API contract.
[IndexFlatCodes implementation](https://github.com/facebookresearch/faiss/blob/main/faiss/IndexFlatCodes.cpp#L99-L108)
Impact
Applications that merge shards using disjoint ID ranges may retrieve the wrong document or metadata record for every moved FastScan entry. The merge reports success, making the problem difficult to detect.
Scope
No metadata, payload, predicate, filter, wrapper, or where behavior is involved.
Deduplication
As of 2026-09-07, no exact public FAISS Issue or PR was found for IndexFastScan::merge_from ignoring a nonzero add_id.
Related but distinct reports include:
- [Issue #3651](https://github.com/facebookresearch/faiss/issues/3651): callers omit the required offset, causing overlapping IDs.
- [Issue #1074](https://github.com/facebookresearch/faiss/issues/1074): general discussion about IDs after
merge_from. - [Issue #577](https://github.com/facebookresearch/faiss/issues/577):
merge_frommissing fromIndexIDMap.
The current upstream IndexFastScan.cpp implementation still contains the unused add_id parameter and the same positional copy behavior.
Source: facebookresearch/faiss