HNSW loses vector dimension state after a rejected mixed-dimension batch
How to reproduce this bug?
import requests
BASE = "http://localhost:8080"
requests.post(
f"{BASE}/v1/schema",
json={
"class": "DimensionRetry",
"vectorizer": "none",
"vectorIndexType": "hnsw",
"properties": [{"name": "text", "dataType": ["text"]}],
},
).raise_for_status()
def batch(vectors):
return requests.post(
f"{BASE}/v1/batch/objects",
json={"objects": [
{
"class": "DimensionRetry",
"properties": {"text": str(i)},
"vector": vector,
}
for i, vector in enumerate(vectors)
]},
)
print("mixed batch:", batch([[1, 0, 0, 0], [0, 1, 0]]).text)
print("valid retry:", batch([[1, 0, 0, 0], [0, 1, 0, 0]]).text)
r = requests.post(
f"{BASE}/v1/objects",
json={
"class": "DimensionRetry",
"properties": {"text": "wrong dimension"},
"vector": [1, 2, 3],
},
)
print("wrong dimension:", r.status_code, r.text)What is the expected behavior?
The successful four-dimensional retry should initialize the HNSW dimension to 4. Any later vector with a different dimension should be rejected.
What is the actual behavior?
After the rejected mixed-dimension batch, the valid four-dimensional retry succeeds, but the HNSW dimension remains unset or zero. Consequently, a three-dimensional vector may bypass dimension validation.
Supporting information
The issue is caused by the sync.Once dimension initializer being consumed when the first mixed-dimension batch returns an error. The callback returns before storing h.dims, so later calls cannot initialize the dimension.
Relevant files:
adapters/repos/db/vector/hnsw/insert.goadapters/repos/db/vector/hnsw/index.go
This is different from a normal wrong-dimension insertion error because the trigger is a rejected first batch followed by a valid retry.
Server Version
1.40.0-dev
Weaviate Setup
Single Node
Nodes count
No response
Code of Conduct
- I have read and agree to the Weaviate's Contributor Guide and Code of Conduct
Source: weaviate/weaviate