A non-selective `range_index` numeric filter is fully materialized on every search and is not bounded by `search_cutoff_ms`, so concurrent searches OOM-kill the node
Bug Description
When a search's filter_by contains a numeric range_index clause that is non-selective (a >/>=/< comparison that matches almost every document, e.g. expires_at:>NOW on data where nothing has expired yet), Typesense materializes the full matching id array on every request, even when the same && (AND) expression contains a far more selective sibling clause that could drive the iteration instead.
Three things compound:
- The non-selective clause is always fully materialized. Per-request memory is proportional to the cardinality of that clause (millions of ids), and is essentially independent of how many documents the overall query returns (a query returning 0–few hits costs the same).
search_cutoff_msdoes not bound the filter-build phase. The cutoff is only checked while iterating results, not while the filter id set is being materialized, so a heavy filter build runs to completion regardless of the configured cutoff.- The OOM resource guard covers writes only.
memory-used-max-percentagerejects writes when memory is high, but the search path is not gated, so concurrent heavy searches can drive the process past physical RAM.
Together, a burst of concurrent searches that each carry a per-request value in the non-selective clause (so the result cache never hits) accumulates in_flight × per_request_alloc and OOM-kills the node. A single such query is cheap enough to look harmless; the failure only appears under concurrency.
Reproduction Steps
Self-contained: starts a memory-capped Typesense in Docker, loads 2M synthetic docs (all curl for Typesense calls; python3 only to generate the JSONL and drive concurrency), then ramps concurrent searches until the container is OOM-killed. Deterministic — OOM-kills at 40 concurrent workers on a 1.5 GB cap.
bash repro.sh#!/usr/bin/env bash
# A non-selective range_index numeric filter is fully materialized on every search
# (ignoring a more selective AND-sibling), and search_cutoff_ms does not bound the
# filter-materialization phase. Under a burst of concurrent searches the per-request
# allocation accumulates and OOM-kills the node.
#
# Requires: docker, curl, python3 (only to generate synthetic JSONL + drive concurrency).
# Usage: bash repro.sh
set -u
NAME=ts-repro
KEY=localkey
PORT=8108
URL="http://localhost:${PORT}"
DOCS=${DOCS:-2000000}
MEM=${MEM:-1500m}
IMG=typesense/typesense:30.2
NOW=1780000000
PY=""
for c in python3 python; do if "$c" -c 'import sys' >/dev/null 2>&1; then PY=$c; break; fi; done
if [ -z "$PY" ]; then echo "need python3 (or python) on PATH"; exit 1; fi
echo "== 1. start memory-capped Typesense (${MEM}) =="
docker rm -f $NAME >/dev/null 2>&1
docker volume rm ${NAME}-data >/dev/null 2>&1
docker run -d --name $NAME --memory=$MEM --memory-swap=$MEM -p ${PORT}:8108 \
-v ${NAME}-data:/data \
$IMG --data-dir=/data --api-key=$KEY --thread-pool-size=384 --enable-search-analytics=false >/dev/null
until [ "$(curl -s ${URL}/health -H "x-typesense-api-key: ${KEY}")" = '{"ok":true}' ]; do sleep 1; done
echo "healthy"
echo "== 2. create collection =="
curl -s "${URL}/collections" -H "x-typesense-api-key: ${KEY}" -H 'content-type: application/json' -d '{
"name":"listings","token_separators":["-","/"],
"fields":[
{"name":"title","type":"string"},
{"name":"brand","type":"string","optional":true},
{"name":"subject","type":"string","optional":true},
{"name":"show_result","type":"bool"},
{"name":"ts","type":"int32","range_index":true,"sort":true},
{"name":"expires_at","type":"int64","range_index":true,"sort":true}
]}' >/dev/null
echo "== 3. load ${DOCS} synthetic docs (expires_at ALL in the future; ts ~10% selective) =="
"$PY" - "$DOCS" "$NOW" <<'PY' | curl -s "${URL}/collections/listings/documents/import?action=create" \
-H "x-typesense-api-key: ${KEY}" -H 'content-type: text/plain' --data-binary @- >/dev/null
import sys, json, random
docs, now = int(sys.argv[1]), int(sys.argv[2])
W="alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima mike november".split()
for i in range(docs):
r=random.Random(i)
print(json.dumps({"id":str(i),
"title":" ".join(r.choice(W) for _ in range(r.randint(3,8))),
"brand":r.choice(W),"subject":r.choice(W),"show_result":True,
"ts": now-100_000_000 + (i%100)*1_000_000, # only the top bucket passes ts:>NOW-10M
"expires_at": now+10_000_000 + (i%1000)})) # ALL strictly > now
PY
echo "loaded $(curl -s ${URL}/collections/listings -H "x-typesense-api-key: ${KEY}" | "$PY" -c 'import sys,json;print(json.load(sys.stdin)["num_documents"])')"
echo "== 4. filter cardinality: the >NOW range clause is NON-SELECTIVE (matches every doc) =="
probe(){ curl -s "${URL}/multi_search?use_cache=false" -H "x-typesense-api-key: ${KEY}" -H 'content-type: application/json' \
-d "{\"searches\":[{\"collection\":\"listings\",\"q\":\"*\",\"filter_by\":\"$1\",\"per_page\":0}]}" \
| "$PY" -c 'import sys,json;r=json.load(sys.stdin)["results"][0];print(" found",r["found"],"search_time_ms",r["search_time_ms"])'; }
echo "expires_at:>NOW ->"; probe "expires_at:>${NOW}"
echo "ts:>NOW-10M ->"; probe "ts:>$((NOW-10000000))"
echo "combined (AND) ->"; probe "(show_result:true&&expires_at:>${NOW}) && (ts:>$((NOW-10000000)))"
echo "== 5. concurrent stampede: each request uses a unique expires_at literal (cache miss) =="
echo " ramps workers until the container is OOM-killed (State.OOMKilled=true)"
"$PY" - "$URL" "$KEY" "$NOW" "$NAME" <<'PY'
import sys, json, re, threading, time, subprocess, urllib.request
URL,KEY,NOW,NAME=sys.argv[1],sys.argv[2],int(sys.argv[3]),sys.argv[4]
BODY={"searches":[{"collection":"listings","q":"alpha bravo charlie delta echo foxtrot golf hotel",
"query_by":"title,brand,subject","filter_by":f"(show_result:true&&expires_at:>{NOW}) && (ts:>{NOW-10000000})",
"sort_by":"ts:desc,_text_match:desc","per_page":50,"search_cutoff_ms":10000}]}
stop=threading.Event(); dead=threading.Event(); lock=threading.Lock(); recs=[]; seq=[0]
def fire():
while not stop.is_set() and not dead.is_set():
with lock: s=seq[0]; seq[0]+=1
b=json.loads(json.dumps(BODY))
b["searches"][0]["filter_by"]=re.sub(r"expires_at:>(\d+)",lambda m:f"expires_at:>{int(m.group(1))-s}",b["searches"][0]["filter_by"])
try:
rq=urllib.request.Request(URL+"/multi_search?use_cache=true",data=json.dumps(b).encode(),
method="POST",headers={"x-typesense-api-key":KEY,"content-type":"application/json"})
t0=time.time()
with urllib.request.urlopen(rq,timeout=60) as r: st=json.load(r)["results"][0].get("search_time_ms")
with lock: recs.append(("ok",round((time.time()-t0)*1000),st))
except Exception:
with lock: recs.append(("err",0,0))
def state(): return subprocess.run(["docker","inspect","-f","{{.State.OOMKilled}}|{{.State.Status}}",NAME],capture_output=True,text=True).stdout.strip()
for n in (40,80,160):
if dead.is_set(): break
stop.clear()
with lock: recs.clear()
ws=[threading.Thread(target=fire,daemon=True) for _ in range(n)]
[w.start() for w in ws]
end=time.time()+25
while time.time()<end:
time.sleep(2); st=state()
with lock:
done=len(recs); ok=sum(1 for r in recs if r[0]=="ok"); lat=sorted(r[1] for r in recs if r[0]=="ok") or [0]
print(f" workers={n:>3} done={done} ok={ok} p50={lat[len(lat)//2]}ms state={st}",flush=True)
if st.startswith("true"): print(f" !! CONTAINER OOM-KILLED (State.OOMKilled=true) at {n} workers"); dead.set(); break
stop.set(); time.sleep(0.5)
print("RESULT:", "REPRODUCED - OOM-killed" if dead.is_set() else "no OOM at this ramp")
PY
echo "== teardown: docker rm -f ${NAME} && docker volume rm ${NAME}-data =="== 1. start memory-capped Typesense (1500m) ==
healthy
== 2. create collection ==
== 3. load 2000000 synthetic docs (expires_at ALL in the future; ts ~10% selective) ==
loaded 2000000
== 4. filter cardinality: the >NOW range clause is NON-SELECTIVE (matches every doc) ==
expires_at:>NOW ->
found 2000000 search_time_ms 90
ts:>NOW-10M ->
found 180000 search_time_ms 8
combined (AND) ->
found 180000 search_time_ms 73
== 5. concurrent stampede: each request uses a unique expires_at literal (cache miss) ==
ramps workers until the container is OOM-killed (State.OOMKilled=true)
workers= 40 done=1083 ok=0 p50=0ms state=true|exited
!! CONTAINER OOM-KILLED (State.OOMKilled=true) at 40 workers
RESULT: REPRODUCED - OOM-killedNotes:
- The non-selective clause
expires_at:>NOWmatches 2,000,000 / 2,000,000 docs; the selective siblingts:>NOW-10Mmatches 180,000 (~9%). The combined AND result is 180,000, yet the node still pays the full 2M-id materialization. - Each request uses a unique
expires_atliteral, so the result cache never hits — exactly how a per-request "now" timestamp behaves in a real workload. - If the same identical query (fixed literal) is repeated instead, it is served from the result cache after the first compute and does not OOM — confirming the cost is the per-request filter materialization, not the result set.
Expected vs Actual
Expected behavior
A burst of concurrent searches with a bounded search_cutoff_ms should not be able to exhaust node memory and crash the process. An AND filter should be able to drive iteration from its most selective clause rather than fully materializing a clause that matches ~all documents.
Actual behavior
The non-selective range_index clause is fully materialized on every request (cost independent of result count); search_cutoff_ms does not bound the materialization; concurrent requests accumulate in_flight × per_request_alloc and the process is OOM-killed (State.OOMKilled=true, container exits, index reloads from disk on restart).
Environment
- Typesense version:
v30.2(Dockertypesense/typesense:30.2). The relevant code paths are unchanged on the current default branch (v31). - Operating system: reproduced on Docker; the OOM-kill is the container cgroup memory limit (1.5 GB in the script).
- Client library & version: none — raw HTTP via
curl/urllib.
Schema / Configuration
{
"name": "listings",
"token_separators": ["-", "/"],
"fields": [
{ "name": "title", "type": "string" },
{ "name": "brand", "type": "string", "optional": true },
{ "name": "subject", "type": "string", "optional": true },
{ "name": "show_result", "type": "bool" },
{ "name": "ts", "type": "int32", "range_index": true, "sort": true },
{ "name": "expires_at", "type": "int64", "range_index": true, "sort": true }
]
}Server flags: --thread-pool-size=384 --enable-search-analytics=false, memory-used-max-percentage at its default.
Additional Context
Pointers to the relevant code paths on v30.2 (also present on v31), for whoever picks this up:
- Non-selective
range_indexleaf is always fully materialized —src/filter_result_iterator.cppinit(), theif (f.range_index)branch (~L1148–1199) callstrie->search_greater_than(...)and setsis_filter_result_initialized = truewith the full id array. Unlike the non-range numeric branch, it does not consultenable_lazy_evaluation.NumericTrie::search_greater_than(src/numeric_range_trie.cpp) walks the trie andor_scalars the full array. search_cutoff_msnot checked during materialization —is_timed_out()is only consulted innext()/advance/and_filter_iterators, not inside the leaf materialization ininit().- Resource guard is write-only —
http_req::do_resource_check()(src/http_data.cpp) is consulted from the write paths (src/raft_server.cpp,src/batched_indexer.cpp); the search path does not checkcached_resource_stat. - The housekeeper logs expensive in-flight queries (
Detected bad query,src/housekeeper.cpp) but only logs — it does not cancel them, and it is a periodic poll, so short bursts that complete between polls are never recorded.
Possible directions (non-prescriptive): give the range_index leaf an iterator/lazy mode so an AND can drive from its selective clause; honor search_cutoff_ms during filter materialization; and/or apply a memory guard to the search path.
Source: typesense/typesense