LAPI decision stream returns HTTP 200 with a valid empty snapshot after a database query failure

Author: pschiffeCreated Sep 16, 2026Updated Sep 16, 2026

Labelskind/bugneeds/triageversion/1.8.1

What happened?

On CrowdSec v1.8.1, a failed database query inside GET /v1/decisions/stream?startup=true&dedup=false&scopes=ip,range can return HTTP 200 with a syntactically valid empty snapshot:

http

HTTP/1.1 200 OK
Content-Type: application/json

{"new": [], "deleted": []}

This is indistinguishable from a successfully queried, genuinely empty decision set. A bouncer that uses startup=true to replace its authoritative active set could consequently remove retained bans because of a server-side query failure.

The same database fault is correctly reported by the non-streaming endpoint, GET /v1/decisions?scope=ip,range&type=ban:

http

HTTP/1.1 500 Internal Server Error
Content-Type: application/json; charset=utf-8

{"message":"query decision failed: unable to query"}

I reproduced this with plain curl, independently of any SDK. Before fault injection, the startup stream returned the test ban normally. Selected response headers are shown above; both responses were read to completion.

This report concerns error signaling, not an expectation that CrowdSec should operate normally against a deliberately damaged database schema. The schema change is a deterministic way to exercise the decision-query error path.

What did you expect to happen?

A failed decision query must not be presented as a successfully completed authoritative snapshot.

If the failure occurs before the response is committed, return an appropriate non-2xx status.
If streaming has already started, make the failure detectable, for example by aborting the response or leaving it incomplete rather than closing a valid success-shaped JSON document.
Retaining the server-side cursor for a later retry does not protect a client that has already accepted this particular startup=true response as a complete snapshot.

I am not suggesting that clients reject legitimate empty snapshots; the problem is that the failed and successful-empty cases currently look identical.

How can we reproduce it (as minimally and precisely as possible)?

Run only against the disposable container created by this script. It intentionally makes a database column unavailable. Never run the fault-injection SQL against a production or otherwise valuable database.

Requirements: Linux, Bash, Docker or Podman, curl, and Python 3 with sqlite3.

The script uses a digest-pinned official v1.8.1 image, a new database, an isolated container network, and an ephemeral port bound only to loopback. It removes the container, network, and temporary files on exit. The hard-coded bouncer key is a public disposable test credential, not a production key. I executed this script successfully using Podman 5.8.4's Docker-compatible CLI.

For Podman explicitly, set CONTAINER_RUNTIME=podman before running it.

bash

#!/usr/bin/env bash
# Destructive fault injection, restricted to a NEW disposable container/database.
set -euo pipefail
runtime="${CONTAINER_RUNTIME:-docker}"
image='docker.io/crowdsecurity/crowdsec:v1.8.1@sha256:0f2523fa61ef507f15d953045cface490cc880670c62f2755ced17524107f71a'
workdir="$(mktemp -d -t crowdsec-stream-repro.XXXXXXXX)"
name="$(basename "$workdir")"
network="${name}-net"
key='xxx-real-lapi-compatibility-key' # Public disposable fixture key only.
cleanup() {
  "$runtime" rm -f "$name" >/dev/null 2>&1 || true
  "$runtime" network rm "$network" >/dev/null 2>&1 || true
  rm -rf -- "$workdir"
}
trap cleanup EXIT
mkdir "$workdir/data"
printf '[]\n' > "$workdir/feature.yaml"
"$runtime" network create "$network" >/dev/null
"$runtime" run --pull=missing --detach --rm --name "$name" --network "$network" \
  -p '127.0.0.1::8080' \
  -e "BOUNCER_KEY_XXX=$key" \
  -e CROWDSEC_FEATURE_CHUNKED_DECISIONS_STREAM=false \
  -e CROWDSEC_BYPASS_DB_VOLUME_CHECK=true \
  -e DISABLE_AGENT=true -e DISABLE_ONLINE_API=true \
  -v "$workdir/data:/var/lib/crowdsec/data:rw,Z" \
  -v "$workdir/feature.yaml:/etc/crowdsec/feature.yaml:ro,Z" \
  "$image" >/dev/null
base="http://$("$runtime" port "$name" 8080/tcp)"
ready=false
for ((i=0; i<90; i++)); do
  if curl --max-time 2 -fsS -H "X-Api-Key: $key" \
    "$base/v1/decisions?scope=ip,range&type=ban" >/dev/null 2>&1; then
    ready=true
    break
  fi
  sleep 1
done
if [[ "$ready" != true ]]; then
  "$runtime" logs "$name"
  exit 1
fi
"$runtime" exec "$name" cscli decisions add \
  --ip 198.51.100.10 --duration 20m --reason stream-error-repro
printf '\n--- Before fault: startup stream contains the decision ---\n'
curl --max-time 10 -sS -i -H "X-Api-Key: $key" \
  "$base/v1/decisions/stream?startup=true&dedup=false&scopes=ip,range"
printf '\n'
# Only this fresh container's database is modified. Do NOT target a live LAPI DB.
"$runtime" exec "$name" chmod 0666 /var/lib/crowdsec/data/crowdsec.db
python3 - "$workdir/data/crowdsec.db" <<'PY'
import sqlite3, sys
with sqlite3.connect(sys.argv[1], timeout=5) as db:
    db.execute('ALTER TABLE decisions RENAME COLUMN "until" TO "unavailable_until"')
    # Preserve the v1.8.1 pre-stream ID cursor query; fail the decision query itself.
    assert db.execute('SELECT MAX(id) FROM decisions').fetchone()[0] is not None
PY
printf '\n--- After fault: stream returns HTTP 200 and an empty snapshot ---\n'
curl --max-time 10 -sS -i -H "X-Api-Key: $key" \
  "$base/v1/decisions/stream?startup=true&dedup=false&scopes=ip,range"
printf '\n\n--- Same fault: non-streaming endpoint returns HTTP 500 ---\n'
curl --max-time 10 -sS -i -H "X-Api-Key: $key" \
  "$base/v1/decisions?scope=ip,range&type=ban"
printf '\n'

Observed sequence:

Before the fault: the startup stream returns HTTP 200 and includes the ban for 198.51.100.10.
After the fault: the startup stream returns HTTP 200 with {"new": [], "deleted": []}.
With the same fault still present: the non-streaming decisions endpoint returns HTTP 500 and an error message.

The fault preserves SELECT MAX(id) FROM decisions while making the decision query fail. This matters on v1.8.1: dropping the entire decisions table instead causes the pre-stream cursor lookup to fail and produces an HTTP error before reaching the problematic streaming path.

Anything else we need to know?

This is reproducible with the official v1.8.1 image; it is not dependent on a custom CrowdSec build or SDK decoding behavior.
The relevant source is pkg/apiserver/controllers/v1/decisions.go at v1.8.1. In StreamDecisionChunked, the new-decision query error path closes the JSON with empty deleted content before returning the error. The normal stream response has already started at that point.
PR #4413 made chunked decision streaming mandatory. Setting CROWDSEC_FEATURE_CHUNKED_DECISIONS_STREAM=false with an empty feature.yaml does not restore the former non-streaming behavior on this version.
A client-side workaround is to fetch complete snapshots using GET /v1/decisions / APIClient.Decisions.List, which returned HTTP 500 for this same injected failure. This increases database work and bandwidth relative to incremental streaming.
The observed downstream risk described above concerns clients that replace their active state from startup snapshots. I have not established that every existing bouncer behaves this way, or demonstrated a remotely triggerable exploit.

The diagnostics below were captured from an equivalent fresh v1.8.1 fixture with additional setup and diagnostic requests; request counts therefore differ from the shorter standalone reproduction above.

Crowdsec version

Image:

docker.io/crowdsecurity/crowdsec:v1.8.1@sha256:0f2523fa61ef507f15d953045cface490cc880670c62f2755ced17524107f71a

cscli version

bash

$ cscli version
version: v1.8.1-909b5157
Codename: alphaga
BuildDate: 2026-09-03_11:03:45
GoVersion: 1.26.8
Platform: docker
libre2: C++
User-Agent: crowdsec/v1.8.1-909b5157-docker
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_kubernetes, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlog, db_mysql, db_postgres, db_sqlite

OS version

Container runtime: Podman 5.8.4 via its Docker-compatible CLI. Linux x86_64 host; container userspace and shared host kernel are shown below.

Container OS and kernel

bash

$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.24.1
PRETTY_NAME="Alpine Linux v3.24"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

$ uname -a
Linux df8047d92420 7.2.5-200.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 11 15:11:05 UTC 2026 x86_64 Linux

Enabled collections and parsers

The agent was disabled (DISABLE_AGENT=true). The following hub items were installed/enabled in the image, but the reproduction uses only LAPI and manually added decisions.

cscli hub list -o raw

bash

$ cscli hub list -o raw
name,status,version,description,type
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/geoip-enrich,enabled,0.5,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/public-dns-allowlist,enabled,0.1,Allow events from public DNS servers,parsers
crowdsecurity/sshd-logs,enabled,3.1,Parse openSSH logs,parsers
crowdsecurity/sshd-success-logs,enabled,0.1,Parse successful ssh logins,parsers
crowdsecurity/syslog-logs,enabled,1.0,,parsers
crowdsecurity/whitelists,enabled,0.3,Whitelist events from private ipv4 addresses,parsers
crowdsecurity/cdn-whitelist,enabled,0.5,Whitelist CDN providers,postoverflows
crowdsecurity/google-special-crawlers-whitelist,enabled,0.1,"Whitelist events from Google special crawlers (e.g. Google-InspectionTool, GoogleOther)",postoverflows
crowdsecurity/rdns,enabled,0.4,Lookup the DNS associated to the source IP only for overflows,postoverflows
crowdsecurity/seo-bots-whitelist,enabled,0.5,Whitelist good search engine crawlers,postoverflows
crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-cve-2024-6387,enabled,0.2,Detect exploitation attempt of CVE-2024-6387,scenarios
crowdsecurity/ssh-generic-test,enabled,0.2,Crowdsec Generic Test Scenario: SSH brute force trigger,scenarios
crowdsecurity/ssh-refused-conn,enabled,0.1,Detect sshd refused connections,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios
crowdsecurity/ssh-time-based-bf,enabled,0.3,Detect time-based ssh bruteforce attempts that evade rate limiting (with false positive reduction),scenarios
crowdsecurity/bf_base,enabled,0.1,,contexts
crowdsecurity/linux,enabled,0.4,core linux support : syslog+geoip+ssh,collections
crowdsecurity/sshd,enabled,0.9,sshd support : parser and brute-force detection,collections
crowdsecurity/whitelist-good-actors,enabled,0.4,Good actors whitelists,collections

Acquisition config

The agent was disabled. The image contained this dummy acquisition configuration; no real logs were acquired for the reproduction.

--- /etc/crowdsec/acquis.yaml ---
{"source": "file", "filename": "/does/not/exist", "labels": {"type": "syslog"}}

No files under acquis.d were printed by the diagnostic file scan.

Config show

cscli config show

bash

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Notification Folder    : /etc/crowdsec/notifications
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : info
   - Log Media              : stdout
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              : 
API Client:
  - URL                     : http://0.0.0.0:8080/
  - Login                   : localhost
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 0.0.0.0:8080
  - Listen Socket           : 
  - Profile File            : /etc/crowdsec/profiles.yaml

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 168h0m0s
      - Flush size          : 5000

Prometheus metrics

cscli metrics after reproduction

bash

$ cscli metrics
+------------------------------+
| Local API Alerts             |
+----------------------+-------+
| Reason               | Count |
+----------------------+-------+
| xxx-real-lapi        | 1     |
+----------------------+-------+
+---------------------------------------------------+
| Local API Metrics                                 |
+-----------------------------------+--------+------+
| Route                             | Method | Hits |
+-----------------------------------+--------+------+
| /v1/alerts                        | POST   | 1    |
| /v1/allowlists/check/:ip_or_range | GET    | 1    |
| /v1/decisions                     | DELETE | 1    |
| /v1/decisions                     | GET    | 1    |
| /v1/decisions/stream              | GET    | 4    |
| /v1/watchers/login                | POST   | 2    |
+-----------------------------------+--------+------+
+---------------------------------------------------+
| Local API Bouncers Metrics                        |
+------------+----------------------+--------+------+
| Bouncer    | Route                | Method | Hits |
+------------+----------------------+--------+------+
| XXX        | /v1/decisions/stream | GET    | 4    |
| XXX        | /v1/decisions        | GET    | 1    |
+------------+----------------------+--------+------+
+---------------------------------------------------------------+
| Local API Machines Metrics                                    |
+-----------+-----------------------------------+--------+------+
| Machine   | Route                             | Method | Hits |
+-----------+-----------------------------------+--------+------+
| localhost | /v1/decisions                     | DELETE | 1    |
| localhost | /v1/allowlists/check/:ip_or_range | GET    | 1    |
| localhost | /v1/alerts                        | POST   | 1    |
+-----------+-----------------------------------+--------+------+

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

No custom parsers, scenarios, notification plugins, or CrowdSec code changes were used.

Reproduction-specific settings:

DISABLE_AGENT=true
DISABLE_ONLINE_API=true
CROWDSEC_BYPASS_DB_VOLUME_CHECK=true
CROWDSEC_FEATURE_CHUNKED_DECISIONS_STREAM=false
/etc/crowdsec/feature.yaml contains []
A disposable bouncer is registered via BOUNCER_KEY_XXX.
A temporary bind mount supplies a fresh SQLite data directory.
The only intentional fault is renaming the decisions.until column after creating a valid decision, while keeping the ID cursor lookup operational.

Source: crowdsecurity/crowdsec

View original on GitHub View discussion on GitHub