Add source-aware detection readiness and integrity status

Motivation

Falco currently exposes a health endpoint whose handler returns {"status": "ok"} when the embedded webserver is running:

• [userspace/falco/webserver.cpp](https://github.com/falcosecurity/falco/blob/master/userspace/falco/webserver.cpp)

The Falco Helm chart currently uses the configured health endpoint for startup, liveness, and readiness probes:

• [chart/falco/templates/pod-template.tpl](https://github.com/falcosecurity/falco/blob/master/chart/falco/templates/pod-template.tpl)

This makes process liveness indistinguishable from detection availability. A Falco instance can therefore be reported as ready while an enabled event source is:

• still initializing or blocked during startup;
• no longer making progress;
• failing or stopped;
• losing events or state updates;
• unable to provide complete workload metadata;
• experiencing output queue loss.

For a runtime security sensor, “the process is alive” and “the detection pipeline is operational” are different guarantees.

This distinction is operationally important. For example, [#3692](https://github.com/falcosecurity/falco/issues/3692) describes container information disappearing while Falco continues running, resulting in false positives, alert fatigue, and eventually removal of Falco. The [0.45.0 release tracker](https://github.com/falcosecurity/falco/issues/3967) also identifies source initialization, multi-source behavior, event drops, and metrics stability as areas requiring validation.

Falco should expose a machine-readable indication of whether its enabled detection sources are initialized, responsive, and capable of processing events.

Feature

Introduce a source-aware detection integrity status in the Falco application.

The proposed application-level status model would track every enabled source independently using a small, bounded set of states, for example:

• starting
• ready
• degraded
• stalled
• failed
• stopped

Each state could include stable reason codes such as source_initializing, event_loop_stalled, event_drops, state_table_drops, metadata_unavailable, output_queue_drops, or source_error.

The initial feature would expose this information through three complementary contracts:

1. Preserve the existing /healthz behavior as a process and webserver liveness check. This avoids breaking existing deployments.

2. Add /readyz as a detection-readiness endpoint. It would return success only when every required enabled source has completed initialization and its processing loop remains responsive. A successful poll or timeout should count as progress, so an idle environment does not incorrectly become unready merely because it has no events.

3. Add /status as a versioned JSON snapshot containing aggregate and per-source status. The endpoint could also report bounded state-engine, enrichment, event-loss, and output-health information.

An illustrative response, not intended as a fixed schema, could be:

{
  "schema_version": 1,
  "status": "degraded",
  "ready": true,
  "sources": [
    {
      "name": "syscall",
      "status": "degraded",
      "reasons": ["event_drops"]
    }
  ],
  "outputs": {
    "status": "ready",
    "queue_drops": 0
  }
}

Equivalent bounded signals should be available through the existing Prometheus metrics integration when metrics are enabled.

degraded should not make Falco unready by default. Event loss or incomplete enrichment often requires investigation or tuning, while restarting Falco may increase the detection blind spot. Readiness should instead fail when a required source is unavailable, has failed initialization, or its processing loop is no longer responsive.

Most of the required lifecycle information is already observed by Falco:

• [userspace/falco/app/actions/process_events.cpp](https://github.com/falcosecurity/falco/blob/master/userspace/falco/app/actions/process_events.cpp) handles source initialization, poll timeouts, event processing, and fatal read errors.
• [userspace/falco/event_drops.cpp](https://github.com/falcosecurity/falco/blob/master/userspace/falco/event_drops.cpp) calculates event-loss ratios and actions.
• Existing metrics already expose output queue and state-related counters.

These signals could update a small thread-safe status registry. HTTP requests would read a cached snapshot and must not inspect live libsinsp instances or collect new snapshots, avoiding synchronization hazards and keeping the endpoint inexpensive.

Because the webserver may listen on 0.0.0.0, public responses should expose only bounded reason codes and non-sensitive counters. Raw errors, paths, endpoints, credentials, and other environment details should remain in logs or local support output.

Expected behavior would include:

• a source blocked during initialization never reports ready;
• an idle but responsive source remains ready;
• a stalled or failed required source becomes unready within a documented interval;
• event, state, enrichment, and output loss can be represented as degradation without causing restart loops;
• reload and shutdown produce deterministic status transitions;
• status requests remain race-free during reload and shutdown;
• single-source and multi-source configurations are covered by automated tests;
• status bookkeeping introduces no material event-processing regression.

The Helm chart could initially offer /readyz as an opt-in readiness probe while retaining /healthz for liveness. A default chart change should be considered only after the application-level contract has matured.

Alternatives

1. Change the semantics of /healthz.

   This would be simpler but could break existing deployments and turn recoverable degradation into Kubernetes restart loops. Keeping liveness and readiness separate provides a safer compatibility path.

2. Infer health exclusively from metrics, logs, or external dashboards.

   Operators can already combine several signals, but this requires deployment-specific logic and does not provide a canonical Falco readiness contract. It may also depend on the same output pipeline that is experiencing degradation.

3. Use the timestamp of the last processed event.

   This would incorrectly classify quiet environments as unhealthy. Source responsiveness should be based on processing-loop progress, including successful polling and timeout handling, rather than event volume.

4. Terminate Falco whenever a source becomes unhealthy.

   This would allow Kubernetes to restart the process, but it would not distinguish transient degradation from terminal failure and could create repeated blind spots.

5. Generate a periodic synthetic event as a positive-control canary.

   An end-to-end canary could eventually provide stronger verification, but it introduces additional security, portability, correlation, and performance considerations. It should be evaluated separately and should not block a passive application-level status model.

Additional context

This proposal is intended to establish the operational contract before implementation details are fixed. If maintainers agree with the problem and general direction, I would be happy to follow up with a design proposal covering:

• state definitions and transition rules;
• readiness aggregation across multiple sources;
• time bounds for detecting stalled processing;
• API and metric stability;
• reload and shutdown behavior;
• security constraints for remotely accessible status responses;
• feature maturity and Helm chart rollout.

The first iteration would not:

• automatically restart Falco when it becomes degraded;
• claim that every user rule detects every possible attack;
• verify delivery after an alert leaves Falco;
• require immediate changes to libscap, libsinsp, driver, or plugin APIs;
• alter the existing /healthz contract.

The design should remain compatible with the proposed [multi-threaded Falco architecture](https://github.com/falcosecurity/falco/blob/master/proposals/20251205-multi-thread-falco-design.md), where independent and cached source-level status becomes even more valuable.

The central goal is to make the following distinction observable and machine-verifiable:

A running Falco process does not necessarily mean that its detection pipeline is available.