[Feature] Store BanyanDB's own logs in BanyanDB (self-stored logs)

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

1. Feature introduction

BanyanDB self-stores its metrics, but not its logs.

The entire log output story is one line, fixed at logger.Init() time:

// pkg/logger/setting.go
if development { w = zerolog.ConsoleWriter{Out: os.Stderr, …} } else { w = os.Stderr }
ctx := zerolog.New(w).Level(lvl).With().Timestamp()

Proposal: give logs the same two-destination model metrics already have — console and self-storage — reusing the native-metrics machinery wherever it already solves the problem.

Non-goals: replacing console output; ingesting non-BanyanDB logs; changing the module tree, level filtering, or existing flags.

2. Deployment architecture

flowchart TB
    C["clients (OAP, bydbctl)"] --> LB["gRPC load balancer :17912"]

    subgraph LT["Liaison tier — stateless, NO disk"]
        L0["liaison-0<br/>:17912 client · :18912 peer<br/>+ FODC agent"]
        L1["liaison-1<br/>+ FODC agent"]
    end

    subgraph DT["Data tier — OWNS THE DISK"]
        DH["data-hot-0 :17912<br/>+ FODC agent<br/>+ lifecycle sidecar<br/>+ backup sidecar<br/>+ restore init container<br/>PVC: measure/stream/trace/property"]
        DW["data-warm-0"]
        DC["data-cold-0"]
    end

    FP["fodc-proxy · 1 per cluster<br/>:17913 /metrics · /cluster/topology"]

    LB --> L0 & L1
    L0 & L1 -- "write / query :17912" --> DH
    DH -- "hot→warm→cold" --> DW --> DC
    L0 & L1 -. "gRPC register" .-> FP
    DH & DW & DC -. "gRPC register" .-> FP
    FP --> P["Prometheus → Grafana"]

Pairing rules

liaison and data are cluster-scoped peers. Everything else is a per-node companion glued to one node by 127.0.0.1 or a shared volume.

3. Role → function → log destination

Self-storage needs a process owning the storage engine — the component holding queryable groups on disk. Only data and standalone do. Everything else must reach one, and the hop count differs.

Two corrections to a common mental model:

• A liaison is not diskless. It has a write queue at --measure-data-path / --stream-data-path (banyand/measure/wqueue.go) where it buffers parts before syncing them out. It lacks the storage engine, so it can hold a log batch in flight but can never be where logs are read back from.
• lifecycle and backup never go through a liaison. pub.NewWithoutMetadata(nil) defaults to ROLE_DATA, so they publish straight to their co-located data node — a strictly shorter path than the liaison's.

restore and migration are not gaps to fill later. A tool that runs while the database is down cannot log into that database — and for migration, a self-storing sink would violate its own precondition that nothing else writes to the target paths.

4. Proposed approach

4.1 The seam — one event, two writers

zerolog always encodes an event to JSON internally, and ConsoleWriter is itself an io.Writer that re-formats that JSON. So MultiLevelWriter hands the sink the fully-encoded line — module, level, timestamp, message, all structured fields — one source, one format, one set of parameters.

flowchart LR
    A["call site<br/>logger.GetLogger(measure).Info()<br/>.Str(group, g).Msg(flushed part)"] --> B["zerolog encodes ONE JSON event"]
    B --> CW["console writer<br/>stderr · ALWAYS ON"]
    B --> SW["switchableWriter<br/>atomic.Pointer"]
    SW --> S2["logSink — ONE ring buffer<br/>alive from Init()"]
    SW --> S3["io.Discard — after GracefulStop"]

❌ zerolog.Hook rejected — hooks see level + message but not the accumulated structured fields.

The writer contract

Three constraints come from zerolog's own implementation (v1.34.0), not from BanyanDB:

type switchableWriter struct{ target atomic.Pointer[zerolog.LevelWriter] }

func (s *switchableWriter) WriteLevel(l zerolog.Level, p []byte) (int, error) {
    if w := s.target.Load(); w != nil {
        (*w).WriteLevel(l, p)   // errors deliberately ignored
    }
    return len(p), nil          // always
}

atomic.Pointer rather than a mutex: WriteLevel runs on every log line from every goroutine, while the pointer is swapped once per process (at GracefulStop). A lock-free load is the right trade at that ratio.

4.2 Lifecycle — deferred activation

Same deferral pendingMeasures / native.InitSchema already use for metric schemas — but applied to the consumer, not the buffer.

Init()        install switchableWriter → logSink   (buffer live, NO consumer, no I/O)
PreRun()      register drop counters                            (no metadata)
Serve()       1. create group + stream schema (idempotent)
              2. START the consumer goroutine (§4.3)
GracefulStop  closer.CloseNotify() → consumer drains once and exits (bounded)
              swap → io.Discard · close publisher

One buffer, two phases. The buffer is allocated at Init() and never replaced; "activation" only starts draining it. There is no second buffer and no hand-off, so a goroutine that loaded the writer just before activation cannot strand its line in an abandoned buffer. Producers see one unchanging target for the whole process lifetime.

4.3 Write workflow

The consumer is a dedicated goroutine, following accesslog.startConsumer — not a timestamp.Scheduler job as FlushMetrics uses. A scheduler job is a periodic callback: it fires on the tick and can do nothing between ticks, so it supports a time trigger and nothing else. The size trigger needs something that observes every push, which only a goroutine selecting on the buffer can do.

for {
    select {
    case <-s.closer.CloseNotify():
        s.flush(batch)                      // final drain, bounded
        return
    case <-flushTicker.C:                   // TIME trigger  (--logging-native-flush-interval, 5s)
        if len(batch) > 0 { s.flush(batch); batch = batch[:0] }
    case entry := <-s.buffered():
        batch = append(batch, entry)
        if len(batch) >= s.flushSize {      // SIZE trigger  (--logging-native-flush-size, 1024)
            s.flush(batch); batch = batch[:0]
        }
    }
}

flowchart LR
    R["buffer<br/>bounded · drop on full"] --> T{"consumer goroutine<br/>flush trigger"}
    T -- "interval 5s" --> B["build InternalWriteRequest"]
    T -- "size ≥ 1024" --> B
    T -- "CloseNotify" --> B
    B --> N{"nodeSelector"}
    N -- "nil (data/standalone)" --> LOC["queue.Local() → own shard"]
    N -- "set (liaison/lifecycle)" --> LO["Locate()"]
    LO -- ok --> PUB["pub → TopicStreamWrite → data node"]
    LO -- "fail" --> DROP["count no_node, drop<br/>(do NOT publish empty nodeID)"]

4.4 Schema

Derived field by field from the native-metrics schema in pkg/meter/native/provider.go. Each row names the proto field, so the diff against the existing implementation is explicit.

Running example — this line, emitted on node data-hot-0:

{"level":"warn","module":"MEASURE","group":"sw_metric","time":"2026-09-11T10:23:45.123Z","message":"flush took longer than expected"}

Group — common.v1.Group

Resource — metrics use database.v1.Measure, logs use database.v1.Stream

Not a rename. Measure and Stream are two distinct proto messages that coexist; nothing is renamed or modified. Logs simply instantiate a different existing type. "Resource" is BanyanDB's own umbrella term for what a group holds — docs/concept/data-model.md: "A group's catalog fixes which one kind of resource it holds (MEASURE, STREAM, TRACE, or PROPERTY)." There is no Resource type in code.

Write payload

tag_families:
  searchable: node_type, node_id, module, level, grpc_address, http_address, message
  data:       body  TAG_TYPE_DATA_BINARY   ← the complete original JSON line

• grpc_address / http_address — searchable, not in the entity. Not cardinality (both follow from node_id) but address churn: a restarted pod keeps its node_id and gets a new IP, which would open a new series and fragment that node's history.
• body holds the whole original line — no field lost to schema drift, byte-identical to the console.
• element_id = <node_id>-<process_epoch>-<seq>. timestamp = the event's time, not flush time, so lines buffered before activation land correctly on the time axis.

4.5 Two things that differ from metrics

Recursion guard, three layers: module denylist for write-path modules (primary) → sink never logs through pkg/logger, only rate-limited stderr → bounded ring caps amplification (backstop).

⚠️ The denylist is load-bearing for the liaison specifically: its two-tier path traverses the write queue and part-sync, both of which log.

Invariant across every failure mode: console output is never degraded by the sink. Self-storage is best-effort; the console is not.

4.6 Implementation phases

See this comment — delivery order, and why each phase adds exactly one new failure domain.

5. Parameters and configuration

Derived from the two existing surfaces — prefix from console logging, vocabulary from metrics:

pkg/logger/setting.go RegisterFlags       observability/services/service.go FlagSet
  --logging-env      prod                   --observability-listener-addr         :2121
  --logging-level    info                   --observability-modes          [prometheus]
  --logging-modules  nil                    --observability-metrics-interval        15s
  --logging-levels   nil                    --observability-native-flush-interval    5s
        └──── "logging-" prefix ──┐   ┌──── "modes" + "-native-" infix ────┘
                                  ▼   ▼
                           --logging-native-*

Deliberately not copied from metrics: --observability-listener-addr (logs are push-only — no pull endpoint to scrape) and --observability-metrics-interval (that is the Prometheus collection tick; logs have no collection phase).

Unchanged: the four existing --logging-env / -level / -modules / -levels flags. They apply upstream of both writers; altering them is an explicit non-goal. Each new flag gets the standard BYDB_* env binding, as logger.RegisterFlags already provides. restore / migration accept --logging-modes but reject native at Validate().

6. Reading logs back

No new query surface — _monitoring_log is an ordinary stream group:

bydbctl stream query -f - <<EOF
name: "log"
groups: ["_monitoring_log"]
timeRange: { begin: 2026-09-10T00:00:00Z, end: 2026-09-11T00:00:00Z }
criteria:
  condition: { name: "level", op: "BINARY_OP_EQ", value: { str: { value: "error" } } }
projection:
  tagFamilies:
    - { name: "searchable", tags: ["node_id", "module", "level", "message"] }
    - { name: "data", tags: ["body"] }
EOF

7. Failure modes

Each row is one stage of the write path, in path order. Derived by walking the pipeline and asking, at each stage, "what can fail here, and what happens when it does?"

producer ──► BUFFER ──► consumer ──► schema ──► Locate ──► publish ──► data node
   │          │          │           │          │           │           │
   │          │          │           │          │           │           └─ slow
   │          │          │           │          │           └─ error / timeout
   │          │          │           │          └─ no node available
   │          │          │           └─ CreateGroup / CreateStream fails
   │          │          └─ not started yet (pre-Serve)
   │          └─ full
   └─ never blocks  ← the invariant, not a failure

Every stage that can fail gets exactly one row, one behaviour, one counter label. The review test is therefore not "are these cases interesting" but "does every arrow in that diagram have a row" — a stage with no row is a stage whose failure is unhandled.