perf(ingest): DBM canonicalization runs on every log record of every stream; gate it on the destination stream

Problem

With ZO_DB_MONITORING_ENABLED on (the default), every log record ingested into any stream goes through DBM server-vantage canonicalization (db_monitoring::server_vantage::canonicalize_dbm_record). It is called per record from the logs ingest paths: the _json / _bulk path in src/core/src/logs/ingest.rs, and three record-assembly sites in src/core/src/logs/otlp.rs.

For an ordinary log record it finds nothing to do, but only after:

• a pre-scan of every key for the o2_dbm_ prefix (has_reserved_dbm_key);
• the full canonicalize_record dispatch, which runs unconditionally. It probes o2_pg_event (three times), o2_my_event, o2_maria_event and o2_recipe, and calls resolve_event_name twice, each call probing o2_event_name plus postgresql_calls / postgresql_state / postgresql_query_id;
• take_unrecognized_recipe and the OSS recipe check, which probe o2_recipe again.

That is about 16 hash lookups on the record map per record (more in enterprise builds), all of them misses for ordinary logs.

Measurement

POST /v1/logs (OTLP protobuf), 30M records, none of them DBM records, sample profile of the ingest handler on main (19a405c96b):

Function	Samples	Share of handler
logs::otlp::handle_request	23,926	100%
canonicalize_dbm_record	1,210	5.1%
↳ canonicalize_record	1,070	4.5%
↳ resolve_event_name	692	2.9%
flatten_with_level (for comparison)	945	3.9%

DBM canonicalization costs more than flattening the record. The JSON direct-to-Arrow path (#14558) avoids most of it with a per-column key check, but every record that falls back to the JSON path, and every stream with a pipeline or user-defined schema, still pays per record.

Why a per-record check is the wrong granularity

DBM server-vantage data has exactly one destination:

• the shipped collector recipes export to _o2_dbm_server (stream-name: ${DBM_SERVER_STREAM} in web/src/components/ingestion/setupCard/content/dbmShared.ts);
• the DBM read APIs only ever read that stream (DEFAULT_SERVER_STREAM in src/api/management/src/request/db_monitoring/service/common.rs); there is no setting to point them anywhere else;
• the _json, _bulk and HEC paths reject user writes to _o2_* streams (is_internal_rollup_stream), so on those paths an ordinary stream can never feed the DBM pages.

Canonicalizing records in any other stream only adds o2_dbm_* columns that nothing reads, and stripping client-supplied o2_dbm_* keys there protects no reader.

Checking record keys can be made cheaper (an exact-key pre-check was prototyped: handler share 5.1% → 1.3%, CPU per record −2.3%), but it still scans every key of every record, and the key list has to be kept in sync with the dispatch arms, including enterprise-only hooks that the OSS tests cannot see.

Proposal: match on the destination stream name, not on the record

Decide once per request, per destination stream:

1. Share the stream name. Add pub const DBM_SERVER_STREAM: &str = "_o2_dbm_server" to config::meta::db_monitoring, and use it both in the read API (replacing DEFAULT_SERVER_STREAM) and on the ingest side.
2. Gate canonicalization on the stream. In logs/ingest.rs and logs/otlp.rs, compute dbm_stream = cfg.db_monitoring.enabled && stream_name == DBM_SERVER_STREAM once per request, and call canonicalize_dbm_record only when it is true. Records routed by a pipeline are checked against their destination stream, not the source.
3. Gate the stream-level DBM work the same way. batch_has_dbm_records + ensure_server_stream_index_field in write_logs then only run for the DBM stream.
4. JSON direct-to-Arrow path. The per-column dbm_key fallback in logs/columnar.rs only applies when the destination is the DBM stream; other streams never fall back because of an o2_* column.

For every other stream this costs one string comparison per request instead of a key scan per record, and nothing has to mirror the dispatch key list anymore.

Points to confirm

• Pipelines: a recipe or user pipeline may route records into _o2_dbm_server. Canonicalization already runs on pipeline outputs in otlp.rs; that call site should check the destination stream.
• Existing data outside _o2_dbm_server: records canonicalized into other streams are already invisible to the DBM pages, so dropping canonicalization there changes no DBM view. It does stop adding o2_dbm_* columns to those streams.
• Spoofing: client-supplied o2_dbm_* keys are still stripped on _o2_dbm_server, the only stream the DBM reads use.
• Source-shape tests: every_logs_ingest_path_applies_canonicalization (db_monitoring/tests_server_vantage.rs) counts the call sites. It should keep doing so and also require the stream gate, so a new ingest path cannot skip either.

Out of scope: traces

Client-side DBM enrichment (db_monitoring::enrich_with_opts) annotates application spans in any traces stream by design, so a stream-name gate does not apply there. It already short-circuits cheaply for ordinary spans: span kind must be CLIENT/PRODUCER, then any db.* / db_* key must be present.

For DB-heavy trace loads, the cost is in the enrichment itself and belongs in a separate issue. On a load where 90% of spans carry DB attributes:

• normalize_cached is 9.8% of the traces handler, 95% of it waiting on the LRU shard mutex;
• attribute resolve lookups are 3.6%;
• service-graph attribute lookups (span_graph_attr, with a replace and two format! per probe) are 10.9%.

The JSON traces path also removes 20 derived fields (ALL_DB_FIELDS + ALL_INFER_FIELDS) from every record unconditionally. A cheap exact-key pre-check can skip that for records that carry none.