Extract, transform and index CLI telemetry data into BigQuery

Once CLI usage telemetry data is being written to GCS, it needs to be indexed into BigQuery for analysis, dashboarding, etc. using DBT.

Data summary

Each record in each .ndjson file is an aggregate, counting multiple requests made by the CLI to one of three API surfaces: Elasticsearch, Kibana, or Cloud control plane, grouped by a number of fields in order to reduce the size of the dataset and effectively anonymize its contents.

Each record contains a client_meta field, which is a compressed string format specified here. When indexing into BigQuery, the client_meta field must be exploded out into several human-readable fields, so that we can filter and group by any of the dimensions of data stored within it. @MattDevy has done some work to parse this data out into multiple fields in DBT already, for BQ tables that store Elasticsearch language clients' telemetry, but those only measure Elasticsearch usage.

Raw telemetry sources

• Cloud control plane requests: gs://elastic-telemetry-prod-data-processed/batch/cloud-api-client-meta/
• Kibana requests: gs://elastic-telemetry-prod-data-processed/batch/kibana-client-meta/
• Elasticsearch requests: gs://elastic-telemetry-prod-data-processed/batch/lang-client-meta/

Filtering rules

Documents should only be indexed into BQ if they have a client_meta field and the user-agent matches ^elastic-cli\/.*.

Deduplication rules

Documents should use the BigQuery equivalent of a stable document ID in Elasticsearch to deduplicate, so that pushing the same record twice does not produce duplicate data. Existing lang-client-meta work in DBT should inform how this works.