Support Druid multi-value dimensions: restrict the group key, not just the row
Is your feature request related to a problem? Please describe.
Druid string columns can hold several values per row (multi-value dimensions). Cube renders a filter on such a column as a row-level WHERE, but Druid splits a multi-value column into one row per value after the filter has run. Every value the matching row carries then becomes its own group, so the result contains groups for values the filter excluded.
With a category column where a row looks like ["Shopping", "Travel"], this query:
{
"measures": ["events.spend"],
"dimensions": ["events.category"],
"filters": [
{ "member": "events.category", "operator": "equals", "values": ["Shopping", "Saving"] }
]
}generates:
SELECT category, SUM(spend)
FROM events
WHERE category IN ('Shopping', 'Saving')
GROUP BY 1and returns a Travel row, plus a row for every other category that co-occurs with Shopping or Saving anywhere in the scanned data. Cube has no notion of a dimension whose column is multi-valued, so there is nothing to set in the model to prevent it.
Describe the solution you'd like
A way to declare a dimension as multi-valued in the model:
dimensions:
- name: category
sql: category
type: string
multi_value: trueand have the Druid dialect restrict the group key with the function that matches the filter's operator, rather than leaving the group key bare:
| Cube operator | Group key |
|---|---|
equals |
MV_FILTER_ONLY(category, ARRAY['Shopping', 'Saving']) |
notEquals |
MV_FILTER_NONE(category, ARRAY['Travel']) |
contains |
MV_FILTER_REGEX(category, 'Shop') |
startsWith |
MV_FILTER_PREFIX(category, 'Shop') |
endsWith |
MV_FILTER_REGEX(category, 'ing$') |
So the query above would generate:
SELECT MV_FILTER_ONLY(category, ARRAY['Shopping', 'Saving']), SUM(spend)
FROM events
WHERE category IN ('Shopping', 'Saving')
GROUP BY 1The row filter stays, so Druid still prunes rows before reading them.
MV_FILTER_REGEX and MV_FILTER_PREFIX are recent — they are absent from the 26, 28 and 31 docs
and present in latest - so an implementation needs a fallback for older clusters. Restricting the
grouped value with a HAVING works there. notContains / notStartsWith / notEndsWith have no
direct function in any version and need either a negative-lookahead pattern or the same HAVING
fallback.
Two cases need explicit handling either way: ungrouped queries, where there is no group key and
rewriting the SELECT list would corrupt the row's own values; and single-value exact matches,
where Druid folds the group key to a constant on its own and no wrap is needed.
Describe alternatives you've considered
- A
dialectFactorythat subclasses the Druid query class and overridesdimensionSqlandbaseHaving. This is what we run today. It works, but it is bypassed by the Rust planner:buildSqlAndParamsreturnsbuildSqlAndParamsRust()whenuseNativeSqlPlanneris set, and that path never calls either method. We pinCUBEJS_TESSERACT_SQL_PLANNER=falseto keep it alive, which is not somewhere we want to stay. Our cluster predatesMV_FILTER_REGEX, so we coverequalswithMV_FILTER_ONLYandcontainswith aHAVING. - Expressing it in the model's
sql:. All four functions need the filter's values, which aren't known when the model is written, so this cannot be a static expression. - Exploding the column upstream into one row per value in the ETL. That changes the row count, so every other measure on the table has to be reworked.
- Dropping the surplus groups client-side. The measure values for the requested groups are
correct, so filtering the response would give the right numbers. It falls apart with
limit/offsetand server-side ordering: the extra groups consume page slots and shift the ranking, so the client cannot reconstruct the intended page.
Additional context
Happy to put up a PR if you can point at where it should live — the JS query builder, the Tesseract
planner, or both - and whether multi_value on the dimension is the shape you'd want, versus
something dialect-scoped.
Source: cube-js/cube