Visual query builder

A point-and-click UI for constructing Arkime search expressions without hand-writing expression syntax — field / operator / value rows, AND/OR/NOT grouping, parentheses — with lossless round-trip to/from the existing text expression bar.

Part of the Arkime 7 UI epic (#3926). Large. Scoping below is settled; remaining open questions are small and listed at the end.

Decisions (settled)

Architecture — the AST becomes the center

Today: expression text → arkimeparser.js (jison) → ES query, server-side only. There is no representation that is editable or that can be turned back into text.

Target:

                 ┌──────────────────────────────┐
   grammar  ─────▶            AST                 │   (one source of truth)
 (one .jison)    └──────────────────────────────┘
                    │            │            │
        AST→ES  ────┘   AST→UI ──┘   AST→text ┘
       (backend)        (builder rows)   (serializer)

Three consumers of the AST:

1. AST → ES query (backend refactor). Replace the inline jison semantic actions that build ES directly. The leaf builders (formatQuery, formatExists, parseIpPort, stringQuery, termOrTermsInt, …) already exist as standalone functions and are reused verbatim. Only the four combinators move into an AST walker: &&→bool.filter, ||→bool.should, !→bool.must_not, ()→passthrough. parser.yy context (fieldsMap, views, shortcuts, prefix, requiredRight, emailSearch) is threaded into the walker unchanged.
2. AST → builder UI. Render the tree as nested group/rule rows in the popover.
3. AST → text serializer (new). Pretty-print the tree back to expression text with normalized formatting; re-insert parentheses wherever child precedence requires it to preserve meaning.

Proposed AST node model

Logical   { type: 'and' | 'or', children: Node[] }
Not       { type: 'not', child: Node }
Term      { type: 'term', field: string, op: 'eq'|'ne'|'lt'|'lte'|'gt'|'gte', value: Value }

Value (tagged so the row UI can pick the right editor):
  { kind: 'scalar',   text }                 // bare / quoted string, wildcard *
  { kind: 'regex',    pattern }              // /.../
  { kind: 'list',     mode: 'or'|'and', items: Value[] }   // [a,b] vs ]a,b[
  { kind: 'exists' }                         // EXISTS!
  { kind: 'shortcut', name }                 // $name
  { kind: 'ip',       text }                 // CIDR / :port shorthand (ip builder)
  { kind: 'range',    text }                 // integer/float range 1-5 (range UI)
  { kind: 'view',     name }                 // view embed (viewand)

(Group nesting is explicit in the tree via and/or children; the serializer adds parens from precedence. The db: field prefix is carried on Term.field.)

Grammar coverage checklist (full grammar)

Every construct below must parse → edit → serialize and survive text→AST→ES byte-for-byte against the golden tests:

• Comparison ops: == != < <= > >= (and lexer aliases =, normalized to ==)
• Boolean: && || ! (and aliases & |), correct precedence/associativity
• Parentheses / nested groups
• Quoted strings, escaped quotes
• Regex values /.../ (with the back-slash-forward-slash rule)
• Wildcards * inside string values
• OR lists [a,b,c] and AND lists ]a,b,c[
• EXISTS!
• Shortcuts $name (incl. inside lists, with */? glob expansion)
• IP field shorthand: CIDR, /8 /16 /24 short forms, :port (v4) / .port (v6), :: ipv6, arrays of these
• Integer/float ranges 1-5
• View embedding (viewand — view == name)
• db: field prefix
• Negative-number production (- e)

Backend refactor — blast radius & safety net

Touches the proven query path — handle with care.

Consumers of arkimeparser.parse() that must keep producing identical ES:

• viewer/buildQuery.js — request expression, view expansion (buildQuery.js:129), user forced-expression + emailSearch.
• viewer/apiCrons.js — periodic/cron query expression + user expression.

Safety net (must stay green, unchanged):

• tests/api-buildquery.t (~292 lines of exact esquery snapshot assertions)
• tests/api-views.t, tests/api-cron.t, tests/api-spiview.t

Plan: land the text→AST→ES refactor first, prove byte-identical output via the snapshot suite, then build the frontend on top. The grammar is jison-generated (arkimeparser.js from arkimeparser.jison, jison 0.4.18) — regeneration step must be documented/wired so the checked-in artifact can't drift from the grammar source.

Frontend components (new)

• QueryBuilderPopover.vue — anchored dropdown off the search bar (new ⊞ button next to the existing expand/save/clear buttons in ExpressionTypeahead.vue). Owns open state; parses bar text → AST on open; serializes AST → bar + emits applyExpression on Apply.
• QueryBuilderGroup.vue — recursive: renders a group's and/or toggle, its child rows/sub-groups, + rule / + group, and group delete/negate. Self-references for nesting.
• QueryBuilderRow.vue — one term: field typeahead → type-aware operator select → value editor.
• ValueEditor.vue — switches editor by Value.kind: scalar typeahead (live api/unique), list chips, regex toggle, EXISTS! toggle, ip builder, range inputs, $shortcut picker, view picker.
• Autocomplete extraction — pull the field/op/value resolution + value-fetch logic out of ExpressionTypeahead.vue into a reusable composable/service so both the bar and every builder row share it (no duplicated typeahead logic). FieldService already centralizes field metadata + value fetch.

Reuse: BigExpressionModal placement pattern, TypeaheadResults dropdown, teleport-positioning approach, arkime-input-group styling.

UX

• Launch: new icon button in the search-bar input group (alongside expand / save / clear).
• Popover shows the parsed tree as nested rule rows; page and bar remain visible behind it.
• Footer: Apply (write text → bar + search), Cancel (discard), Clear.
• If the bar text fails to parse on open: don't silently drop it — show a non-blocking notice ("expression too complex / invalid to edit visually") and offer to start fresh; leave the text bar untouched. (Decide exact treatment — see open questions.)

Phasing

1. Backend AST refactor — one grammar → AST, AST→ES walker, golden snapshot suite green & byte-identical. (No UI yet.)
2. AST→text serializer + a text→AST→text round-trip test harness over the api-buildquery.t corpus (normalization-aware).
3. Builder shell — popover, recursive group/row rendering, AST→UI binding, Apply path, parse-on-open.
4. Value editors — wire each Value.kind editor + full autocomplete reuse.
5. Polish — invalid-expression handling, keyboard nav, a11y, i18n strings, UI review.

Out of scope for v1

• Builder in views editor / cron / hunt / SPIView / SPIGraph (shared-component rollout = follow-up).
• Live two-way sync while the popover is open (Apply-only for v1).
• Byte-exact preservation of untouched formatting (we normalize).
• Drag-to-reorder / live result-count preview pane (dedicated-editor ideas, not this surface).

Remaining open questions (small)

• Exact treatment when the bar text won't parse on open: hard-disable the builder button vs. open-with-notice vs. best-effort partial parse.
• Serializer paren policy: minimal parens (precedence-driven only) vs. always-parenthesize groups for readability.
• Where the shared autocomplete logic should live (common/vueapp composable vs. viewer search/) given the future cross-app rollout.
• Whether db:-prefixed fields are surfaced in the field typeahead or only preserved on round-trip.