[Feature] Native GEOGRAPHY and GEOMETRY design and implementation plan

• https://github.com/StarRocks/starrocks/pull/79043
• https://github.com/StarRocks/starrocks/pull/79035
• https://github.com/StarRocks/starrocks/pull/79005
• https://github.com/StarRocks/starrocks/pull/78999
• https://github.com/StarRocks/starrocks/pull/78986
• https://github.com/StarRocks/starrocks/pull/78916
• https://github.com/StarRocks/starrocks/pull/78700
• https://github.com/StarRocks/starrocks/pull/78701
• https://github.com/StarRocks/starrocks/pull/78713

Feature request

Problem

StarRocks currently has legacy spatial functions operating primarily on VARCHAR values, but it does not have a complete native geo type system with an explicit contract for semantics, transport, vectorized execution, persistence, and compatibility.

A raw WKB payload alone cannot determine whether a value represents GEOGRAPHY with spherical semantics or GEOMETRY with planar semantics. If this distinction is not preserved explicitly across FE/BE/CN boundaries, exchange, spill, serde, and future table metadata, the same bytes may be silently reinterpreted and require another type-format redesign later.

This issue is the umbrella design and implementation tracker for native GEOGRAPHY and GEOMETRY support. It is intended to link the focused PRs that implement each contract incrementally.

Proposed design

Logical and physical model

• GEOGRAPHY and GEOMETRY are distinct FE/SQL logical types.
• GEOGRAPHY uses spherical semantics; GEOMETRY uses planar semantics.
• There is no implicit conversion between the two types.
• Both types share one vectorized physical representation: GeoColumn, backed by BinaryColumn and storing ISO/OGC WKB per row. Scans preserve the raw source WKB without eager normalization.
• WKB bytes never determine the logical semantics by themselves. The semantics are carried by GeoColumnDesc throughout the complete type and data path.

Descriptor contract

GeoColumnDesc separates semantics from representation:

• GeoTypeDesc: semantic parameters such as coordinate system, edge algorithm, CRS identifier, and optional parsed SRID. The primitive SQL type is authoritative; any duplicated logical-kind field must agree with it and must never override it.
• GeoStorageDesc: encoding (WKB initially), declared dimension (UNKNOWN, XY, XYZ, XYM, XYZM, or MIXED), and producer-validation state.

Semantic GeoTypeDesc metadata is carried directly by FE types and scalar Thrift/protobuf and BE TypeDescriptor representations. Physical GeoColumnDesc combines that semantic metadata with GeoStorageDesc; encoding, dimension, and validation state are not scalar type identity. The complete primitive/semantic/storage information must be preserved through the applicable vectorized serde, exchange, spill, intermediate materialization, function-result and future tablet/storage paths.

All new Thrift and protobuf fields must remain optional and use fresh, stable field numbers and enum ordinals. Unknown numeric enum values must either survive metadata transport unchanged or be explicitly rejected before conversion; silently dropping them or replacing them with UNKNOWN or a supported default is forbidden. In particular, proto2 closed-enum values can reside in the unknown-field set, so accessor-only object round trips are not sufficient evidence of compatibility.

Descriptor compatibility and integration rules:

• Represent SPHERICAL, VINCENTY, THOMAS, ANDOYER, KARNEY, and PLANAR distinctly. Do not collapse external algorithms into a generic GEODESIC value. Metadata recognition/transport does not enable computation: initial native Geography compute supports only the reviewed spherical contract.
• Keep semantic identity separate from storage compatibility. The descriptor-only slice provides semantic comparison and exact metadata equality for transport tests, not SQL assignment or UNION ALL analysis. FE owns assignment compatibility, UNION ALL type resolution, and the resulting dimension. Defer operation-specific BE assignment and UNION ALL helpers until concrete callers are introduced, together with FE/BE consistency tests and end-to-end tests of the FE-selected result descriptor. Producer-validation state and strict/try constructor policy do not affect SQL type identity or compatibility. Exact descriptor equality must not substitute for semantic comparison.
• Semantic compatibility requires the same logical kind and compatible CRS/edge semantics; Geography and Geometry are never implicitly compatible. Encoding does not affect function overload resolution, but pass-through and sink compatibility require a supported encoding or an explicit semantics-preserving conversion. Initially, the supported payload encoding is WKB.
• UNKNOWN and MIXED dimensions are allowed for scan/pass-through and serialization; they are not proof that every row is XY. Compute functions validate supported row-level dimensions. Assignment to a constrained target must satisfy its dimension guarantee or fail explicitly; it must not silently relabel, truncate, or drop coordinates.
• Native primitive types remain authoritative; VARBINARY plus a geo descriptor is not a native-type substitute. FE native construction/conversion owns semantic validation, including non-geo attachments and inconsistent primitive/kind/coordinate/edge combinations. Contract 2.3 keeps BE TypeDescriptor conversion free of general capability validation, retaining debug assertions for geo attachments; these assertions are not controlled runtime rejection. The core runtime integration and activation work must establish controlled validation/error boundaries before native execution is enabled.

Representation invariants

• ISO/OGC WKB is the canonical external payload format. This format contract does not require eager byte normalization during scans; valid source WKB is preserved. WKB alone does not preserve logical kind, CRS, or edge algorithm.
• S2, GEOS, H3, and legacy GeoShape internal bytes are not persisted as native geo values.
• Legacy VARCHAR spatial functions and legacy GeoShape payloads remain unchanged during rollout.
• Migration from legacy values requires explicit conversion functions; legacy values are never silently reinterpreted as native geo values.
• Initial GEOGRAPHY uses OGC:CRS84: X is longitude, Y is latitude, and edges are spherical. SRID 4326/EWKB may be accepted and canonicalized only at explicit SQL boundaries.

Unsupported operations, runtime, and result contracts

• Reject native geo equality (including null-safe equality), grouping and DISTINCT, ordering, geo join keys, implicit casts to or from VARCHAR/VARBINARY, and unsupported sinks. FE analysis must report clear errors; BE must return controlled errors if an unsupported operation bypasses FE checks. Explicit reviewed constructors and serializers remain the supported conversion boundaries.
• Disable ordinary min/max, dictionary, bloom-filter, and geo-bound pruning for geo columns until a separate spatial-pruning design is reviewed. Dictionary encoding/decoding for storage remains distinct from dictionary-based predicate pruning. Unrelated supported columns retain their existing pruning behavior.
• Preserve raw WKB during scans without eager normalization. Parsing must have explicit bounds on payload size, nesting depth, coordinate counts, allocation, and work. Validate buffer lengths and counts before allocation or access.
• Define ownership and lifetime for parsed-geometry caches: account memory, bound retained entries/bytes, prevent references from outliving source buffers, and invalidate cached objects when payload or descriptor changes. Parsed objects are execution-local and never serialized as persistent or interchange payloads.
• Initial GEOGRAPHY MySQL results use raw WKB bytes through the existing VARBINARY output path and its column metadata, not a server-generated hex string or GeoJSON. This is a public output representation, not an implicit SQL cast or a replacement for native internal identity. Any later GeoJSON/text or configurable output format requires separate review and applies only at the public result boundary. Exchange, spill, and intermediate values remain native WKB plus primitive type and descriptors.
• Gate type transport, native reads, public rendering, constructors, and compute separately, with explicit prerequisites. Invalidate cached Iceberg schemas and revalidate dependent plans when gates or capabilities change. Native GEOMETRY remains disabled throughout the GEOGRAPHY milestone.
• Iceberg writes and other unsupported sinks remain disabled until separately reviewed and gated.

Incremental implementation plan

The architecture above is the long-term target. Delivery is deliberately ordered so that connector interoperability is established and reviewed before StarRocks commits to engine-wide native geo types.

Milestone 1 — Iceberg/Parquet geo compatibility foundation

Goal: recognize and preserve standardized external geo metadata without exposing a new StarRocks SQL type or changing execution semantics.

• Contract 1.1 — Iceberg v3 schema recognition: recognize Iceberg v3 geography(CRS, edge_algorithm) and geometry(CRS), including defaults, and preserve their logical kind, CRS, edge algorithm, and geospatial annotation metadata. PR: #78700.
• Contract 1.2 — Parquet logical annotations: parse Parquet GEOGRAPHY and GEOMETRY logical annotations on BYTE_ARRAY, preserve their parameters, and establish ISO/OGC WKB as the canonical external payload contract. PR: #78701 (wire-definition foundation).
• Contract 1.3 — Schema validation and compatibility behavior: validate Iceberg/Parquet logical-kind, CRS, edge-algorithm, physical-type, and annotation conflicts. Disable ordinary min/max, dictionary, bloom-filter, and geo-bound pruning for geo columns. While the native feature gate is off, keep affected columns unsupported/UNKNOWN_TYPE; never silently expose annotated geo bytes as ordinary VARBINARY. PR: #78713.

Milestone 1 is connector-focused. It does not add public GEOGRAPHY/GEOMETRY SQL types, GeoColumn, GEOS/S2 execution, native-table persistence, or spatial SQL functions.

Milestone 2 — Shared native type and transport foundation

Goal: introduce the common runtime representation only after the external schema contract is established.

• Contract 2.1 — Descriptor and transport contract: define GeoTypeDesc, GeoStorageDesc, distinct edge algorithms, compatible Thrift/protobuf schemas, standalone BE descriptor round trips, and semantic comparison. Keep the first PR focused on descriptors, transport, and semantic comparison. Assignment and UNION ALL BE helpers are deferred until concrete callers are introduced, with FE/BE consistency tests; FE remains responsible for SQL compatibility and result dimensions. Native GEOGRAPHY/future GEOMETRY primitive-type integration is split into Contract 2.3 (protocol/BE) and Contract 2.4 (FE), with authority and consistency checks. Until then, keep native TypeDescriptor integration disabled and do not use VARBINARY as a proxy geo type. PR: #78916
• Contract 2.2 — Vectorized column: introduce WKB-backed GeoColumn with GeoColumnDesc while keeping GEOGRAPHY and GEOMETRY as distinct logical types. Define bounded parsing, memory accounting, and parsed-cache ownership/lifetime under the runtime contract above. PR: #78986.
• Contract 2.3 — Protocol and BE native type integration: add distinct GEOGRAPHY/GEOMETRY primitive identities, optional semantic GeoTypeDesc fields directly in scalar Thrift/protobuf types, and BE TypeDescriptor round trips with value-owned metadata. Keep GEO in the existing scalar list and provide minimal runtime traits and default GeoColumn construction so unchanged ColumnHelper and FunctionHelper factories use general dispatch, including nested arrays/maps/structs. Generic creation produces untyped physical placeholders with WKB, unknown dimension and unvalidated storage state; it does not propagate TypeDescriptor.geo_type or infer logical kind, CRS/SRID or edge semantics. Explicit descriptor construction and typed clone/copy preserve metadata; incompatible typed-to-placeholder copies fail rather than relabel payloads. Reject unsupported scalar access, rendering, range bounds and visitor operations inside GeoColumn, without scattered caller checks or capability-filtered dispatch lists. Generic ColumnViewer, including all-NULL inputs, remains unsupported. FE owns semantic validation; BE descriptor conversion retains debug assertions rather than general runtime validation. Reject direct/nested GEO storage-schema conversion with NotSupported. Descriptor-aware generic creation remains deferred; the former Contract 2.5 work is replanned under Milestone 3 and its required core prerequisites; native reads, SQL activation and payload serde remain outside this slice. PR: #78999.
• Contract 2.4 — FE native type integration: depend on Contract 2.3 and introduce distinct FE geo types, immutable metadata, and Thrift/protobuf descriptor conversions. Reject geo JOIN keys, GROUP BY, ORDER BY, DISTINCT, and distribution keys through existing centralized FE capability checks, including nested geo fields. Preserve ordinary type behavior and keep native SQL activation disabled. PR: #79005 (depends on #78999). Contract 2.5 — Previous runtime/propagation umbrella (replanned, not completed): the next deliverables move to Milestone 3 as three separate PRs: 3.1a MySQL WKB output, 3.1b CN transport, and 3.1c gated end-to-end Iceberg GEOGRAPHY reads. This is a scope/sequence change, not completion of the former umbrella. Descriptor-aware shared creation, descriptor ownership/lifetimes, and necessary value/column interfaces remain prerequisites for the paths that consume them; implement only the required pieces in focused reviewed changes, not one broad runtime PR. Preserve primitive and semantic/storage metadata without silent relabeling. Remaining spill integration, broader viewer/function-result support, nested end-to-end paths, and unsupported-operation work remain deferred and must be reviewed before their respective activation. No implicit binary equality/order/hash semantics are enabled.

Contracts 2.3 and 2.4 separate protocol/BE and FE review as requested by Alvin. The merged foundation includes minimal traits and default GEO creation through general dispatch, but generic factories still create untyped placeholders rather than descriptor-preserving native values. It does not enable native SQL execution or establish every controlled runtime error boundary. As subsequently agreed with Alvin, continue with three separate PRs in Milestone 3, starting with GEOGRAPHY: MySQL output, then CN transport, then end-to-end Iceberg reads. Do not combine these into one PR or add a milestone between 2 and 3.

Milestone 3 — Native GEOGRAPHY for Iceberg

Goal: turn the preserved Iceberg metadata and WKB payload into a usable native GEOGRAPHY value behind explicit capability checks.

Contract 3.1 — Initial GEOGRAPHY SELECT path, delivered in three separate PRs: follow this order, as agreed with Alvin. The sublabels preserve existing references to Contracts 3.2/3.3.

• Contract 3.1a — MySQL WKB output (first PR): output GEOGRAPHY WKB using the existing VARBINARY binary-result path and column metadata. Preserve raw WKB bytes and SQL NULL behavior; do not hex-encode on the server or add GeoJSON/configurable formats in this PR. A client may choose its own display of binary bytes. Keep native logical identity and descriptors internally. This PR alone does not enable CN transport or native Iceberg reads. PR: #79035
• Contract 3.1b — CN transport (second PR): implement correct and efficient serialization/deserialization between CNs, preserving WKB, GEOGRAPHY identity, semantic/storage descriptors and NULLs. Include the core creation/value-access support needed for this path without scattered GEO checks. Validate buffer bounds, ownership, malformed input and unsupported-peer behavior. Do not expand this PR to Iceberg activation, public text rendering, or unrelated runtime operations. PR: #79043.
• Contract 3.1c — Gated Iceberg native read path (third PR): combine the reviewed output and CN transport paths to make SELECT * FROM iceberg_table work end to end for native GEOGRAPHY. Enable reads only when participating components advertise the required support; otherwise retain Milestone 1 unsupported behavior. Preserve source WKB without eager normalization and retain primitive/descriptor identity throughout scan and materialization. Separately gate transport/reads/output/constructors/compute and invalidate cached Iceberg schemas when gates or capabilities change. Keep native GEOMETRY, Iceberg writes, new SQL constructors and compute functions out of this PR. PR: #79110.

The first three PRs start with GEOGRAPHY only. Contracts 3.2 and 3.3 below remain later work; they are not prerequisites to bundle into the initial SELECT path.

• Contract 3.2 — GEOGRAPHY SQL boundary: start with OGC:CRS84 longitude/latitude, spherical edges, longitude in [-180, 180], latitude in [-90, 90], and all seven OGC geometry families plus EMPTY. Add reviewed WKT/WKB constructors and serializers, with any later GeoJSON rendering reviewed separately and confined to the public result boundary; initial SELECT output remains WKB via the VARBINARY path. Construction/serialization support for all seven families and EMPTY does not imply compute support for every family. Enforce the unsupported-operation contract with FE checks and BE fallbacks.
• Contract 3.3 — Initial GEOGRAPHY function overloads: introduce exactly ST_X(GEOGRAPHY), ST_Y(GEOGRAPHY), ST_GEOMETRYTYPE(GEOGRAPHY), and ST_DISTANCE(GEOGRAPHY, GEOGRAPHY). ST_X/ST_Y accept POINT values and return longitude/latitude in degrees; ST_GEOMETRYTYPE identifies any supported geometry family. ST_DISTANCE initially accepts only POINT/POINT inputs and returns distance in meters under the reviewed CRS84 spherical contract. Specify NULL/EMPTY behavior and controlled errors for unsupported input families, dimensions, and descriptors. GEOMETRY overloads remain disabled in this milestone and are added by Contract 4.3. Broader predicates, measurements, and input-family extensions require separate review and capability gates.

Milestone 4 — Planar GEOMETRY and typed SQL API (proposed for review)

Goal: enable planar GEOMETRY using the shared native representation and establish type-specific SQL overloads. This expands the previously deferred planar-geometry scope; it does not change Milestones 1–3 or enable GEOMETRY during the GEOGRAPHY milestone.

• Contract 4.1 — GEOMETRY semantics and activation: enable native planar GEOMETRY after the shared type/transport foundation and Milestone 3, behind separately reviewed capabilities. Define CRS/SRID identity, axis order, coordinate units, supported dimensions, and behavior for unspecified CRS. Require compatible CRS/semantic descriptors for binary operations; reject mixed GEOGRAPHY/GEOMETRY operands and incompatible SRIDs without automatic conversion or reprojection. Define GEOS execution integration while keeping GEOS objects execution-local. Constructor/compute activation does not automatically enable external GEOMETRY reads, native-table persistence, or unsupported sinks.
• **Contract