[UDT] Delivery

MySQL User Defined Type project

[!CAUTION] STATUS: DRAFT

Delivery

This is mostly a concern with project management.

The overall scope is very significant, with many different areas to change independently.

In terms of features, new capabilities are likely to be implemented incrementally, like a new type trait (arrays of T, not just T), new concepts (user defined INDEX on top of a user defined TYPE).

Users will not want to wait for the project total completion to start using basic features implemented very early.

Also, different parts of the code base (parser, data dictionary, optimizer, services, components, storage engines, replication, ...) will be implemented, reviewed, tested independently, typically by different persons or teams.

It is not desirable to have code held in a PR waiting to be used, if it can be delivered when ready.

For a project of this scope, keeping too much code in a branch in-definitively leads to project failures, because the branch diverges and becomes unmaintainable, struggling to keep up with upstream changes, while hitting a final review wall.

The proposed guiding principles for code delivery to mysql-trunk are as follows:

• Use a dedicated branch for the project, to integrate code from various sources.
• Allow incremental delivery to trunk, for code within a well defined area.
  • Formal code review for each deliverable is required before the merge, as a gate to reach production code (mysql-trunk)
  • This is consistent with current development practices for bug fixes and enhancements.
• Use feature flags to gate new features in the trunk branch, to ensure overall stability for MySQL.
  • For example, the parser syntax may be present in trunk, but CREATE TYPE might say UNSUPPORTED unless the feature flag for user defined types is opened.
  • The incremental risk to have a CREATE TYPE statement that returns an unsupported error (with code merged) instead of a syntax error (with code is not merged) is minimum.
• Over time, mature code part of the development branch are merged to mysql-trunk, which decreases the burden for the project
• When all the parts required to support a use case end to end are merged to trunk, the feature flag can be opened, exposing the new, complete, functionality. Multiple feature flags can be defined to deliver each new capabilities over time.

Regarding target versions, the proposal is:

• to deliver in the trunk main branch only (releases 26.7, 26.10, 27.1, 27.4, ...), all published every 3 months.
• no back port to LTS releases 8.4 or 9.7

Cross releases maintainability

For maintenance of previous releases (8.4-LTS, 9.7-LTS), bug fixed in previous releases must also be merged upstream to the main branch (trunk).

Heavy refactoring in trunk will increase the risk of merge collisions, increasing burden in the medium term, until 9.7 reach EOL. Implementing voluntarily minimal changes in trunk, will decrease merge collisions in the short term, but increases the risk to have code more difficult to maintain in the long term, even after 9.7 reached EOL.

In particular, the representation of a data type in the code base is likely to be impacted, and the current code structure is not designed to support the changes needed.

An additional project management constraint is to find the proper design that mitigates:

• maintainability of the current branch alone (trunk)
• maintainability of merges from older releases.

---

Proposed plan (draft)

Milestone 1: confirm feasibility

There is technical risk overall, about feasibility. Unknown areas must be identified, investigated, until we have a good grasp of the overall technical solution.

This milestone uses proof-of-concept (poc) code only.

Entry criteria:

• GitHub is opened for community contributions

Tasks:

• poc-parser
• poc-create-type
• poc-delete-type
• poc-data-dictionary
• poc-mdl-type
• poc-runtime
• poc-udt-services
• poc-udt-demo-component

Exit criteria:

• decision to continue or cancel this project, due to technical risk and/or development effort.
• no code is merged into mysql-trunk (main) during this milestone.

Upon decision to continue, enter milestone 2.

Milestone 2: bootstrap components

Entry criteria:

• Decision to proceed, and deliver incrementally under a feature flag.

Tasks:

• Define a CMAKE compiling option, WITH_EXPERIMENTAL_UDT
• parser
  • Finalize SQL parser syntax
  • Finalize naming (lower case table names, name collisions with with stored functions, UDF)
• mdl-type
• runtime
• udt-services
• udt-demo-component
• Stored procedures
• Storage engines

Each task is merged into mysql-trunk (main) when ready. At all times, the code in main is protected by the WITH_EXPERIMENTAL_UDT flag:

• Code in main should build
• User defined type code paths are not reachable (the feature is disabled), to ensure stability.

Exit criteria:

• The server contains minimal code to invoke a component implementing a user defined type.
• Services are available for a third party to implement a component exposing a user defined type.

All is not ready for production (no privileges), but finalizing the server and components can proceed in parallel.

Milestone 3: strengthen (single instance)

Tasks:

• privileges
  • GRANT CREATE TYPE
  • GRANT ALTER TYPE
  • GRANT DELETE TYPE
• create-type
  • Implement privileges checks
• alter-type
  • Implement privileges checks
  • Implement constraints checks (can not alter a type if used)
• delete-type
  • Implement privileges checks
  • Implement constraints checks (can not delete a type if used)
• data-dictionary
  • Expand INFORMATION_SCHEMA.TYPES view with more columns
  • Define/adjust more views to find type usage:
    • Find all tables/columns using a type
    • Find all stored procedures using a type
• runtime
  • Implement strict type conversions

Exit criteria:

• The server contains production quality code to invoke a component implementing a user defined type, in a single instance.
• Features are still missing, but the core runtime to use a user defined type is functional.

Milestone 3: strengthen (full deployment)

Tasks:

• replication
• upgrade / downgrade of MySQL version
• upgrade / downgrade of component implementation for a type
  • Provide an example (USBN10 to USBN13), document guidance
• Backup / Restore
• Clone plugin
• Observability
  • Opentelemetry traces, samplers
  • Instrument user defined types usage, operations

Milestone 4: UDT GA

Tasks:

• integration testing
• performance benchmarks
• full user documentation (using a UDT in an application)
• full implementer SDK documentation (implementing a UDT component)
• Roll out (blogs)

Exit criteria:

• All the code is merged into mysql-trunk (main)
• The WITH_UDT compiling flag is available
• The option is no longer experimental (renamed from WITH_EXPERIMENTAL_UDT)