[Proposal] Operational insights & ETA overshoot on large UUID-keyed tables (140M+ rows)
Author: soepic1Created Sep 15, 2026Updated Sep 15, 2026
Background & Operational Context
During a live zero-downtime migration on a ~390GB, 144M row production table with a UUID (VARCHAR(36)) primary key on MySQL (Cloud SQL), we navigated two notable operational edge cases that we would like to share and propose improvements for:
Progress percentage overshoot & ETA calculation on UUID/String PKs:
- The migration progress consistently overshot 100% (reaching
139%,146%, and164.8%) displayingETA: duefor hours while actively copying keys. - Root Cause: When the iteration key is non-numeric (hexadecimal UUID strings), statistical estimation from
EXPLAINdiffers substantially from the actual distribution in the lexicographical traversal space (0000...toffff...), leaving operators without clear visibility into true completion times.
- The migration progress consistently overshot 100% (reaching
Throttled State vs. Postponed Cut-Over Lifecycle:
- Long-duration throttling via
--throttle-additional-flag-filepauses active heartbeat writes and changelog application. Over extended maintenance windows, this can cause MySQL client connections to exceedwait_timeout(28,800s idle limits) and disconnect. - In contrast, using
--postpone-cut-over-flag-file(or socketpostpone) allows row copying to finish 100% and maintains continuous lightweight binlog streaming (Lag: 0.04s, active heartbeats), completely preventing idle connection timeouts while safely waiting for a scheduled cut-over window.
- Long-duration throttling via
Proposed Contributions
We would like to contribute the following to gh-ost:
Documentation PR:
- Add an operational guide clarifying the operational differences between
--throttle-additional-flag-file(load shedding/pause) and--postpone-cut-over-flag-file(scheduled cut-overs) during multi-hour maintenance windows. - Document the behavior and interpretation of progress metrics when migrating tables with UUID/alphanumeric primary keys.
- Add an operational guide clarifying the operational differences between
Discussion on ETA / Metric Bounds:
- Discuss potential heuristics to detect string/UUID primary keys and cap or weight the progress estimation in
go/logic/migrator.goto avoid confusing>100% / ETA: dueoutput.
- Discuss potential heuristics to detect string/UUID primary keys and cap or weight the progress estimation in
We have a local branch ready and would love maintainer feedback before opening the documentation PR!
Source: github/gh-ost