Epic: Stable Row IDs to production ready

This project tracks the work to bring Stable Row IDs—which have been experimental for some time—to be production ready. Not just something that we can optionally turn on, but something we are confident we can make the default.

There are several changes to make, discussed in depth in this doc: https://docs.google.com/document/d/1x6Xon1XNZ5XaXHekk5kU0Ujg0A9FCOSLAP1Ry9XTMx8/edit?tab=t.0

This supersedes previous projects:

• https://github.com/lance-format/lance/issues/2454
• https://github.com/lance-format/lance/issues/2307

Issues

• Bug fixes
  • https://github.com/lance-format/lance/issues/6734
  • https://github.com/lance-format/lance/issues/8466
  • https://github.com/lance-format/lance/issues/8851
  • https://github.com/lance-format/lance/issues/8848
  • https://github.com/lance-format/lance/issues/8850
• §5.1 Fragment Reuse Hardening
  • https://github.com/lance-format/lance/issues/8977
• §5.2 Secondary indexes store row addresses, not row IDs
  • https://github.com/lance-format/lance/issues/8085
• §5.3 Row ID sequences spill to data files as a hidden column
  • https://github.com/lance-format/lance/issues/9250
• §5.4 Fast row-ID lookup via a BTree index on _rowid
  • https://github.com/lance-format/lance/issues/9474