Epic: clustered indices

Description

Currently data is inserted into the database in insertion order. However, many databases allow users to define a primary index or "clustered index". Data in the table is then stored in sorted order (sorted by the index column). This can be very useful for various sharding / distributed execution strategies. Most of this work will need to happen in lance. However, lancedb will need to support defining a clustered index and potentially calling the appropriate periodic maintenance in lance.

Tasks

• Lance support for reordering data

Depends on:

• https://github.com/lancedb/lance/issues/2064