Support HDFS as a storage backend

Background

HDFS is widely used in enterprise data platforms, and many teams want to use Lance as their columnar format on top of HDFS. Full HDFS support is currently blocked by two upstream dependencies.

Related PR

• lance-format/lance#5472 — HDFS support for Lance

Blocking dependencies

This PR is currently blocked by two upstream issues:

1. hdrs — missing HDFS copy support

The hdrs crate (used by Lance's HDFS object store provider) does not implement the copy operation. This causes failures when Lance needs to copy objects on HDFS, for example during optimize_indice on inverted indexes.

• Upstream PR: Xuanwo/hdrs#143

2. opendal — rename_with_if_not_exist

HdfsSafeCommitHandler need this.

• Upstream PR: apache/opendal#7815

Call for help

We'd appreciate the community's help in reviewing and merging the two upstream PRs so that HDFS support can land in Lance. If there are alternative approaches or workarounds, please share them in this issue or the PR discussion.