Support Apache Ozone as Deep Storage
Description
We would like to request support for Apache Ozone as a deep storage option in Apache Druid.
Apache Ozone provides an S3-compatible object store and is designed to integrate with the Hadoop ecosystem. Since the Hadoop version used by the Druid druid-hdfs-storage extension is compatible with the Hadoop libraries required by Apache Ozone, we believe Ozone could potentially be supported without introducing a completely separate storage implementation.
We would therefore like to ask whether support for Apache Ozone could be added either:
- To the existing
druid-hdfs-storageextension, leveraging the Hadoop/Ozone filesystem integration; or - Through a dedicated Druid Ozone storage extension, if integrating it into
druid-hdfs-storageis not considered appropriate.
Motivation
We are using Apache Ozone as the object storage layer of our Big Data platform and would like to use it as Druid's deep storage as well.
Apache Ozone provides a Hadoop-compatible filesystem interface, allowing applications to access Ozone through the Hadoop FileSystem API. This could potentially allow Druid to access Ozone directly through its existing Hadoop-based deep storage implementation.
Although Apache Ozone provides an S3-compatible API, relying on its S3 Gateway is not ideal for our use case, as it introduces an additional access layer and associated request overhead. Direct access through Ozone's native Hadoop FileSystem integration would avoid this extra layer and provide lower-latency access to deep storage.
Source: apache/druid