incubator-xtable

Apache XTable™ (Incubating)

Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines. Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake.

Apache XTable™ simplifies data lake operations by leveraging a common model for table representation. This allows users to write data in one format while still benefiting from integrations and features available in other formats. For instance, Apache XTable™ enables existing Hudi users to seamlessly work with Databricks's Photon Engine or query Iceberg tables with Snowflake. Creating transformations from one format to another is straightforward and only requires the implementation of a few interfaces, which we believe will facilitate the expansion of supported source and target formats in the future.

Building the project and running tests.

1. Use Java 11 for building the project. If you are using another Java version, you can use jenv to use multiple Java versions locally.
2. Build the project using ./mvnw clean package. Use ./mvnw clean package -DskipTests to skip tests while building.
3. Use ./mvnw clean test or ./mvnw test to run all unit tests. If you need to run only a specific test you can do this by something like ./mvnw test -Dtest=TestDeltaSync -pl xtable-core.
4. Similarly, use ./mvnw clean verify or ./mvnw verify to run integration tests.

Automation and coding agents should also follow the repository-specific workflow in AGENTS.md, especially for dependency updates, shaded modules, and targeted validation.

Note: When using Maven version 3.9 or above, Maven automatically caches the build. To ignore build caching, you can add the -Dmaven.build.cache.enabled=false parameter. For example, ./mvnw clean package -DskipTests -Dmaven.build.cache.enabled=false

Style guide

1. We use Maven Spotless plugin and Google java format for code style.
2. Use ./mvnw spotless:check to find out code style violations and ./mvnw spotless:apply to fix them. Code style check is tied to compile phase by default, so code style violations will lead to build failures.

Running the bundled jar

1. Get a pre-built bundled jar or create the jar with ./mvnw install -DskipTests
2. Create a yaml file that follows the format below:

…

• sourceFormat is the format of the source table that you want to convert
• targetFormats is a list of formats you want to create from your source tables
• tableBasePath is the basePath of the table
• tableDataPath is an optional field specifying the path to the data files. If not specified, the tableBasePath will be used. For Iceberg source tables, you will need to specify the /data path.
• namespace is an optional field specifying the namespace of the table and will be used when syncing to a catalog.
• partitionSpec is a spec that allows us to infer partition values. This is only required for Hudi source tables. If the table is not partitioned, leave it blank. If it is partitioned, you can specify a spec with a comma separated list with format path:type:format
  • path is a dot separated path to the partition field
  • type describes how the partition value was generated from the column value
    • VALUE: an identity transform of field value to partition value
    • YEAR: data is partitioned by a field representing a date and year granularity is used
    • MONTH: same as YEAR but with month granularity
    • DAY: same as YEAR but with day granularity
    • HOUR: same as YEAR but with hour granularity
  • format: if your partition type is YEAR, MONTH, DAY, or HOUR specify the format for the date string as it appears in your file paths

1. The default implementations of table format converters can be replaced with custom implementations by specifying a converter configs yaml file in the format below:

…

1. A catalog can be used when reading and updating Iceberg tables. The catalog can be specified in a yaml file and passed in with the --icebergCatalogConfig option. The format of the catalog config file is:

catalogImpl: io.my.CatalogImpl
catalogName: name
catalogOptions: # all other options are passed through in a map
  key1: value1
  key2: value2

1. Run with java -jar xtable-utilities/target/xtable-utilities_2.12-0.5.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml [--hadoopConfig hdfs-site.xml] [--convertersConfig converters.yaml] [--icebergCatalogConfig catalog.yaml] The bundled jar includes hadoop dependencies for AWS, Azure, and GCP. Sample hadoop configurations for configuring the converters can be found in the xtable-hadoop-defaults.xml file. The custom hadoop configurations can be passed in with the --hadoopConfig [custom-hadoop-config-file] option. The config in custom hadoop config file will override the default hadoop configurations. For an example of a custom hadoop config file, see hadoop.xml.

Running using docker

1. Build the docker image using docker build . -t xtable
2. Mount the config files on the container and run the container:

docker run \
  -v ./xtable/config.yml:/xtable/config.yml \
  -v ./xtable/core-site.xml:/xtable/core-site.xml \
  -v ./xtable/catalog.yml:/xtable/catalog.yml \
  xtable \
  --datasetConfig /xtable/config.yml --hadoopConfig /xtable/core-site.xml --icebergCatalogConfig xtable/catalog.yml

Contributing

Setup

For setting up the repo on IntelliJ, open the project and change the Java version to Java 11 in File->ProjectStructure

Found a bug or have a cool idea to contribute? Open a GitHub issue to get started. For more contribution guidelines and ways to stay involved, visit our community page.

Adding a new target format

Adding a new target format requires a developer implement ConversionTarget. Once you have implemented that interface, you can integrate it into the ConversionController. If you think others may find that target useful, please raise a Pull Request to add it to the project.

Overview of the sync process