olake

OLake Go

OLake Go is a high-performance, open-source data ingestion engine for replicating databases, S3, and Kafka into Apache Iceberg (or plain Parquet).
Built for scalable, real-time pipelines, OLake Go provides a simple web UI and CLI - used to ingest into vendor-lock-in free Iceberg tables supporting all the query-engines/warehouses.

Read the docs and benchmarks at olake.io/docs. Join our active community on Slack.

> [!NOTE] > **OLake Fusion is now live!** — Automate your Apache Iceberg Table Maintenance. Check it out here → [github.com/datazip-inc/olake-fusion](https://github.com/datazip-inc/olake-fusion) ## OLake Go — Super-fast Sync to Apache Iceberg **OLake Go** supports replication from **transactional databases** such as **PostgreSQL, MySQL, MongoDB, Oracle, DB2, and MSSQL**, **event-streaming systems like Apache Kafka** and **Object-store like S3**, into open data lakehouse formats such as **Apache Iceberg** or **Plain Parquet** — delivering blazing-fast performance with minimal infrastructure cost.

--- ### Why OLake Go? - **Smart sync**: Full + CDC replication with automatic schema discovery & schema evolution - ⚡ **High throughput**: 580K RPS (Postgres) & 338K RPS (MySQL) - ➡️ **Exactly once delivery & Arrow writes**: Accuracy with speed. - **Iceberg-native**: Supports Glue, Hive, JDBC, REST catalogs - ️ **Self-serve UI**: Deploy via Docker Compose and sync in minutes - **Infra-light**: No Spark, no Flink, no Kafka, no Debezium --- ### Benchmarks #### Full Load | Source → Destination | Full Load | Relative Performance (Full Load) | Full Report | |----------------------|-----------------|--------------------------------------|--------------------------------------------------------------| | Postgres → Iceberg
(as of 30th Jan 2026) | 5,80,113 RPS | 12.5× faster than Fivetran | [Full Report](https://olake.io/docs/benchmarks?tab=postgres) | | MySQL → Iceberg
(as of 30th May 2026) | 1,39,773 RPS | 1.91× faster than Fivetran | [Full Report](https://olake.io/docs/benchmarks/?tab=mysql) | | MongoDB → Iceberg
(as of 5th Feb 2026) | 37,879 RPS | - | [Full Report](https://olake.io/docs/benchmarks/?tab=mongodb) | | Oracle → Iceberg
(as of 30th Jan 2026) | 5,26,337 RPS | - | [Full Report](https://olake.io/docs/benchmarks/?tab=oracle) | | Kafka → Iceberg
(as of 27th Feb 2026) | 2,09,065 MPS (Bounded Incremental) | 1.23x slower than Flink | [Full Report](https://olake.io/docs/benchmarks/?tab=kafka) | | MSSQL → Iceberg
(as of 09th June 2026) | 3,45,866 MPS | 4.32x faster than Fivetran | [Full Report](https://olake.io/docs/benchmarks/ingestion/?tab=mssql) | #### CDC | Source → Destination | CDC | Relative Performance (CDC) | Full Report | |----------------------|-----------------|--------------------------------------|--------------------------------------------------------------| | Postgres → Iceberg
(as of 30th Jan 2026) | 55,555 RPS | 2× faster than Fivetran | [Full Report](https://olake.io/docs/benchmarks?tab=postgres) | | MySQL → Iceberg
(as of 30th May 2026) | 59,951 RPS | 1.52× faster than Fivetran | [Full Report](https://olake.io/docs/benchmarks/?tab=mysql) | | MongoDB → Iceberg
(as of 5th Feb 2026) | 10,692 RPS | - | [Full Report](https://olake.io/docs/benchmarks/?tab=mongodb) | --- ### Supported Sources and Destinations #### Sources (Databases) | Source | Full Load | CDC | Incremental | Notes | Documentation | |---------------|--------------|---------------|-------------------|-----------------------------|-----------------------------| | PostgreSQL | ✅ | ✅ `pgoutput` | ✅ |`wal2json` deprecated |[Postgres Docs](https://olake.io/docs/connectors/postgres/overview) | | MySQL | ✅ | ✅ | ✅ | Binlog-based CDC | [MySQL Docs](https://olake.io/docs/connectors/mysql/overview) | | MongoDB | ✅ | ✅ | ✅ | Oplog-based CDC |[MongoDB Docs](https://olake.io/docs/connectors/mongodb/overview) | | Oracle | ✅ | WIP | ✅ | JDBC based Full Load & Incremental | [Oracle Docs](https://olake.io/docs/connectors/oracle/overview) | | DB2 | ✅ | - | ✅ | JDBC based Full Load & Incremental | [DB2 Docs](https://olake.io/docs/connectors/db2/) | | MSSQL | ✅ | ✅ | ✅ | Full Load, CDC & Incremental | [MSSQL Docs](https://olake.io/docs/connectors/mssql/) | #### Source (S3) | Source | Full Load | CDC | Incremental | Notes | Documentation | |---------------|--------------|---------------|-------------------|-----------------------------|-----------------------------| |S3 | ✅ | - | ✅ | Ingests from Amazon S3 or S3-compatible (MinIO, LocalStack) | [S3 Docs](https://olake.io/docs/connectors/s3/) | #### Source (Kafka) | Source | Bounded Incremental | Notes | Documentation | |--------|--------------------|-----------------------------------|---------------| | Kafka | ✅ | Latest offset bounded incremental sync | [Kafka Docs](https://olake.io/docs/connectors/kafka) | #### Destinations | Destination | Format | Supported Catalogs | |----------------|-----------|---------------------------------------------------------------| | Iceberg | ✅ | Glue, Hive, JDBC, REST (Nessie, Polaris, Unity, Lakekeeper, AWS S3 tables) | | Parquet | ✅ | Filesystem | ##### Writer Docs 1. Apache Iceberg Docs 1. Catalogs 1. [AWS Glue Catalog](https://olake.io/docs/writers/iceberg/catalog/glue) 2. [REST Catalog](https://olake.io/docs/writers/iceberg/catalog/rest) - Generic - Lakekeeper - Nessie - S3 Tables - Unity - Apache Polaris 3. [JDBC Catalog](https://olake.io/docs/writers/iceberg/catalog/jdbc) 4. [Hive Catalog](https://olake.io/docs/writers/iceberg/catalog/hive) 2. [Azure ADLS Gen2](https://olake.io/docs/writers/iceberg/azure) 3. [Google Cloud Storage (GCS)](https://olake.io/docs/writers/iceberg/gcp/) 4. [MinIO (local)](https://olake.io/docs/writers/iceberg/troubleshooting-local/?view=local#local-testing) 2. Parquet Writer 1. [AWS S3 Docs](https://olake.io/docs/writers/parquet/config/) 2. [Google Cloud Storage (GCS)](https://olake.io/docs/writers/parquet/config/#using-gcs-compatible-s3-credentials) 3. Local FileSystem Docs --- ### Quickstart (UI + Docker) OLake UI is a web-based interface for managing OLake Go jobs, sources, destinations and configurations. You can run the entire OLake Go stack (UI, Backend, and all dependencies) using Docker Compose. This is the recommended way to get started. Run the UI, connect your source DB, and start syncing in minutes. ```sh curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -d ``` **Access the UI:** - **OLake UI:** [http://localhost:8000](http://localhost:8000) - Log in with default credentials: `admin` / `password` Detailed getting started using OLake UI can be found [here](https://olake.io/docs/getting-started/quickstart/). #### Creating Your First Job With the UI running, you can create a data pipeline in a few steps: 1. **Configure Source:** Navigate to **Source** tab and click **Create Source**. Set up your source connection. 2. **Configure Destination:** Navigate to **Destination** tab and click **Create Destination**. Set up your destination. 3. **Create a Job:** Navigate to the **Jobs** tab and click **Create Job**. 4. **Configure & Run:** Give your job a name, set a schedule, select your source and destination and click **Next** to finish. 5. **Select Streams:** Choose which tables to sync and configure their sync mode (`CDC`, `Full Refresh` or `Incremental`). For a detailed walkthrough, refer to the [Jobs documentation](https://olake.io/docs/getting-started/creating-first-pipeline/). --- ### ️ CLI Usage (Advanced) For advanced users and automation, OLake Go's core logic is exposed via a powerful CLI. The core framework handles state management, configuration validation, logging, and type detection. It interacts with drivers using four main commands: * `spec`: Returns a render-able JSON Schema for a connector's configuration. * `check`: Validates connection configurations for sources and destinations. * `discover`: Returns all available streams (e.g., tables) and their schemas from a source. * `sync`: Executes the data replication job, extracting from the source and writing to the destination. * `clear-destination` : Clears data in the destination, only for the selected streams defined in `streams.json`. **Find out more about CLI [here](https://olake.io/docs/install/docker-cli/).** --- #### Install OLake Go Below are other different ways you can run OLake Go: 1. [OLake Go UI (Recommended)](https://olake.io/docs/getting-started/quickstart/) 2. [Kubernetes using Helm](https://olake.io/docs/install/kubernetes) 3. [Standalone Docker container](https://olake.io/docs/install/docker-cli) 4. [Airflow on EC2](https://olake.io/blog/olake-airflow-on-ec2/) 5. [Airflow on Kubernetes](https://olake.io/blog/olake-airflow) --- ### Playground 1. [OLake Go + Apache Iceberg + REST Catalog + Presto](https://olake.io/docs/playground/olake-iceberg-presto) 2. [OLake Go + Apache Iceberg + AWS Glue + Trino](https://olake.io/iceberg/olake-iceberg-trino) 3. [OLake Go + Apache Iceberg + AWS Glue + Athena](https://olake.io/iceberg/olake-iceberg-athena) 4. [OLake Go + Apache Iceberg + AWS Glue + Snowflake](https://olake.io/iceberg/olake-glue-snowflake) 5. [OLake Go + Apache Iceberg + REST Catalog + Spark](https://olake.io/docs/getting-started/playground/) --- ### Use Cases - ✅ Migrate from OLTP to Iceberg without Spark or Flink - ✅ Enable BI over fresh CDC data using Athena, StarRocks, Trino, Presto, Dremio, Databricks, Snowflake and more! - ✅ Build near real-time data lake-house on cost-efficient clo