Fully Managed, Streaming Ingestion (CDC) into your Lakehouse
Fully Managed, Streaming Ingestion (CDC) into your Lakehouse
Moonlink is an Iceberg-native ingestion engine bringing streaming inserts and upserts to your lakehouse.
Ingest Postgres CDC, event streams (Kafka), and OTEL into Iceberg without complex maintenance and compaction.
Moonlink buffers, caches, and indexes data so Iceberg tables stay read-optimized.
…Note: Moonlink is in preview. Expect changes. Join our Community to stay updated!
Traditional ingestion tools write data and metadata files per update into Iceberg. That's fine for slow-changing data, but on real-time streams it causes:
which leads to:
Moonlink minimizes write amplification and metadata churn by buffering incoming data, building indexes and caches on NVMe, and committing read-optimized files and deletion vectors to Iceberg.
Inserts are buffered and flushed as size-tuned Parquet
┌───moonlink───┐ ┌────iceberg───┐
│┌─ ─ ─ ─ ─ ─ ┐│ │┌─ ─ ─ ─ ─ ─ ┐│
raw insert│ │ │ │
────────► ││ Arrow │├─►││ Parquet ││
│ │ │ │
│└─ ─ ─ ─ ─ ─ ┘│ │└─ ─ ─ ─ ─ ─ ┘│
└──────────────┘ └──────────────┘Deletes are mapped to deletion vectors using an index built on row positions
┌───moonlink───┐ ┌────iceberg───┐
│┌─ ─ ─── ─ ─ ┐│ │┌─ ─ ─ ─ ─ ─ ┐│
raw deletes│ │ │ │
────────►││ index │├──►││ deletion ││
│ │ │ vectors │
│└─ ─ ─── ─ ─ ┘│ │└─ ─ ─ ─ ─ ─ ┘│
└──────────────┘ └──────────────┘Moonlink supports multiple input sources for ingest:
Moonlink commits data as Iceberg v3 tables with deletion vectors. These tables can be queried from any Iceberg-compatible engine.
Engines
pg_duckdb or pg_mooncakeCatalogs
For workloads requiring sub-second visibility into new data, Moonlink supports real-time querying:
duckdb_mooncake extension.pg_mooncake extension.Moonlink DatafusionClone the repository and build the service binary:
git clone https://github.com/Mooncake-Labs/moonlink.git
cd moonlink
cargo build --release --bin moonlink_serviceStart the Moonlink service, which will store data in the ./data directory:
./target/release/moonlink_service ./dataCheck that the service is running properly:
curl http://localhost:3030/healthCreate a table with a defined schema. Here's an example creating a users table:
curl -X POST http://localhost:3030/tables/users \
-H "Content-Type: application/json" \
-d '{
"database": "my_database",
"table": "users",
"schema": [
{"name": "id", "data_type": "int32", "nullable": false},
{"name": "name", "data_type": "string", "nullable": false},
{"name": "email", "data_type": "string", "nullable": true},
{"name": "age", "data_type": "int32", "nullable": true},
{"name": "created_at", "data_type": "date32", "nullable": true}
],
"table_config": {"mooncake": {"append_only": true}}
}'Insert data into the created table:
curl -X POST http://localhost:3030/ingest/users \
-H "Content-Type: application/json" \
-d '{
"operation": "insert",
"request_mode": "async",
"data": {
"id": 1,
"name": "Alice Johnson",
"email": "[email protected]",
"age": 30,
"created_at": "2024-01-01"
}
}'Roadmap (near‑term):
We’re grateful for our contributors. If you'd like to help improve Moonlink, join our community.
No open issues yet, or sync has not completed.