Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
M

moonlink

> 编程语言
Open source

Fully Managed, Streaming Ingestion (CDC) into your Lakehouse

351 stars0 likes0 views
WebsiteGitHub

About

Fully Managed, Streaming Ingestion (CDC) into your Lakehouse

Overview

Moonlink is an Iceberg-native ingestion engine bringing streaming inserts and upserts to your lakehouse.

Ingest Postgres CDC, event streams (Kafka), and OTEL into Iceberg without complex maintenance and compaction.

Moonlink buffers, caches, and indexes data so Iceberg tables stay read-optimized.

…

Note: Moonlink is in preview. Expect changes. Join our Community to stay updated!

Why Moonlink?

Traditional ingestion tools write data and metadata files per update into Iceberg. That's fine for slow-changing data, but on real-time streams it causes:

  • Tiny data files — frequent commits create thousands of small Parquet files
  • Metadata explosion — equality-deletes compound this problem

which leads to:

  • Slow read performance — query planning overhead scales with file count
  • Manual maintenance — periodic Spark jobs for compaction/cleanup

Moonlink minimizes write amplification and metadata churn by buffering incoming data, building indexes and caches on NVMe, and committing read-optimized files and deletion vectors to Iceberg.

Inserts are buffered and flushed as size-tuned Parquet

          ┌───moonlink───┐  ┌────iceberg───┐
          │┌─ ─ ─ ─ ─ ─ ┐│  │┌─ ─ ─ ─ ─ ─ ┐│
raw insert│              │  │              │
────────► ││   Arrow    │├─►││   Parquet  ││
          │              │  │              │
          │└─ ─ ─ ─ ─ ─ ┘│  │└─ ─ ─ ─ ─ ─ ┘│
          └──────────────┘  └──────────────┘

Deletes are mapped to deletion vectors using an index built on row positions

           ┌───moonlink───┐   ┌────iceberg───┐
           │┌─ ─ ─── ─ ─ ┐│   │┌─ ─ ─ ─ ─ ─ ┐│
raw deletes│              │   │              │
  ────────►││   index    │├──►││  deletion  ││
           │              │   │   vectors    │
           │└─ ─ ─── ─ ─ ┘│   │└─ ─ ─ ─ ─ ─ ┘│
           └──────────────┘   └──────────────┘

Write Paths

Moonlink supports multiple input sources for ingest:

  1. PostgreSQL CDC — ingest via logical replication with millisecond-level latency
  2. REST API — simple HTTP endpoint for direct event ingestion
  3. Kafka — sink support coming soon
  4. OTEL — sink support on the roadmap

Read Path

Moonlink commits data as Iceberg v3 tables with deletion vectors. These tables can be queried from any Iceberg-compatible engine.

Engines

  1. DuckDB
  2. Apache Spark
  3. Postgres with pg_duckdb or pg_mooncake

Catalogs

  1. AWS Glue — coming soon
  2. Unity Catalog — coming soon

Real-Time Reads (<s freshness)

For workloads requiring sub-second visibility into new data, Moonlink supports real-time querying:

  1. DuckDB — with the duckdb_mooncake extension.
  2. Postgres — with the pg_mooncake extension.
  3. DataFusion – with Moonlink Datafusion

Quick Start

1. Clone & Build

Clone the repository and build the service binary:

bash
git clone https://github.com/Mooncake-Labs/moonlink.git
cd moonlink
cargo build --release --bin moonlink_service

2. Start the Moonlink Service

Start the Moonlink service, which will store data in the ./data directory:

bash
./target/release/moonlink_service ./data

3. Verify Service Health

Check that the service is running properly:

bash
curl http://localhost:3030/health

4. Create a Table

Create a table with a defined schema. Here's an example creating a users table:

bash
curl -X POST http://localhost:3030/tables/users \
  -H "Content-Type: application/json" \
  -d '{
    "database": "my_database",
    "table": "users",
    "schema": [
      {"name": "id", "data_type": "int32", "nullable": false},
      {"name": "name", "data_type": "string", "nullable": false},
      {"name": "email", "data_type": "string", "nullable": true},
      {"name": "age", "data_type": "int32", "nullable": true},
      {"name": "created_at", "data_type": "date32", "nullable": true}
    ],
    "table_config": {"mooncake": {"append_only": true}}
  }'

5. Insert Data

Insert data into the created table:

bash
curl -X POST http://localhost:3030/ingest/users \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "insert",
    "request_mode": "async",
    "data": {
      "id": 1,
      "name": "Alice Johnson",
      "email": "[email protected]",
      "age": 30,
      "created_at": "2024-01-01"
    }
  }'

Roadmap and Contributing

Roadmap (near‑term):

  1. Kafka sink preview
  2. Schema evolution from Postgres and Kafka
  3. Catalog integrations (AWS Glue, Unity Catalog)
  4. REST API stabilization (Insert, Upsert into Iceberg directly)

We’re grateful for our contributors. If you'd like to help improve Moonlink, join our community.

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Rust

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言