百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
M

moonlink

> 编程语言
开源

全面管理的流式数据采集 (CDC) 到您的 Lakehouse

351 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

全面管理的流式数据采集 (CDC) 到您的 Lakehouse

Overview

Moonlink is an Iceberg-native ingestion engine bringing streaming inserts and upserts to your lakehouse.

Ingest Postgres CDC, event streams (Kafka), and OTEL into Iceberg without complex maintenance and compaction.

Moonlink buffers, caches, and indexes data so Iceberg tables stay read-optimized.

…

Note: Moonlink is in preview. Expect changes. Join our Community to stay updated!

Why Moonlink?

Traditional ingestion tools write data and metadata files per update into Iceberg. That's fine for slow-changing data, but on real-time streams it causes:

  • Tiny data files — frequent commits create thousands of small Parquet files
  • Metadata explosion — equality-deletes compound this problem

which leads to:

  • Slow read performance — query planning overhead scales with file count
  • Manual maintenance — periodic Spark jobs for compaction/cleanup

Moonlink minimizes write amplification and metadata churn by buffering incoming data, building indexes and caches on NVMe, and committing read-optimized files and deletion vectors to Iceberg.

Inserts are buffered and flushed as size-tuned Parquet

          ┌───moonlink───┐  ┌────iceberg───┐
          │┌─ ─ ─ ─ ─ ─ ┐│  │┌─ ─ ─ ─ ─ ─ ┐│
raw insert│              │  │              │
────────► ││   Arrow    │├─►││   Parquet  ││
          │              │  │              │
          │└─ ─ ─ ─ ─ ─ ┘│  │└─ ─ ─ ─ ─ ─ ┘│
          └──────────────┘  └──────────────┘

Deletes are mapped to deletion vectors using an index built on row positions

           ┌───moonlink───┐   ┌────iceberg───┐
           │┌─ ─ ─── ─ ─ ┐│   │┌─ ─ ─ ─ ─ ─ ┐│
raw deletes│              │   │              │
  ────────►││   index    │├──►││  deletion  ││
           │              │   │   vectors    │
           │└─ ─ ─── ─ ─ ┘│   │└─ ─ ─ ─ ─ ─ ┘│
           └──────────────┘   └──────────────┘

Write Paths

Moonlink supports multiple input sources for ingest:

  1. PostgreSQL CDC — ingest via logical replication with millisecond-level latency
  2. REST API — simple HTTP endpoint for direct event ingestion
  3. Kafka — sink support coming soon
  4. OTEL — sink support on the roadmap

Read Path

Moonlink commits data as Iceberg v3 tables with deletion vectors. These tables can be queried from any Iceberg-compatible engine.

Engines

  1. DuckDB
  2. Apache Spark
  3. Postgres with pg_duckdb or pg_mooncake

Catalogs

  1. AWS Glue — coming soon
  2. Unity Catalog — coming soon

Real-Time Reads (<s freshness)

For workloads requiring sub-second visibility into new data, Moonlink supports real-time querying:

  1. DuckDB — with the duckdb_mooncake extension.
  2. Postgres — with the pg_mooncake extension.
  3. DataFusion – with Moonlink Datafusion

Quick Start

1. Clone & Build

Clone the repository and build the service binary:

bash
git clone https://github.com/Mooncake-Labs/moonlink.git
cd moonlink
cargo build --release --bin moonlink_service

2. Start the Moonlink Service

Start the Moonlink service, which will store data in the ./data directory:

bash
./target/release/moonlink_service ./data

3. Verify Service Health

Check that the service is running properly:

bash
curl http://localhost:3030/health

4. Create a Table

Create a table with a defined schema. Here's an example creating a users table:

bash
curl -X POST http://localhost:3030/tables/users \
  -H "Content-Type: application/json" \
  -d '{
    "database": "my_database",
    "table": "users",
    "schema": [
      {"name": "id", "data_type": "int32", "nullable": false},
      {"name": "name", "data_type": "string", "nullable": false},
      {"name": "email", "data_type": "string", "nullable": true},
      {"name": "age", "data_type": "int32", "nullable": true},
      {"name": "created_at", "data_type": "date32", "nullable": true}
    ],
    "table_config": {"mooncake": {"append_only": true}}
  }'

5. Insert Data

Insert data into the created table:

bash
curl -X POST http://localhost:3030/ingest/users \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "insert",
    "request_mode": "async",
    "data": {
      "id": 1,
      "name": "Alice Johnson",
      "email": "[email protected]",
      "age": 30,
      "created_at": "2024-01-01"
    }
  }'

Roadmap and Contributing

Roadmap (near‑term):

  1. Kafka sink preview
  2. Schema evolution from Postgres and Kafka
  3. Catalog integrations (AWS Glue, Unity Catalog)
  4. REST API stabilization (Insert, Upsert into Iceberg directly)

We’re grateful for our contributors. If you'd like to help improve Moonlink, join our community.

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Rust

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言