Baike.dev
All toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
D

duckle

> 数据库
Open source

Open-source ETL/ELT on DuckDB. Write, wire, or draw one pipeline: 350+ components, 160+ connectors, dbt, CDC, data quality, a Python API, and MCP for AI agents.

898 stars0 likes0 views
WebsiteGitHub

About

Open-source ETL/ELT on DuckDB. Write, wire, or draw one pipeline: 350+ components, 160+ connectors, dbt, CDC, data quality, a Python API, and MCP for AI agents.


Quick links

Get started

  • Where Duckle runs
  • What is Duckle?
  • What's new in v0.7.3
  • What's new in v0.7.2
  • What's new in v0.7.1
  • What's new in v0.7.0
  • What's new in v0.6.1
  • What's new in v0.6.0
  • Quickstart (60 s)
  • Download / Install
  • Build from source
  • Run your first pipeline

Use the product

  • Meet Duckie (AI)
  • How to use Duckle
  • Recipes / examples
  • In-app Git (GitHub/GitLab)
  • Workspace + Git flow
  • Schedules
  • Plans
  • Server deployment
  • Sign-in and roles
  • How a request is decided
  • API keys, for machines
  • MCP server: connect Claude, Cursor or any agent
  • Connection management
  • Context variables

Reference

  • Capabilities matrix
  • Sources
  • Transforms
  • Sinks
  • Data quality
  • Custom code
  • Control flow
  • Advanced settings
  • Engines
  • Configuration

Resources

  • Architecture
  • Clean data for AI
  • Performance tips
  • FAQ
  • Troubleshooting
  • CI / CD
  • Status
  • Roadmap
  • Contributing
  • Sponsor Duckle
  • License
  • Releases
  • Roadmap doc
  • Contributing doc

What is Duckle?

An open-source ETL platform you run on your own infrastructure. Drag sources, transforms, validators and sinks onto a canvas, wire them together, and press Run. Duckle compiles the graph to SQL and executes it on a real columnar engine, with live previews, the generated SQL visible on every node, and no hidden state.

You build a pipeline on a laptop and deploy that same file to a server, where it runs on a schedule under a web console with roles, alerts and an audit log. Nothing is rewritten in between, and nothing is metered.

In short: a free, open-source, single-engine alternative to hosted, per-row-priced ETL platforms like Fivetran and Airbyte - one pipeline for ingest, transform, and load that runs anywhere, and can also run dbt on DuckDB inside the same tool.

Three things set it apart:

  1. An AI assistant that ships in the box. Describe the pipeline you want in English; Duckie writes the JSON and drops it onto the canvas. The model runs wherever Duckle does - no API key, no telemetry, no vendor round-trip. Point it at your own OpenAI-compatible endpoint instead if you would rather it did not run in-process.
  2. 400+ components ready at install time. Files, lakehouses, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, SaaS REST/GraphQL APIs, even FTP and IMAP - working today, not coming-soon.
  3. A self-contained binary you can audit. 73 to 110 MB depending on your platform. Engines install on first launch. Workspaces are plain files in a folder you choose. Diff them, branch them, ship them.

Why Duckle is different

Visual, never opaque The canvas compiles to SQL you can read, and every node has a live preview tab. No black box.
An assistant with no API key Runs in-process by default, or against your own OpenAI-compatible endpoint. Your prompts and your data stay inside your infrastructure either way.
Single-file binary, no bundled DB 73 to 110 MB depending on platform (it embeds the headless runner + MCP server). DuckDB downloads on first launch with a guided step. AI engine is opt-in.
Native speed Execution runs through DuckDB: vectorized, columnar, local. A clean-and-export job that crawls in a spreadsheet finishes in milliseconds.
Git-friendly by design Pipelines, connections, contexts, and routines persist as plain files in a folder you pick. Diff them, branch them, review them.
400+ components ready today Files, databases, warehouses, lakehouses, object stores, SaaS APIs, NoSQL, streaming brokers, vector DBs, FTP, IMAP, SMTP. Each is covered by tests.
Honest about scope Single-machine and embedded by design. Built to make local and small-team data work fast, not to replace a distributed warehouse.
60 UI languages Topbar, palette, chat assistant, properties panel, and common dialogs ship localized. English, Spanish, Chinese (Simplified + Traditional), Hindi, Arabic, Portuguese (Brazil), Bengali, Russian, Japanese, Punjabi, German, Korean, French, Vietnamese, Telugu, Marathi, Turkish, Tamil, Urdu, Persian, Polish, Italian, Ukrainian, Indonesian, Thai, Dutch, Hebrew, Swedish, Greek, Czech, Hungarian, Romanian, Filipino, Malay, Norwegian, Danish, Finnish, Catalan, Bulgarian, Slovak, Croatian, Serbian, Slovenian, Lithuanian, Latvian, Estonian, Khmer, Burmese, Sinhala, Nepali, Swahili, Afrikaans, Welsh, Irish, Icelandic, Albanian, Azerbaijani, Mongolian, Kazakh. RTL (Arabic, Hebrew, Persian, Urdu) supported. Switch languages from the topbar globe.
Open source Dual-licensed MIT OR Apache-2.0. Yours to use, fork, and extend.

Screenshots

Real pipelines, built and run in Duckle - not mockups.

A 5M-row pipeline: a CSV, a Parquet file, a DuckDB table, and a SQLite table enriched through one visual Map (3-way join), no SQL.

Left: the visual Map editor - main plus lookups, per-output expressions, an inline filter. Right: Parallelize fanning out aggregate, window, and top-N branches.

One run, many branches: 16 nodes finish in a few seconds. Concurrency auto-detects from CPU cores; branches write to Parquet, CSV, DuckDB, and SQLite at once.

Left: DuckLake CDC change-feed mirrored via upsert + delete propagation (100k rows). Right: watermark incremental load over 5M rows, advancing state only on a fully successful run.


Quickstart (60 seconds)

  1. Download the binary for your OS (see Download / Install above) - or build from source.
  2. Launch it. First run shows the setup modal:
    • Click Install on DuckDB (required, takes ~30 s).
    • Optionally click Install on Duckie AI Assistant (~1.1 GB, takes 5-10 min on average broadband).
  3. Pick a workspace folder. Pipelines, connections, context variables, and routines live there as plain files.
  4. Build a pipeline two ways:
    • Drag + wire: drag a CSV source in, point it at samples/orders.csv, hit Autodetect schema. Drag a Filter, wire it up. Drag a Parquet sink with an output path. Press Run, watch the nodes light up.
    • Ask Duckie: click the Sparkles icon (top-right of the toolbar), type "read orders.csv, filter where status = 'paid', write to paid.parquet". When Duckie streams back a pipeline, click Insert into canvas.
  5. Inspect. Click any node to see its generated SQL in the Plan tab and a live row sample in the Preview tab.

That's a real, native ETL pipeline built and run in under a minute. CSV is just the easiest first node; swap in Parquet, JSON, S3, Snowflake, MongoDB, or Stripe the same way.


Download / Install

Pick the binary for your OS from the latest release:

OS Asset How to run
Windows Duckle-windows-x64.exe Double-click. Unsigned binary - Windows SmartScreen will warn the first time; click "More info" -> "Run anyway".
macOS (Apple Silicon) Duckle-macos-arm64 chmod +x Duckle-macos-arm64 && ./Duckle-macos-arm64. Right-click -> Open the first time to bypass Gatekeeper.
Linux (x86_64) Duckle-linux-x64 chmod +x Duckle-linux-x64 && ./Duckle-linux-x64. Requires WebKitGTK 4.1 (libwebkit2gtk-4.1-0 on Debian / Ubuntu).

The single-file binary above is all you need for Build Pipeline too: the headless runner is embedded into the app at build time, and exporting a pipeline produces ONE self-contained executable (the engine, the DuckDB CLI, any needed extensions, and the resolved pipeline are all inside that one file). Copy that single file to your server and run or schedule it - no separate runner download required.

One command, nothing installed: it scaffolds sample data and a pipeline, compiles it to SQL, runs it on DuckDB, and shows you the rows.

uvx duckle quickstart

Let an agent do it

Paste this into Claude Code, Cursor, or Codex:

Run uvx duckle quickstart to build my first pipeline and run it

Nothing to install first. The agent fetches Duckle and the DuckDB engine on demand, runs a real pipeline, and shows you the rows.

CLI only (CI, cron, containers)

If you do not want the desktop studio, install just the headless runner. It is about 27 MB rather than 100 MB or more, has no GUI dependency, and is what a build step actually needs.

pip install duckle

That is the whole install. It brings the DuckDB CLI with it (via the duckdb-cli package published by the DuckDB Foundation), so there is nothing else to fetch and it works offline. Wheels ship for Linux, macOS and Windows on x86-64 and arm64.

It also gives you a Python API, where pipelines are built as code and executed by DuckDB rather than by Python:

import duckle
from duckle import col

(duckle.read_csv("orders.csv")
    .where(col.amount >= 20)
    .derive(total="round(amount * 1.2, 2)")
    .write_parquet("out.parquet")
    .run())

Python expressions compile to vectorized SQL at plan time, so no rows pass through the interpreter. See the PyPI page for the full API.

The same package provides the duckle command-line runner for CI, cron, and containers - it bundles the headless runner and the MCP server per platform:

pip install duckle          # or run ad hoc, no install: uvx duckle --help

Pipelines execute as SQL on the DuckDB CLI, so the runner needs a duckdb on PATH or DUCKLE_DUCKDB_BIN set (pip install duckdb-cli is the quickest route). Validation does not:

duckle validate                 # compile-check every pipeline under ./pipelines
duckle validate --json          # machine-readable, for a CI step
duckle --pipeline my.json       # run one

validate opens no source and writes no sink, so it needs no engine, no credentials and no network. Exit codes are stable: 0 clean, 1 a real finding (a pipeline failed or did not compile), 2 the runner could not start (bad usage, unreadable file, missing engine).

The binary is 73 to 110 MB depending on platform (it embeds the headless runner and the bundled MCP server). On first launch you'll be guided through downloading two engines into your app-data directory:

Engine Size Required? What it powers
DuckDB CLI ~30 MB + extensions Yes - cannot run pipelines without it Every source / transform / sink that runs as SQL
Duckie AI Assistant ~1.1 GB (llama-server + Qwen 2.5 Coder 1.5B GGUF) Optional The chat sidebar that generates pipelines from natural language

App-data location:

  • Windows: %APPDATA%\io.duckle.app\engines\
  • macOS: `~/Library/Application Supp

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Rustai-agentscdcconnectorsdata-engineering

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category数据库
PricingOpen source

> Related tools

P
PostgreSQL
功能强大的开源关系型数据库
R
Redis
内存数据结构存储,常用作缓存与队列
M
MySQL
广泛使用的开源关系型数据库