[Roadmap] Apache SeaTunnel 3.0.0 Roadmap

[Roadmap] Apache SeaTunnel 3.0.0 Roadmap Task

Hello guys,

Thanks everyone for the active discussion around the Apache SeaTunnel 3.0.0 roadmap.

This is a roadmap draft based on the proposals collected in the discussion so far. It is only a discussion summary, not a final decision. Please feel free to correct, supplement, or volunteer for any item.

Roadmap for Apache SeaTunnel 3.0.0:

• Zeta Runtime Metadata and SQL Extensibility

  • [1] [Accepted] [Feature][Zeta] Add datasource SPI #10586
  • [2] [Accepted] [Feature][Zeta] Add /option-rules REST API for runtime connector metadata #10585
  • [3] [Accepted] [Feature][Transform] Add Zeta SQL UDF context and lifecycle hooks with E2E and docs updates #10489

• Runtime and Connector Semantics

  • [4] [Accepted] [Feature][Zeta] STIP-23 Phase 1: Add engine timer flush core flow and FlushSignal handling #10800
  • [5] [Pending] [Feature][Connector-V2] STIP-23 Phase 2: JDBC connector adoption for engine timer flush #10801
  • [6] [Pending] [Docs][Connector-V2] STIP-23 Phase 3: Add timer flush feature flag to all sink connector docs #10802
  • [7] [Accepted] [Feature][Core] Support table-level fault isolation for multi-table synchronization #10600
  • [8] [Accepted] [Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention for file source continuous discovery #10563
  • [9] [Accepted] [Feature][Core] Error Data Bypass Capability for Transform and Sink, including Dead Letter Queue support #10196 #10306

• Build, Release, and Project Structure

  • [10] [Accepted] [Feature][Shade] Refactor the seatunnel-shade modules into an independent seatunnel-shade project, publish reusable shade jars to Maven Central, reduce repeated build costs for heavy modules such as Hadoop3 Uber and Hadoop AWS, and improve overall release stability and packaging efficiency #9993

• Zeta Engine Observability

  • [11] [Accepted] [Feature] Add StainTrace for end-to-end data lineage tracing and performance analysis in Zeta #10491
  • [12] [Accepted] [Zeta] Add observability metrics for engine state stores backed by Hazelcast IMap #10860
  • [13] [Accepted] [Feature][Core] Support real-time observability for Zeta engine tasks, including task busyness, queue, and backpressure visibility #10463

• Zeta State Management and Storage Evolution

  • [14] [Accepted] [Feature][Zeta] Decouple Hazelcast IMap via StateStore abstraction #10812
  • [15] [Pending] [Feature][Zeta] Add compaction support to IMap external storage #10399 (This is not included in the 3.0.0 Roadmap when discussed with PR contributor)

• AI for Data Engineering

  • [16] [Accepted] [Feature][CLI] Add SeaTunnel CLI for natural language configuration generation / AI-Native CLI #10789

• Performance and Internal Data Model (This is not included in the 3.0.0 Roadmap when discussed with PR contributor)

  • [17] [Pending] [STIP-27] Introduce Apache Arrow as an optional internal data model in SeaTunnel to improve columnar I/O performance, enable vectorized execution for batch transforms such as projection, filter, and cast, and reduce JVM overhead in scenarios such as Parquet, ORC, ClickHouse, and Doris #10806

• Edge Collection This is an Experimental feature.

  • [18] [Accepted] [Feature][Zeta] STIP-24 Phase 1: Support lightweight EdgeSocket ingress for edge collector MVP, updated PR #10878
  • [19] [Accepted] [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent MVP and dist integration, updated PR #10879

• Cloud-native Deployment and Compatibility

  • [20] [Accepted] Improve the SeaTunnel 3.0.0 cloud-native deployment experience, including Kubernetes and Helm support, deployment and restart behavior, persistent checkpoint storage, configuration management, secret management, rolling upgrade, resource isolation, and operational guidance

Next steps:

• Decide which items should be included in the Apache SeaTunnel 3.0.0 roadmap.
• Decide which items should be marked as experimental.