百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
S

skale

> 编程语言
开源

高性能分布式数据处理引擎

397 stars0 点赞5 次浏览
访问官网GitHub

工具介绍

高性能分布式数据处理引擎

Development activity is stopped, and this project is now archived.

High performance distributed data processing and machine learning.

Skale provides a high-level API in Javascript and an optimized parallel execution engine on top of NodeJS.

Features

  • Pure javascript implementation of a Spark like engine
  • Multiple data sources: filesystems, databases, cloud (S3, azure)
  • Multiple data formats: CSV, JSON, Columnar (Parquet)...
  • 50 high level operators to build parallel apps
  • Machine learning: scalable classification, regression, clusterization
  • Run interactively in a nodeJS REPL shell
  • Docker ready, simple local mode or full distributed mode
  • Very fast, see benchmark

Quickstart

bash
npm install skale

Word count example:

javascript
var sc = require('skale').context();

sc.textFile('/my/path/*.txt')
  .flatMap(line => line.split(' '))
  .map(word => [word, 1])
  .reduceByKey((a, b) => a + b, 0)
  .count(function (err, result) {
    console.log(result);
    sc.end();
  });

Local mode

In local mode, worker processes are automatically forked and communicate with app through child process IPC channel. This is the simplest way to operate, and it allows to use all machine available cores.

To run in local mode, just execute your app script:

bash
node my_app.js

or with debug traces:

bash
SKALE_DEBUG=2 node my_app.js

Distributed mode

In distributed mode, a cluster server process and worker processes must be started prior to start app. Processes communicate with each other via raw TCP or via websockets.

To run in distributed cluster mode, first start a cluster server on server_host:

bash
./bin/server.js

On each worker host, start a worker controller process which connects to server:

bash
./bin/worker.js -H server_host

Then run your app, setting the cluster server host in environment:

bash
SKALE_HOST=server_host node my_app.js

The same with debug traces:

bash
SKALE_HOST=server_host SKALE_DEBUG=2 node my_app.js

Resources

  • Contributing guide
  • Documentation
  • Gitter for support and discussion
  • Mailing list for discussion about use and development

Authors

The original authors of skale are Cedric Artigue and Marc Vertes.

List of all contributors

License

Apache-2.0

Credits

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

JavaScriptaws-s3azure-storageclustermachine-learning

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言