百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
C

chunk

> 编程语言
开源

西部最快的切分器 最高可达 1TB/s 的"语义"切分,快速、简单!

355 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

西部最快的切分器 最高可达 1TB/s 的"语义"切分,快速、简单!

chunk

the fastest text chunking library — up to 1 TB/s throughput


you know how every chunking library claims to be fast? yeah, we actually meant it.

chunk splits text at semantic boundaries (periods, newlines, the usual suspects) and does it stupid fast. we're talking "chunk the entire english wikipedia in 120ms" fast.

want to know how? read the blog post where we nerd out about SIMD instructions and lookup tables.

[!NOTE] chunk was previously known as memchunk. It still contains the chunk method but now also includes other basic operations that power chonkie core.

See benches/ for detailed benchmarks.

Installation

bash
cargo add chunk

looking for python or javascript?

Usage

…

Citation

If you use chunk in your research, please cite it as follows:

bibtex
@software{chunk2025,
  author = {Minhas, Bhavnick},
  title = {chunk: The fastest text chunking library},
  year = {2025},
  publisher = {GitHub},
  howpublished = {\\url{https://github.com/chonkie-inc/chunk}},
}

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Rust

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言