The Fastest Chunker in the West Upto 1TB/s "semantic" chunking, quick and easy!
The Fastest Chunker in the West Upto 1TB/s "semantic" chunking, quick and easy!
chunk
the fastest text chunking library — up to 1 TB/s throughput
you know how every chunking library claims to be fast? yeah, we actually meant it.
chunk splits text at semantic boundaries (periods, newlines, the usual suspects) and does it stupid fast. we're talking "chunk the entire english wikipedia in 120ms" fast.
want to know how? read the blog post where we nerd out about SIMD instructions and lookup tables.
[!NOTE] chunk was previously known as memchunk. It still contains the
chunkmethod but now also includes other basic operations that power chonkie core.
See benches/ for detailed benchmarks.
cargo add chunklooking for python or javascript?
…If you use chunk in your research, please cite it as follows:
@software{chunk2025,
author = {Minhas, Bhavnick},
title = {chunk: The fastest text chunking library},
year = {2025},
publisher = {GitHub},
howpublished = {\\url{https://github.com/chonkie-inc/chunk}},
}Licensed under either of Apache License, Version 2.0 or MIT license at your option.
No open issues yet, or sync has not completed.