百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
C

charabia

> 编程语言
开源

Meilisearch 使用的库,用于对查询和文档进行标记化

349 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

Meilisearch 使用的库,用于对查询和文档进行标记化

Charabia

Library used by Meilisearch to tokenize queries and documents

Role

The tokenizer’s role is to take a sentence or phrase and split it into smaller units of language, called tokens. It finds and retrieves all the words in a string based on the language’s particularities.

Details

Charabia provides a simple API to segment, normalize, or tokenize (segment + normalize) a text of a specific language by detecting its Script/Language and choosing the specialized pipeline for it.

Supported languages

Charabia is multilingual, featuring optimized support for:

Script / Language specialized segmentation specialized normalization Segmentation Performance level Tokenization Performance level
Latin ✅ CamelCase segmentation ✅ compatibility decomposition + lowercase + nonspacing-marks removal + Ð vs Đ spoofing normalization + ı normalization ~23MiB/sec ~9MiB/sec
Latin - German ✅ CamelCase segmentation + German word segmentation ✅ compatibility decomposition + lowercase + nonspacing-marks removal + Ð vs Đ spoofing normalization + ı normalization ~23MiB/sec ~9MiB/sec
Greek ❌ ✅ compatibility decomposition + lowercase + final sigma normalization ~27MiB/sec ~8MiB/sec
Cyrillic - Georgian ❌ ✅ compatibility decomposition + lowercase ~27MiB/sec ~9MiB/sec
Chinese CMN ✅ jieba ✅ compatibility decomposition + kvariant conversion ~10MiB/sec ~5MiB/sec
Hebrew ❌ ✅ compatibility decomposition + nonspacing-marks removal ~33MiB/sec ~11MiB/sec
Arabic ✅ ال segmentation ✅ compatibility decomposition + nonspacing-marks removal + [Tatweel, Alef, Yeh, and Taa Marbuta normalization] + Persian digits to ASCII conversion (۰-۹ → 0-9) ~36MiB/sec ~11MiB/sec
Japanese ✅ lindera IPA-dict ❌ compatibility decomposition ~3MiB/sec ~3MiB/sec
Korean ✅ lindera KO-dict ❌ compatibility decomposition ~2MiB/sec ~2MiB/sec
Thai ✅ dictionary based ✅ compatibility decomposition + nonspacing-marks removal ~22MiB/sec ~11MiB/sec
Khmer ✅ dictionary based ✅ compatibility decomposition ~7MiB/sec ~5MiB/sec

We aim to provide global language support, and your feedback helps us move closer to that goal. If you notice inconsistencies in your search results or the way your documents are processed, please open an issue on our GitHub repository.

If you have a particular need that charabia does not support, please share it in the product repository by creating a dedicated discussion.

About Performance level

Performances are based on the throughput (MiB/sec) of the tokenizer (computed on a scaleway Elastic Metal server EM-A410X-SSD - CPU: Intel Xeon E5 1650 - RAM: 64 Go) using jemalloc:

  • 0️⃣⬛️: 0 -> 1 MiB/sec
  • 1️⃣: 1 -> 3 MiB/sec
  • 2️⃣: 3 -> 8 MiB/sec
  • 3️⃣: 8 -> 20 MiB/sec
  • 4️⃣: 20 -> 50 MiB/sec
  • 5️⃣: 50 MiB/sec or more

Examples

Tokenization

rust
use charabia::Tokenize;

let orig = "Thé quick (\"brown\") fox can't jump 32.3 feet, right? Brr, it's 29.3°F!";

// tokenize the text.
let mut tokens = orig.tokenize();

let token = tokens.next().unwrap();
// the lemma into the token is normalized: `Thé` became `the`.
assert_eq!(token.lemma(), "the");
// token is classfied as a word
assert!(token.is_word());

let token = tokens.next().unwrap();
assert_eq!(token.lemma(), " ");
// token is classfied as a separator
assert!(token.is_separator());

Segmentation

rust
use charabia::Segment;

let orig = "The quick (\"brown\") fox can't jump 32.3 feet, right? Brr, it's 29.3°F!";

// segment the text.
let mut segments = orig.segment_str();

assert_eq!(segments.next(), Some("The"));
assert_eq!(segments.next(), Some(" "));
assert_eq!(segments.next(), Some("quick"));

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Rust

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言