Hazm - Persian NLP Toolkit
Hazm is a python library to perform natural language processing tasks on Persian text. It offers various features for analyzing, processing, and understanding Persian text. You can use Hazm to normalize text, tokenize sentences and words, lemmatize words, assign part-of-speech tags, identify dependency relations, create word and sentence embeddings, or read popular Persian corpora.
Features
- Normalization: Converts text to a standard form (diacritics removal, ZWNJ correction, etc).
- Tokenization: Splits text into sentences and words.
- Lemmatization: Reduces words to their base forms.
- POS tagging: Assigns a part of speech to each word.
- Dependency parsing: Identifies the syntactic relations between words.
- Embedding: Creates vector representations of words and sentences.
- Hugging Face Integration: Automatically download and cache pretrained models from the Hub.
- Persian corpora reading: Easily read popular Persian corpora with ready-made scripts.
Installation
To install the latest version of Hazm (requires Python 3.12+), run:
pip install hazm
To use the pretrained models from Hugging Face, ensure you have the huggingface-hub package:
pip install huggingface-hub
Pretrained-Models
Hazm supports automatic downloading of pretrained models. You can find all available models (POS Tagger, Chunker, Embeddings, etc.) on our official Hugging Face page:
Roshan Research on Hugging Face
When using Hazm, simply provide the repo_id and model_filename as shown in the examples below, and the library will handle the rest.
Usage
…
Documentation
Visit https://roshan-ai.ir/hazm to view the full documentation.
Evaluation
Module name
DependencyParser
85.6%
POSTagger
98.8%
Chunker
93.4%
Lemmatizer
89.9%
Metric
Value
SpacyPOSTagger
Precision
0.99250
Recall
0.99249
F1-Score
0.99249
EZ Detection in SpacyPOSTagger
Precision
0.99301
Recall
0.99297
F1-Score
0.99298
SpacyChunker
Accuracy
96.53%
F-Measure
95.00%
Recall
95.17%
Precision
94.83%
SpacyDependencyParser
TOK Accuracy
99.06
UAS
92.30
LAS
89.15
SENT Precision
98.84
SENT Recall
99.38
SENT F-Measure
99.11