百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
T

textvec

> 编程语言
开源

文本向量化工具在分类任务中优于 TFIDF

198 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

文本向量化工具在分类任务中优于 TFIDF

WHAT: Supervised text vectorization tool

Textvec is a text vectorization tool, with the aim to implement all the "classic" text vectorization NLP methods in Python. The main idea of this project is to show alternatives for an excellent TFIDF method which is highly overused for supervised tasks. All interfaces are similar to scikit-learn so you should be able to test the performance of this supervised methods just with a few changes.

Textvec is compatible with: Python 3.9+.


WHY: Comparison with TFIDF

As you can read in the different articles1,2 almost on every dataset supervised methods outperform unsupervised. But most text classification examples on the internet ignores that fact.

IMDB_bin RT_bin Airlines Sentiment_bin Airlines Sentiment_multiclass 20news_multiclass
TF 0.8984 0.7571 0.9194 0.8084 0.8206
TFIDF 0.9052 0.7717 0.9259 0.8118 0.8575
TFPF 0.8813 0.7403 0.9212 NA NA
TFRF 0.8797 0.7412 0.9194 NA NA
TFICF 0.8984 0.7642 0.9199 0.8125 0.8292
TFBINICF 0.8984 0.7571 0.9194 NA NA
TFCHI2 0.8898 0.7398 0.9108 NA NA
TFGR 0.8850 0.7065 0.8956 NA NA
TFRRF 0.8879 0.7506 0.9194 NA NA
TFOR 0.9092 0.7806 0.9207 NA NA

Here is a comparison for binary classification on imdb sentiment data set. Labels sorted by accuracy score and the heatmap shows the correlation between different approaches. As you can see some methods are good for to ensemble models or perform features selection.

For more dataset benchmarks (rotten tomatoes, airline sentiment) see Binary classification quality comparison


Install:

Usage:

uv pip install textvec

Source code:

bash
git clone https://github.com/textvec/textvec
cd textvec
uv pip install -e .

HOW: Examples

The usage is similar to scikit-learn:

python
from sklearn.feature_extraction.text import CountVectorizer
from textvec.vectorizers import TfBinIcfVectorizer

cvec = CountVectorizer().fit(train_data.text)

tficf_vec = TfBinIcfVectorizer(sublinear_tf=True)
tficf_vec.fit(cvec.transform(text), y)

For more detailed examples see Basic example and other notebooks in Examples

Currently implemented methods:

  • TfIcfVectorizer
  • TforVectorizer
  • TfgrVectorizer
  • TfigVectorizer
  • Tfchi2Vectorizer
  • TfrfVectorizer
  • TfrrfVectorizer
  • TfBinIcfVectorizer
  • TfpfVectorizer
  • SifVectorizer
  • TfbnsVectorizer

Most of the vectorization techniques you can find in articles1,2,3. If you see any method with wrong name or reference please commit!


TODO

  • Docs

REFERENCE

  • [1] [Deqing Wang and Hui Zhang] Inverse-Category-Frequency based Supervised Term Weighting Schemes for Text Categorization
  • [2] [M. Lan, C. L. Tan, J. Su, and Y. Lu] Supervised and traditional term weighting methods for automatic text categorization
  • [3] [Sanjeev Arora, Yingyu Liang and Tengyu Ma] A Simple But Tough-To-Beat Baseline For Sentence Embeddings
  • [4] Thanks aysent for an inspiration

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Pythonmachine-learningnatural-language-processingnlppython

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言