magika

开源

快速而精确的 AI 驱动文件内容类型检测

17.9K stars0 点赞0 次浏览

工具介绍

快速而精确的 AI 驱动文件内容类型检测

Magika Magika是一款由AI驱动的新型文件类型检测工具,它依靠深度学习的最新进步来提供准确的检测. 在引擎盖下,Magika采用了一种高度优化的自定义模式,它只重了大约几分MB,并且能够在毫秒内精确地识别出文件,即使在运行在单个CPU上. Magika在包含200多类内容的100M样本数据集(既包括二进制文件格式,也包括文本文件格式)上接受了培训和评价,并在我们的测试集中实现了平均99%的精度. 以下是Magika命令行输出的缩写:

核心特点

•Trained and evaluated on a dataset of ~100M files across 200+ content types.
•On our test set, Magika achieves ~99% average precision and recall, outperforming existing approaches -- especially on textual content types.
•After the model is loaded (which is a one-off overhead), the inference time is about 5ms per file, even when run on a single CPU.
•You can invoke Magika with even thousands of files at the same time. You can also use -r for recursively scanning a directory.
•Near-constant inference time, independently from the file size; Magika only uses a limited subset of the file's content.
•The tolerance to errors can be controlled via different prediction modes, such as high-confidence, medium-confidence, and best-guess.
•The client and the bindings are already open source, and more is coming soon!
•Core Concepts
•How Magika works
•Models & content types