jschardet

> 编程语言

Open source

Character encoding auto-detection in JavaScript (port of python's chardet)

740 stars0 likes2 views

Website GitHub

About

Character encoding auto-detection in JavaScript (port of python's chardet)

jschardet

jschardet is a character encoding detector for JavaScript. Runs in Node.js and browsers with zero runtime dependencies.

jschardet 4 is a ground-up TypeScript port of chardet 7. It's much faster and more accurate than jschardet 3, and a drop-in replacement for its documented API.

The API is detect() and detectAll(), returning encoding, confidence, language, and mimeType.

Features

99.4% accuracy on 3,138 test files, up from 43.1% in jschardet 3, with ~6× the throughput and ~10× lower peak memory. Language detection for every result. MIME type detection for binary files.

	jschardet 4.0.0	jschardet 3.1.4	chardet 7.6.0 (Python)
Accuracy (3,138 files)	99.4%	43.1%	99.5%
Speed	780 files/s	126 files/s	222 files/s
Language detection	93.4%	—	93.4%
Peak memory	85.5 MiB	831.8 MiB	37.6 MiB
Bundle size (min / gzip)	1,141 / 691 KiB	334 / 119 KiB	—
Cold start (import + first detect)	109.2 ms	29.9 ms	101.7 ms
Runs in browsers	yes	yes	—
MIME type detection	yes	no	yes
License	0BSD	LGPL	0BSD

Compared to jschardet 3, v4 has a larger bundle and a ~110 ms first-call cost. Both come from shipping a larger detection model; the model decompresses once on first detect() and stays in memory afterwards, so subsequent calls run at full speed.

See docs/performance.md for the full benchmark methodology and per-encoding accuracy.

Installation

npm

bash

npm install jschardet

Both module systems are supported:

javascript

import { detect } from 'jschardet';         // ESM
const { detect } = require('jschardet');    // CommonJS

Browser

Copy and include jschardet.min.js in your page (attaches a global jschardet). For ESM, use jschardet.esm.min.js instead. Unminified builds and source maps are in dist/.

The library is also available via jsDelivr:

Format	URL
IIFE (`


**ESM** (after copying `jschardet.esm.min.js`):

```html

Quick start

javascript

import { detect, detectAll } from 'jschardet';

// string — ASCII
detect("Python is a great programming language for beginners and experts alike.")
// { encoding: 'ascii', confidence: 1, language: 'en', mimeType: 'text/plain' }

// Uint8Array — "The naïve approach doesn't always work in complex systems." in UTF-8
detect(new TextEncoder().encode("The naïve approach doesn't always work in complex systems."))
// { encoding: 'utf-8', confidence: 0.84, language: 'en', mimeType: 'text/plain' }

// Uint8Array — "日本語の文字コード検出テストです。" in EUC-JP
detect(new Uint8Array([
  0xc6, 0xfc, 0xcb, 0xdc, 0xb8, 0xec, 0xa4, 0xce, 0xca, 0xb8,
  0xbb, 0xfa, 0xa5, 0xb3, 0xa1, 0xbc, 0xa5, 0xc9, 0xb8, 0xa1,
  0xbd, 0xd0, 0xa5, 0xc6, 0xa5, 0xb9, 0xa5, 0xc8, 0xa4, 0xc7,
  0xa4, 0xb9, 0xa1, 0xa3,
]))
// { encoding: 'EUC-JP', confidence: 0.56, language: 'ja', mimeType: 'text/plain' }

// Buffer (Node.js) — "Le café est une boisson très populaire en France et dans le monde entier." in windows-1252
const results = detectAll(Buffer.from([
   76, 101,  32,  99,  97, 102, 233,  32, 101, 115, 116,  32, 117, 110, 101,
   32,  98, 111, 105, 115, 115, 111, 110,  32, 116, 114, 232, 115,  32, 112,
  111, 112, 117, 108,  97, 105, 114, 101,  32, 101, 110,  32,  70, 114,  97,
  110,  99, 101,  32, 101, 116,  32, 100,  97, 110, 115,  32, 108, 101,  32,
  109, 111, 110, 100, 101,  32, 101, 110, 116, 105, 101, 114,  46,
]))
for (const r of results.slice(0, 4)) {
  console.log(r.encoding, r.confidence.toFixed(2));
}
// Windows-1252 0.32
// iso8859-15 0.32
// ISO-8859-1 0.32
// MacRoman 0.31

…

ts
interface IOptionsMap {
  minimumThreshold?: number;      // override default 0.20 for detectAll filtering
  detectEncodings?: string[];     // allowlist of encoding names to consider
  excludeEncodings?: string[];    // blocklist of encoding names
}

`enableDebug()`

Logs full candidate lists to the console from detect / detectAll (useful when tuning thresholds or allowlists).

CLI

bash

jschardet somefile.txt
# somefile.txt: utf-8 with confidence 1

jschardet --minimal somefile.txt
# utf-8

# Include detected language
jschardet -l somefile.txt
# somefile.txt: utf-8 en (English) with confidence 1

# Only consider specific encodings
jschardet -i utf-8,windows-1252 somefile.txt
# somefile.txt: utf-8 with confidence 1

# Pipe from stdin
cat somefile.txt | jschardet
# stdin: utf-8 with confidence 1

…

js
import { chardet } from 'jschardet';
import { createReadStream } from 'node:fs';

const detector = new chardet.UniversalDetector();
for await (const chunk of createReadStream('unknown.txt')) {
  detector.feed(chunk);
  if (detector.done) break;
}
console.log(detector.close());
// { encoding: 'utf-8', confidence: 1, language: 'en', mimeType: 'text/plain' }