Unofficial bindings / ports in other languages

The following projects are not maintained by OpenAI. I cannot vouch that any of them are correct or safe to use. Use at your own risk.

Note that if a tokeniser fails to exactly match tiktoken's behaviour, you may get worse results when sampling from models, with no warning.

Javascript

• https://github.com/dqbd/tiktoken
• https://github.com/ceifa/tiktoken-node
• https://github.com/niieani/gpt-tokenizer
• The gpt-3-encoder package will work for most GPT-3 models. However, it will often appear to work for Codex or GPT-3.5 while actually being out of distribution, and will not at all work for GPT-4 or embeddings models.

Rust

• https://github.com/zurawiki/tiktoken-rs

Java

• https://github.com/eisber/tiktoken
• https://github.com/knuddelsgmbh/jtokkit

Ruby

• https://github.com/volition-co/tiktoken

C#

• https://github.com/dmitry-brazhenko/SharpToken

Go

• https://github.com/tiktoken-go/tokenizer
• https://github.com/pkoukk/tiktoken-go

PHP

• https://github.com/danny50610/bpe-tokeniser

Kotlin

• https://github.com/aallam/ktoken

Thanks to everyone for building useful things!

I'm happy to link to other projects in this comment.