tiktoken · Issues· 131 open

Unofficial bindings / ports in other languages

High Token Inflation and Lack of Representation for Bangladeshi Bengali

Tiktoken educational BPE trainer takes long time to train with vocab size 30k

[Security] Reachable Rust panic (denial of service) when tokenizing with an untrusted encoder missing single-byte tokens

Please commit your Cargo.lock file in the repository

tiktoken under counting tokens for openai text-embedding-3-large

tiktoken 0.12 cannot handle qwen3.5 in huggingface

CoreBPE::new has an unconstrained NSE type parameter, making normal downstream calls fail type inference

Extremely Long Text results in PanicException, which is hard to catch in python code

Very slow for inputs like "a" * 100000

Panic (stack overflow) when encoding a certain string

[Bug] tiktoken 0.9.0 ignores TIKTOKEN_OFFLINE and TIKTOKEN_CACHE_DIR, still attempts network download

encode_ordinary_batch — reproducible multi-second tail stalls on 32-core box (o200k_base, num_threads=8)

Use Trusted Publishing to upload to PyPI

Token count discrepancy between tiktoken and API response when messages contain tool calls