Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
T

ttok

> 编程语言
Open source

Count and truncate text based on tokens

399 stars0 likes0 views
WebsiteGitHub

About

Count and truncate text based on tokens

ttok

Count and truncate text based on tokens

Background

Large language models such as GPT-3.5 and GPT-4 work in terms of tokens.

This tool can count tokens, using OpenAI's tiktoken library.

It can also truncate text to a specified number of tokens.

See llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs for more on this project.

Installation

Install this tool using pip:

bash
pip install ttok

Or using Homebrew:

bash
brew install simonw/llm/ttok

Counting tokens

Provide text as arguments to this tool to count tokens:

bash
ttok Hello world
2

You can also pipe text into the tool:

bash
echo -n "Hello world" | ttok
2

Here the echo -n option prevents echo from adding a newline - without that you would get a token count of 3.

To pipe in text and then append extra tokens from arguments, use the -i - option:

bash
echo -n "Hello world" | ttok more text -i -
6

Different models

By default, the tokenizer model for GPT-3.5 and GPT-4 is used.

To use the model for GPT-2 and GPT-3, add --model gpt2:

bash
ttok boo Hello there this is -m gpt2
6

Compared to GPT-3.5:

bash
ttok boo Hello there this is
5

Further model options are documented here.

Truncating text

Use the -t 10 or --truncate 10 option to truncate text to a specified number of tokens:

bash
ttok This is too many tokens -t 3
This is too

Viewing tokens

The --encode option can be used to view the integer token IDs for the incoming text:

bash
ttok Hello world --encode
9906 1917

The --decode method reverses this process:

bash
ttok 9906 1917 --decode
Hello world

Add --tokens to either of these options to see a detailed breakdown of the tokens:

bash
ttok Hello world --encode --tokens
[b'Hello', b' world']

Available models

This is the full list of available models and their corresponding encodings. Model names and encoding names are valid for the -m/--model option.

  • gpt-4 (cl100k_base)
  • gpt-3.5-turbo (cl100k_base)
  • gpt-3.5 (cl100k_base)
  • gpt-35-turbo (cl100k_base)
  • davinci-002 (cl100k_base)
  • babbage-002 (cl100k_base)
  • text-embedding-ada-002 (cl100k_base)
  • text-embedding-3-small (cl100k_base)
  • text-embedding-3-large (cl100k_base)
  • text-davinci-003 (p50k_base)
  • text-davinci-002 (p50k_base)
  • text-davinci-001 (r50k_base)
  • text-curie-001 (r50k_base)
  • text-babbage-001 (r50k_base)
  • text-ada-001 (r50k_base)
  • davinci (r50k_base)
  • curie (r50k_base)
  • babbage (r50k_base)
  • ada (r50k_base)
  • code-davinci-002 (p50k_base)
  • code-davinci-001 (p50k_base)
  • code-cushman-002 (p50k_base)
  • code-cushman-001 (p50k_base)
  • davinci-codex (p50k_base)
  • cushman-codex (p50k_base)
  • text-davinci-edit-001 (p50k_edit)
  • code-davinci-edit-001 (p50k_edit)
  • text-similarity-davinci-001 (r50k_base)
  • text-similarity-curie-001 (r50k_base)
  • text-similarity-babbage-001 (r50k_base)
  • text-similarity-ada-001 (r50k_base)
  • text-search-davinci-doc-001 (r50k_base)
  • text-search-curie-doc-001 (r50k_base)
  • text-search-babbage-doc-001 (r50k_base)
  • text-search-ada-doc-001 (r50k_base)
  • code-search-babbage-code-001 (r50k_base)
  • code-search-ada-code-001 (r50k_base)
  • gpt2 (gpt2)
  • gpt-2 (gpt2)

ttok --help

…

You can also run this command using:

bash
python -m ttok --help

Development

To contribute to this tool, first checkout the code. Then create a new virtual environment:

bash
cd ttok
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

bash
pip install -e '.[test]'

To run the tests:

bash
pytest

Issues· 17 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Python

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言