百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
T

ttok

> 编程语言
开源

根据标记计算和截断文本

399 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

根据标记计算和截断文本

ttok

Count and truncate text based on tokens

Background

Large language models such as GPT-3.5 and GPT-4 work in terms of tokens.

This tool can count tokens, using OpenAI's tiktoken library.

It can also truncate text to a specified number of tokens.

See llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs for more on this project.

Installation

Install this tool using pip:

bash
pip install ttok

Or using Homebrew:

bash
brew install simonw/llm/ttok

Counting tokens

Provide text as arguments to this tool to count tokens:

bash
ttok Hello world
2

You can also pipe text into the tool:

bash
echo -n "Hello world" | ttok
2

Here the echo -n option prevents echo from adding a newline - without that you would get a token count of 3.

To pipe in text and then append extra tokens from arguments, use the -i - option:

bash
echo -n "Hello world" | ttok more text -i -
6

Different models

By default, the tokenizer model for GPT-3.5 and GPT-4 is used.

To use the model for GPT-2 and GPT-3, add --model gpt2:

bash
ttok boo Hello there this is -m gpt2
6

Compared to GPT-3.5:

bash
ttok boo Hello there this is
5

Further model options are documented here.

Truncating text

Use the -t 10 or --truncate 10 option to truncate text to a specified number of tokens:

bash
ttok This is too many tokens -t 3
This is too

Viewing tokens

The --encode option can be used to view the integer token IDs for the incoming text:

bash
ttok Hello world --encode
9906 1917

The --decode method reverses this process:

bash
ttok 9906 1917 --decode
Hello world

Add --tokens to either of these options to see a detailed breakdown of the tokens:

bash
ttok Hello world --encode --tokens
[b'Hello', b' world']

Available models

This is the full list of available models and their corresponding encodings. Model names and encoding names are valid for the -m/--model option.

  • gpt-4 (cl100k_base)
  • gpt-3.5-turbo (cl100k_base)
  • gpt-3.5 (cl100k_base)
  • gpt-35-turbo (cl100k_base)
  • davinci-002 (cl100k_base)
  • babbage-002 (cl100k_base)
  • text-embedding-ada-002 (cl100k_base)
  • text-embedding-3-small (cl100k_base)
  • text-embedding-3-large (cl100k_base)
  • text-davinci-003 (p50k_base)
  • text-davinci-002 (p50k_base)
  • text-davinci-001 (r50k_base)
  • text-curie-001 (r50k_base)
  • text-babbage-001 (r50k_base)
  • text-ada-001 (r50k_base)
  • davinci (r50k_base)
  • curie (r50k_base)
  • babbage (r50k_base)
  • ada (r50k_base)
  • code-davinci-002 (p50k_base)
  • code-davinci-001 (p50k_base)
  • code-cushman-002 (p50k_base)
  • code-cushman-001 (p50k_base)
  • davinci-codex (p50k_base)
  • cushman-codex (p50k_base)
  • text-davinci-edit-001 (p50k_edit)
  • code-davinci-edit-001 (p50k_edit)
  • text-similarity-davinci-001 (r50k_base)
  • text-similarity-curie-001 (r50k_base)
  • text-similarity-babbage-001 (r50k_base)
  • text-similarity-ada-001 (r50k_base)
  • text-search-davinci-doc-001 (r50k_base)
  • text-search-curie-doc-001 (r50k_base)
  • text-search-babbage-doc-001 (r50k_base)
  • text-search-ada-doc-001 (r50k_base)
  • code-search-babbage-code-001 (r50k_base)
  • code-search-ada-code-001 (r50k_base)
  • gpt2 (gpt2)
  • gpt-2 (gpt2)

ttok --help

…

You can also run this command using:

bash
python -m ttok --help

Development

To contribute to this tool, first checkout the code. Then create a new virtual environment:

bash
cd ttok
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

bash
pip install -e '.[test]'

To run the tests:

bash
pytest

Issues· 17 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言