Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
L

llamacpp-python

> 编程语言
Open source

Python bindings for llama.cpp

199 stars0 likes1 views
WebsiteGitHub

About

Python bindings for llama.cpp

Python bindings for llama.cpp

Important

  • The Python API has changed significantly in the recent weeks and as a result, I have not had a chance to update cli.py or chat.py to reflect the new changes. The scripts under examples/simple.py and examples/simple_low_level.py should give you an idea of how to use the library.

Install

From PyPI

bash
pip install llamacpp

Build from Source

bash
pip install .

Get the model weights

You will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.g https://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.

ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model

Convert the weights to GGML format using llamacpp-convert. Then use llamacpp-quantize to quantize them into INT4. For example, for the 7B parameter model, run

llamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cli

Note that running llamacpp-convert requires torch, sentencepiece and numpy to be installed. These packages are not installed by default when your install llamacpp.

Command line interface

The package installs the command line entry point llamacpp-cli that points to llamacpp/cli.py and should provide about the same functionality as the main program in the original C++ repository. There is also an experimental llamacpp-chat that is supposed to bring up a chat interface but this is not working correctly yet.

API

Documentation is TBD. But the long and short of it is that there are two interfaces

  • LlamaInference - this one is a high level interface that tries to take care of most things for you. The demo script below uses this.
  • LlamaContext - this is a low level interface to the underlying llama.cpp API. You can use this similar to how the main example in llama.cpp does uses the C API. This is a rough implementation and currently untested except for compiling successfully.

Demo script

See llamacpp/cli.py for a detailed example. The simplest demo would be something like the following:

…

ToDo

  • Investigate using dynamic versions using setuptools-scm (Example: https://github.com/pypa/setuptools_scm/blob/main/scm_hack_build_backend.py)

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

C++

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 18, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言