全能型 RWKV 运行时框架,包含嵌入式、RAG、AI 代理等。
AI00 RWKV Server is an inference API server for the RWKV language model based upon the web-rwkv inference engine.
It supports Vulkan parallel and concurrent batched inference and can run on all GPUs that support Vulkan. No need for Nvidia cards!!! AMD cards and even integrated graphics can be accelerated!!!
No need for bulky pytorch, CUDA and other runtime environments, it's compact and ready to use out of the box!
Compatible with OpenAI's ChatGPT API interface.
100% open source and commercially usable, under the MIT license.
If you are looking for a fast, efficient, and easy-to-use LLM API server, then AI00 RWKV Server is your best choice. It can be used for various tasks, including chatbots, text generation, translation, and Q&A.
Join the AI00 RWKV Server community now and experience the charm of AI!
QQ Group for communication: 30920262
RWKV model, it has high performance and accuracyVulkan inference acceleration, you can enjoy GPU acceleration without the need for CUDA! Supports AMD cards, integrated graphics, and all GPUs that support Vulkanpytorch, CUDA and other runtime environments, it's compact and ready to use out of the box!Directly download the latest version from Release
After downloading the model, place the model in the assets/models/ path, for example, assets/models/RWKV-x060-World-3B-v2-20240228-ctx4096.st
Optionally modify assets/configs/Config.toml for model configurations like model path, quantization layers, etc.
Run in the command line
$ ./ai00_rwkv_server
Open the browser and visit the WebUI at http://localhost:65530 (https://localhost:65530 if tls is enabled)
Clone this repository
$ git clone https://github.com/cgisky1980/ai00_rwkv_server.git
$ cd ai00_rwkv_server
After downloading the model, place the model in the assets/models/ path, for example, assets/models/RWKV-x060-World-3B-v2-20240228-ctx4096.st
Compile
$ cargo build --release
After compilation, run
$ cargo run --release
Open the browser and visit the WebUI at http://localhost:65530 (https://localhost:65530 if tls is enabled)
It only supports Safetensors models with the .st extension now. Models saved with the .pth extension using torch need to be converted before use.
(Recommended) Run the python script convert_safetensors.py:
$ python assets/scripts/convert_safetensors.py --input /path/to/model.pth --output /path/to/model.st
Requirements: Python, with torch and safetensors installed.
If you do not want to install python, In the Release you could find an executable called converter. Run
$ ./converter --input /path/to/model.pth --output /path/to/model.st
$ cargo run --release --package converter -- --input /path/to/model.pth --output /path/to/model.st
.st model in the assets/models/ path and modify the model path in assets/configs/Config.toml--config: Configure file path (default: assets/configs/Config.toml)--ip: The IP address the server is bound to--port: Running portThe API service starts at port 65530, and the data input and output format follow the Openai API specification.
Note that some APIs like chat and completions have additional optional fields for advanced functionalities. Visit http://localhost:65530/api-docs for API schema.
/api/oai/v1/models/api/oai/models/api/oai/v1/chat/completions/api/oai/chat/completions/api/oai/v1/completions/api/oai/completions/api/oai/v1/embeddings/api/oai/embeddingsThe following is an out-of-box example of Ai00 API invocations in Python:
…
Since v0.5, Ai00 has a unique feature called BNF sampling. BNF forces the model to output in specified formats (e.g., JSON or markdown with specified fields) by limiting the possible next tokens the model can choose from.
Here is an example BNF for JSON with fields "name", "age" and "job":
start ::= json_object;
json_object ::= "{\n" object_members "\n}";
object_members ::= json_member | json_member ",\n" object_members;
json_member ::= "\t" json_key ": " json_value;
json_key ::= '"' "name" '"' | '"' "age" '"' | '"' "job" '"';
json_value ::= json_string | json_number;
json_string ::= '"'content'"';
content ::= #"\\w*";
json_number ::= positive_digit digits|'0';
digits ::= digit|digit digits;
digit ::= '0'|positive_digit;
positive_digit::="1"|"2"|"3"|"4"|"5"|"6"|"7"|"8"|"9";
text_completions and chat_completionsbatch serveint8 quantizationNF4 quantizationLoRA modelLoRA modelWe are always looking for people interested in helping us improve the project. If you are interested in any of the following, please join us!
No matter your skill level, we welcome you to join us. You can join us in the following ways:
We can't wait to work with you to make this project better! We hope the project is helpful to you!
Thank you to these awesome individuals who are insightful and outstanding for their support and selfless dedication to the project!
顾真牛
研究社交
josc146
l15y
Cahya Wirawan
yuunnn_w ⚠️
longzou ️
luoqiqi
暂无开放 Issues，或尚未同步最近议题。