加速在 Intel XPU 上进行的本地 LLM 推理和微调 (LLaMA、Mistral、ChatGLM、Qwen、DeepSeek、Mixtral、Gemma、Phi、MiniCPM、Qwen-VL、MiniCPM-V 等)。
加速在 Intel XPU 上进行的本地 LLM 推理和微调 (LLaMA、Mistral、ChatGLM、Qwen、DeepSeek、Mixtral、Gemma、Phi、MiniCPM、Qwen-VL、MiniCPM-V 等)。
Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates.
Patches to this project are no longer accepted by Intel.
This project has been identified as having known security issues.
< English | 中文 >
IPEX-LLM is an LLM acceleration library for Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max), NPU and CPU [^1].
[!NOTE]
IPEX-LLMprovides seamless integration with llama.cpp, Ollama, vLLM, HuggingFace transformers, LangChain, LlamaIndex, Text-Generation-WebUI, DeepSpeed-AutoTP, FastChat, Axolotl, HuggingFace PEFT, HuggingFace TRL, AutoGen, ModeScope, etc.- 70+ models have been optimized/verified on
ipex-llm(e.g., Llama, Phi, Mistral, Mixtral, DeepSeek, Qwen, ChatGLM, MiniCPM, Qwen-VL, MiniCPM-V and more), with state-of-art LLM optimizations, XPU acceleration and low-bit (FP8/FP6/FP4/INT4) support; see the complete list here.
ipex-llm.ipex-llm 2.2.0, which includes Ollama Portable Zip and llama.cpp Portable Zip.⚠️ Warning (for llama.cpp Portable Zip)
mmap-based model loading in llama.cpp may leak data via side-channels in multi-tenant or shared-host environments.
To disablemmap, add:--no-mmap
ipex-llm on Intel Arc B580 GPU.ipex-llm on Intel GPU.ipex-llm now supports Axolotl for LLM finetuning on Intel GPU; see the quickstart here. ipex-llm inference, serving and finetuning using the Docker images.ipex-llm on Windows using just "one command".ipex-llm; see the quickstart here.llama.cpp and ollama with ipex-llm; see the quickstart here.ipex-llm now supports Llama 3 on both Intel GPU and CPU.ipex-llm now provides C++ interface, which can be used as an accelerated backend for running llama.cpp and ollama on Intel GPU.bigdl-llm has now become ipex-llm (see the migration guide here); you may find the original BigDL project here.ipex-llm now supports directly loading model from ModelScope (魔搭).ipex-llm added initial INT2 support (based on llama.cpp IQ2 mechanism), which makes it possible to run large-sized LLM (e.g., Mixtral-8x7B) on Intel GPU with 16GB VRAM.ipex-llm through Text-Generation-WebUI GUI.ipex-llm now supports Self-Speculative Decoding, which in practice brings ~30% speedup for FP16 and BF16 inference latency on Intel GPU and CPU respectively.ipex-llm now supports a comprehensive list of LLM finetuning on Intel GPU (including LoRA, QLoRA, DPO, QA-LoRA and ReLoRA).ipex-llm QLoRA, we managed to finetune LLaMA2-7B in 21 minutes and LLaMA2-70B in 3.14 hours on 8 Intel Max 1550 GPU for Standford-Alpaca (see the blog here). ipex-llm now supports ReLoRA (see "ReLoRA: High-Rank Training Through Low-Rank Updates").ipex-llm now supports Mixtral-8x7B on both Intel GPU and CPU. ipex-llm now supports QA-LoRA (see "QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models"). ipex-llm now supports FP8 and FP4 inference on Intel GPU.ipex-llm is available.ipex-llm now supports vLLM continuous batching on both Intel GPU and CPU.ipex-llm now supports QLoRA finetuning on both Intel GPU and CPU.ipex-llm now supports FastChat serving on on both Intel CPU and GPU.ipex-llm now supports Intel GPU (including iGPU, Arc, Flex and MAX).ipex-llm tutorial is released.ipex-llm DemoSee demos of running local LLMs on Intel Core Ultra iGPU, Intel Core Ultra NPU, single-card Arc GPU, or multi-card Arc GPUs using ipex-llm below.
暂无开放 Issues，或尚未同步最近议题。