WizardLM

## WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions

Home Page

HF Repo • Twitter • [WizardLM] @ICLR2024 • [WizardCoder] @ICLR2024 • [WizardMath]

Join our Discord

**Unofficial Video Introductions** Thanks to the enthusiastic friends, their video introductions are more lively and interesting. 1. [NEW WizardLM 70b Giant Model...Insane Performance](https://www.youtube.com/watch?v=WdpiIXrO4_o) 2. [GET WizardLM NOW! 7B LLM KING That Can Beat ChatGPT! I'm IMPRESSED!](https://www.youtube.com/watch?v=SaJ8wyKMBds) 3. [WizardLM: Enhancing Large Language Models to Follow Complex Instructions](https://www.youtube.com/watch?v=I6sER-qivYk) 4. [WizardCoder AI Is The NEW ChatGPT's Coding TWIN!](https://www.youtube.com/watch?v=XjsyHrmd3Xo) ## News - [2024/01/04] We released **WizardCoder-33B-V1.1** trained from deepseek-coder-33b-base, the **SOTA OSS Code LLM** on [EvalPlus Leaderboard](https://evalplus.github.io/leaderboard.html), achieves **79.9 pass@1** on HumanEval, **73.2 pass@1** on HumanEval-Plus, **78.9 pass@1** on MBPP, and **66.9 pass@1** on MBPP-Plus. **WizardCoder-33B-V1.1** outperforms **ChatGPT 3.5**, **Gemini Pro**, and **DeepSeek-Coder-33B-instruct** on HumanEval and HumanEval-Plus pass@1. **WizardCoder-33B-V1.1** is comparable with **ChatGPT 3.5**, and surpasses **Gemini Pro** on MBPP and MBPP-Plus pass@1. - [2023/08/26] We released **WizardCoder-Python-34B-V1.0** , which achieves the **73.2 pass@1** and surpasses **GPT4 (2023/03/15)**, **ChatGPT-3.5**, and **Claude2** on the [HumanEval Benchmarks](https://github.com/openai/human-eval). For more details, please refer to [WizardCoder](https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder). - [2023/06/16] We released **WizardCoder-15B-V1.0** , which surpasses **Claude-Plus (+6.8)**, **Bard (+15.3)** and **InstructCodeT5+ (+22.3)** on the [HumanEval Benchmarks](https://github.com/openai/human-eval). For more details, please refer to [WizardCoder](https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder). | Model | Checkpoint | Paper | HumanEval | HumanEval+ | MBPP | MBPP+ | | ----- |------| ---- |------|-------| ----- | ----- | | GPT-4-Turbo (Nov 2023) | - | - | 85.4 | 81.7 | 83.0 | 70.7 | | GPT-4 (May 2023) | - | - | 88.4 | 76.8 | - | - | | GPT-3.5-Turbo (Nov 2023) | - | - | 72.6 | 65.9 | 81.7 | 69.4 | | Gemini Pro | - | - | 63.4 | 55.5 | 72.9 | 57.9 | | DeepSeek-Coder-33B-instruct | - | - | 78.7 | 72.6 | 78.7 | 66.7 | | WizardCoder-33B-V1.1 | HF Link | [WizardCoder] | 79.9 | 73.2 | 78.9 | 66.9 | | WizardCoder-Python-34B-V1.0 | HF Link | [WizardCoder] | 73.2 | 64.6 | 73.2 | 59.9 | | WizardCoder-15B-V1.0 | HF Link | [WizardCoder] | 59.8 | 52.4 | -- | -- | | WizardCoder-Python-13B-V1.0 | HF Link | [WizardCoder] | 64.0 | -- | -- | -- | | WizardCoder-Python-7B-V1.0 | HF Link | [WizardCoder] | 55.5 | -- | -- | -- | | WizardCoder-3B-V1.0 | HF Link | [WizardCoder] | 34.8 | -- | -- | -- | | WizardCoder-1B-V1.0 | HF Link | [WizardCoder] | 23.8 | -- | -- | -- | - [12/19/2023] We released **WizardMath-7B-V1.1** trained from Mistral-7B, the **SOTA 7B math LLM**, achieves **83.2 pass@1** on GSM8k, and **33.0 pass@1** on MATH. - [12/19/2023] **WizardMath-7B-V1.1** outperforms **ChatGPT 3.5**, **Gemini Pro**, **Mixtral MOE**, and **Claude Instant** on GSM8K pass@1. - [12/19/2023] **WizardMath-7B-V1.1** is comparable with **ChatGPT 3.5**, **Gemini Pro**, and surpasses **Mixtral MOE** on MATH pass@1. - Our **WizardMath-70B-V1.0** model slightly outperforms some closed-source LLMs on the GSM8K, including **ChatGPT 3.5**, **Claude Instant 1** and **PaLM 2 540B**. - Our **WizardMath-70B-V1.0** model achieves **81.6 pass@1** on the [GSM8k Benchmarks](https://github.com/openai/grade-school-math), which is **24.8** points higher than the SOTA open-source LLM. - Our **WizardMath-70B-V1.0** model achieves **22.7 pass@1** on the [MATH Benchmarks](https://github.com/hendrycks/math), which is **9.2** points higher than the SOTA open-source LLM. | Model | Checkpoint | Paper | GSM8k | MATH | | ----- |------| ---- |------|-------| | **WizardMath-7B-V1.1** | HF Link | [WizardMath]| **83.2** | **33.0** | | WizardMath-70B-V1.0 | HF Link | [WizardMath]| **81.6** | **22.7** | | WizardMath-13B-V1.0 | HF Link | [WizardMath]| **63.9** | **14.0** | | WizardMath-7B-V1.0 | HF Link | [WizardMath]| **54.9** | **10.7** | - [08/09/2023] We released **WizardLM-70B-V1.0** model. Here is [Full Model Weight](https://huggingface.co/WizardLM/WizardLM-70B-V1.0). | Model | Checkpoint | Paper |MT-Bench | AlpacaEval | GSM8k | HumanEval | Demo | License| | ----- |------| ---- |------|-------| ----- | ----- | ----- | ----- | | **WizardLM-70B-V1.0** | HF Link |**Coming Soon**| **7.78** | **92.91%** |**77.6%** | **50.6**| | Llama 2 License | | WizardLM-13B-V1.2 | HF Link | | 7.06 | 89.17% |55.3% | 36.6 | [Demo](http://47.103.63.15:50087/) | Llama 2 License | | WizardLM-13B-V1.1 | HF Link | | 6.76 |86.32% | | 25.0 | | Non-commercial| | WizardLM-30B-V1.0 | HF Link | | 7.01 | | | 37.8 | | Non-commercial | | WizardLM-13B-V1.0 | HF Link | | 6.35 | 75.31% | | 24.0 | | Non-commercial| | WizardLM-7B-V1.0 | HF Link | [WizardLM] | | | |19.1 | | Non-commercial| ### Citation Please cite the paper if you use the data or code from WizardLM. ``` @inproceedings{ xu2024wizardlm, title={Wizard{LM}: Empowering Large Pre-Trained Language Models to Follow Complex Instructions}, author={Can Xu and Qingfeng Sun and Kai Zheng and Xiubo Geng and Pu Zhao and Jiazhan Feng and Chongyang Tao and Qingwei Lin and Daxin Jiang}, booktitle={The Twelfth International Conference on Learning Representations}, year={2024}, url={https://openreview.net/forum?id=CfXh93NDgH} } ``` Please cite the paper if you use the data or code from WizardCoder. ``` @inproceedings{ luo2024wizardcoder, title={WizardCoder: Empowering Code Large Language Models with Evol-Instruct}, author={Ziyang Luo and Can Xu and Pu Zhao and Qingfeng Sun and Xiubo Geng and Wenxiang Hu and Chongyang Tao and Jing Ma and Qingwei Lin and Daxin Jiang}, booktitle={The Twelfth International Conference on Learning Representations}, year={2024}, url={https://openreview.net/forum?id=UnUwSIgK5W} } ``` Please cite the paper if you refer to our model or code or data or paper from WizardMath. ``` @article{luo2023wizardmath, title={Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct}, author={Luo, Haipeng and Sun, Qingfeng and Xu, Can and Zhao, Pu and Lou, Jianguang and Tao, Chongyang and Geng, Xiubo and Lin, Qingwei and Chen, Shifeng and Zhang, Dongmei}, journal={arXiv preprint arXiv:2308.09583}, year={2023} } ``` ❗To commen concern about dataset: Recently, there have been clear changes in the open-source policy and regulations of our overall organization's code, data, and models. Despite this, we have still worked hard to obtain opening the weights of the model first, but the data involves stricter auditing and is in review with our legal team . Our researchers have no authority to publicly release them without authorization. Thank you for your understanding. ## Hiring - 📣 We are looking for highly motivated students to join us as interns to create more intelligent AI together. Please contact [email protected] Note for model system prompts usage: To obtain results **identical to our demo**, please strictly follow the prompts and invocation methods provided in the **"src/infer_wizardlm13b.py"** to use our model for inference. Our model adopts the prompt format from Vicuna and supports **multi-turn** conversation. For WizardLM, the Prompt should be as following: ``` A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Hi ASSISTANT: Hello.USER: Who are you? ASSISTANT: I am WizardLM....... ``` For WizardCoder , the Prompt should be as following: ``` "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:" ``` For WizardMath</