KnowLM

Open source

An Open-sourced Knowledgable Large Language Model Framework.

1.4K stars0 likes0 views

About

An Open-sourced Knowledgable Large Language Model Framework.

:speaking_head: [ 中文 | English ]

<br>

Knowledgeable Large Language Model Framework

KnowLM is a knowledgeable Large Language Model (LLM) framework, including data processing, model pre-training, fine-tuning, augmentation and utilization with knowledge. Additionally, KnowLM provides a model zoo featuring readily accessible models like ZhiXi and OneKE, tailored for immediate implementation.

❗Please note that this project is still undergoing optimization and developemnt, and the model weights will be regularly updated to support new features and models!
❗If you are interested in information extraction/knowledge extraction, please refer to the DeepKE. KnowLM is a framework for knowledgeable Large Language Model!

Features

A standard framework for LLM pre-training and fine-tuning.
A model zoo including ZhiXi, KnowLM-IE, OneKE, and OceanGPT, along with open-source data.
A instruction processing module based on EasyInstruct.
A knowlege augmentation module based on RAG (under development).
A hallucination detection module for based on EasyDetect.
A knowlege editing module based on EasyEdit.
Model inference and deployment.

All weights and datasets have been uploaded to HuggingFace. Click here to get started right away!

❗If you encounter any issues during the installation or use of KnowLM, please check FAQ or promptly submit an issue, and we will assist you with resolving the problem!

Category	Base	Name	Version	Download Link	Note
Base Model	LlaMA1	KnowLM-13B-Base	V1.0	HuggingFace WiseModel ModelScope	Base Model
Dialogue Model	LlaMA1	KnowLM-13B-ZhiXi	V1.0	HuggingFace WiseModel ModelScope	Information Extraction Model
Dialogue Model	LlaMA1	KnowLM-13B-IE	V1.0	HuggingFace WiseModel ModelScope	Information Extraction Model
Dialogue Model	LlaMA2	OceanGPT	V1.0	HuggingFace WiseModel	Ocean Model
Dialogue Model	LlaMA2	OneKE	V1.0	HuggingFace WiseModel ModelScope	Information Extraction Model

Instruction Dataset Name	Number	Download Link	Note
KnowLM-CR (CoT&Reasoning, Chinese and English)	202,333	Google Drive HuggingFace
KnowLM-Tool (Tool Learning，English)	38,241	Google Drive HuggingFace
OceanBench (Benchmark，English)	11,000	HuggingFace
InstructIE (Information Extraction, Chinese and English)	364, 076	HuggingFace WiseModel ModelScope	Due to using distant supervision, there exists noise.
IEPile (Information Extraction, Chinese and English)	2,000,000 +	HuggingFace WiseModel ModelScope	It is constructed based on 33 exsiting IE datasets.

Data description: 1. Other data sources for information extraction come from CoNLL, ACE, casis, DuEE, People Daily, DuIE, etc. 2. The KnowLM-Tool dataset comes from the paper "Making Language Models Better Tool Learners with Execution Feedback" and the gitHub can be found here. 3. The InstructIE dataset comes from the paper "InstructIE: A Chinese Instruction-based Information Extraction Dataset" and the gitHub can be found here.

NEWS

[December 2024] We open source the OneKE framework, supporting multi-agent knowledge extraction across various scenarios.
[April 2024] We release a new bilingual (Chinese and English) schema-based information extraction model called OneKE based on Chinese-Alpaca-2-13B.
[March 2024] We release a new paper: "KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents".
[February 2024] We release a large-scale (0.32B tokens) high-quality bilingual (Chinese and English) Information Extraction (IE) instruction dataset named IEPile, along with two models trained with IEPile, baichuan2-13b-iepile-lora and llama2-13b-iepile-lora.
[February 2024] We release a new paper: "EasyInstruct: An Easy-to-use Instruction Processing Framework for Large Language Models" with an HF demo EasyInstruct.
[January 2024] We release a new paper:"A Comprehensive Study of Knowledge Editing for Large Language Models" with a new benchmark KnowEdit.
[August 2023] The full parameters have been released (omitting the parameter consolidation process).
[July 2023] The instruction dataset has been released.
[July 2023] Support instruction fine-tuning and vllm for LLaMA-2
[June 2023] The project name has been changed from CaMA to KnowLM.
[June 2023] Release the first version of pre-trained weights and the LoRA weights.

Technologies in KnowLM

<br>

This is an overview of the KnowLM, which mainly consists of three technical features:

Knowledge Prompting: It generates knowledge prompts based on structured data such as knowledge graphs and utilizes knowledge augmentation constraints to address knowledge extraction and reasoning issues.

Knowledge Editing: It aligns outdated, incorrect, and biased knowledge within large models using knowledge editing techniques to tackle knowledge fallacy problems (English Tutorial).

Knowledge Interaction: It enables dynamic knowledge interaction and feedback to achieve tool-based learning and multi-agent collaboration, resolving the problem of embodiment cognition in LLMs (English Tutorial).

The modules related to these three technologies are EasyInstruct, EasyDetect, EasyEdit. We provide use cases for those modules based on the KnowLMframework.

️ Contents

Quick Start
Cases
Training Details
- Pertraining data and Pretraining scripts
- Instruction data and Instruction-tuning scripts
Limitations
TODO List
❓FAQ
Acknowledgments/Contributors/Citations

All Thanks To Our Contributors :

1. Quick Start

️1.1 Environment Configuration

KnowLM supports both manual and docker image environment configuration, you can choose the appropriate way to build.

Manual Environment Configuration

bash

git clone https://github.com/zjunlp/KnowLM.git
cd KnowLM
conda create -n knowlm python=3.9 -y
conda activate knowlm
pip install torch==1.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
pip install -r requirements.txt

Building With Docker Images

bash

docker pull zjunlp/knowlm:v.1
docker run -it zjunlp/knowlm:v.1 /bin/bash

1.2 Model Usage Guide

1. Reproduce the results in Section 2

The cases in Section 2 were all run on V100. If running on other devices, the results may vary. Please run multiple times or change the decoding parameters. We derived knowlm-13b-zhixi and knowlm-13b-ie through training using LoRA, building upon the foundation of knowlm-13b-base. These models, knowlm-13b-zhixi and knowlm-13b-ie, are the result of merging the trained LoRA weights with the existing knowlm-13b-base model parameters.

If you want to reproduce the results in section 2.1(pretraining cases), please run the following command:
bash
```
python examples/generate_finetune.py --base_model zjunlp/knowlm-13b-base-v1.0
```
The result in section 2.1 can be obtained.
If you want to reproduce the results in section 2.2(information extraction cases), please run the following command:
bash
```
python examples/generate_lora.py --base_model zjunlp/knowlm-13b-zhixi --run_ie_cases
```
The result in section 2.2 can be obtained.
If you want to reproduce the results in section 2.3(**[general abilities cases](#23-general-abili