AutoSchemaKG

AutoSchemaKG: A Knowledge Graph Construction Framework with Schema Generation and Knowledge Graph Completion

This repository contains the implementation of AutoSchemaKG, a novel framework for automatic knowledge graph construction that combines schema generation via conceptualization. The framework is designed to address the challenges of constructing high-quality knowledge graphs from unstructured text.

Homepage and Documentation: https://hkust-knowcomp.github.io/AutoSchemaKG/

This project uses the following paper and data:

• Paper: Read the paper
• Full Data: Download the dataset (huggingface dataset)
• Neo4j CSV Dumps: Download the dataset (huggingface dataset)

Update

• (05/12) Add Documentation for atlas-rag package and example directory. Include quick start examples for knowledge graph construction, hosting, and multi-hop QA evaluation.
• (05/07) Update with batch generation and refactor the codebase. Add comprehensive documentation for examples including PDF/Markdown conversion, multi-language processing, parallel generation, and custom extraction.
• (24/06) Add: ToG, Chinese KG construction (refer to example/multilingual_processing.md for KG construction with different languages). Separate NV-embed-v2 transformers dependency.

AutoSchemaKG Overview

AutoSchemaKG introduces a two-stage approach:

1. Knowledge Graph Triple Extraction: Extract triples comprising entities and events from text by using LLMs
2. Schema Induction: Automatically generate schema for the knowledge graph by using conceptualization and create semantic bridges between seemingly disparate information to enable zero-shot inferencing across domains

The framework achieves state-of-the-art performance on multiple benchmarks and demonstrates strong generalization capabilities across different domains.

ATLAS Knowledge Graphs

ATLAS (Automated Triple Linking And Schema induction) is a family of knowledge graphs created through the AutoSchemaKG framework, which enables fully autonomous knowledge graph construction without predefined schemas. Here's a summary of what ATLAS is and how it works:

Key Features of ATLAS Knowledge Graphs

• Scale: Consists of 900+ million nodes connected by 5.9 billion edges
• Autonomous Construction: Built without predefined schemas or manual intervention
• Three Variants: ATLAS-Wiki (from Wikipedia), ATLAS-Pes2o (from academic papers), and ATLAS-CC (from Common Crawl)

Project Structure

…

The project is organized into several key components:

• atlas_rag/: Core package with KG construction, LLM generation, retrieval, and vector storage
• example/: Complete tutorials, scripts, and sample data for various use cases
• Evaluation directories: Comprehensive metrics for KG quality, factuality, and general performance
• neo4j_scripts/: Database management and hosting utilities
• tests/: Unit tests ensuring code reliability

Install atlas-rag through pip

pip install atlas-rag

To support NV-embed-v2, install the transformers package with the version constraint >=4.42.4,',base_url="")

model_name = "meta-llama/llama-3.1-8b-instruct"

model_name = "meta-llama/Llama-3.1-8B-Instruct" client = pipeline( "text-generation", model=model_name, device_map="auto", ) keyword = 'Dulce' output_directory = f'import/{keyword}' triple_generator = LLMGenerator(client, model_name=model_name) kg_extraction_config = ProcessingConfig( model_path=model_name, data_directory="example_data", filename_pattern=keyword, # Will read the files with string filename_patterns in the data directory as input files batch_size_triple=3, # batch size for triple extraction batch_size_concept=16, # batch size for concept generation output_directory=f"{output_directory}", max_new_tokens=2048, max_workers=3, remove_doc_spaces=True, # For removing duplicated spaces in the document text ) kg_extractor = KnowledgeGraphExtractor(model=triple_generator, config=kg_extraction_config)

Construct entity&event graph

kg_extractor.run_extraction() # Involved LLM Generation

Convert Triples Json to CSV

kg_extractor.convert_json_to_csv()

Concept Generation

kg_extractor.generate_concept_csv_temp(batch_size=64) # Involved LLM Generation

Create Concept CSV

kg_extractor.create_concept_csv()

Convert csv to graphml for networkx

kg_extractor.convert_to_graphml()

…

@misc{bai2025autoschemakgautonomousknowledgegraph, title={AutoSchemaKG: Autonomous Knowledge Graph Construction through Dynamic Schema Induction from Web-Scale Corpora}, author={Jiaxin Bai and Wei Fan and Qi Hu and Qing Zong and Chunyang Li and Hong Ting Tsang and Hongyu Luo and Yauwai Yim and Haoyu Huang and Xiao Zhou and Feng Qin and Tianshi Zheng and Xi Peng and Xin Yao and Huiwen Yang and Leijie Wu and Yi Ji and Gong Zhang and Renhai Chen and Yangqiu Song}, year={2025}, eprint={2505.23628}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2505.23628}, }

## Contact

Jiaxin Bai: [email protected] 

Dennis Hong Ting TSANG : [email protected]

Haoyu Huang: [email protected]