frequencylaw

README.md

FrequencyLaw: Textual Frequency Law on Large Language Models

Project Introduction

This project is the official code repository accompanying the paper Textual Frequency Law on Large Language Models. It implements the three core methods proposed in the paper: Textual Frequency Law (TFL), Textual Frequency Distillation (TFD), and Curriculum Textual Frequency Training (CTFT), verifying the optimization effect of textual frequency on Large Language Models (LLMs) in Mathematical Reasoning (MR) and Machine Translation (MT) tasks.

Based on GSM8K (Mathematical Reasoning) and FLORES-200 (Machine Translation), the project constructs the Textual Frequency Paired Dataset (TFPD). It provides end-to-end code for frequency calculation, dataset processing, model fine-tuning and evaluation, enabling the reproduction of the experimental conclusions in the paper.

Environment Setup

Dependency Installation

pip install -r requirements.txt

If requirements.txt is not generated, run the following command to create it:

pip freeze > requirements.txt

Core Dependencies

• Python 3.9+
• PyTorch 2.0+
• Hugging Face Transformers/Datasets/Accelerate
• NumPy/Pandas (Data Processing)
• LoRA (peft): For lightweight model fine-tuning

Project Structure

…

Community Implementations

• AdamOpt – Turn "Prompt Alchemy" into science. Boost LLM performance 10-20% with zero cost and zero training. Thanks to @happyii for the outstanding work.