LLM-Engineers-Handbook

Find the book on Amazon or Packt

Features

The goal of this book is to create your own end-to-end LLM-based system using best practices:

• Data collection & generation
• LLM training pipeline
• Simple RAG system
• Production-ready AWS deployment
• Comprehensive monitoring
• Testing and evaluation framework

You can download and use the final trained model on Hugging Face.

[!IMPORTANT] The code in this GitHub repository is actively maintained and may contain updates not reflected in the book. Always refer to this repository for the latest version of the code.

Dependencies

Local dependencies

To install and run the project locally, you need the following dependencies.

Cloud services

The code also uses and depends on the following cloud services. For now, you don't have to do anything. We will guide you in the installation and deployment sections on how to use them:

In the LLM Engineer's Handbook, Chapter 2 will walk you through each tool. Chapters 10 and 11 provide step-by-step guides on how to set up everything you need.

️ Project Structure

Here is the directory overview:

.
├── code_snippets/       # Standalone example code
├── configs/             # Pipeline configuration files
├── llm_engineering/     # Core project package
│   ├── application/    
│   ├── domain/         
│   ├── infrastructure/ 
│   ├── model/         
├── pipelines/           # ML pipeline definitions
├── steps/               # Pipeline components
├── tests/               # Test examples
├── tools/               # Utility scripts
│   ├── run.py
│   ├── ml_service.py
│   ├── rag.py
│   ├── data_warehouse.py

llm_engineering/ is the main Python package implementing LLM and RAG functionality. It follows Domain-Driven Design (DDD) principles:

• domain/: Core business entities and structures
• application/: Business logic, crawlers, and RAG implementation
• model/: LLM training and inference
• infrastructure/: External service integrations (AWS, Qdrant, MongoDB, FastAPI)

The code logic and imports flow as follows: infrastructure → model → application → domain

pipelines/: Contains the ZenML ML pipelines, which serve as the entry point for all the ML pipelines. Coordinates the data processing and model training stages of the ML lifecycle.

steps/: Contains individual ZenML steps, which are reusable components for building and customizing ZenML pipelines. Steps perform specific tasks (e.g., data loading, preprocessing) and can be combined within the ML pipelines.

tests/: Covers a few sample tests used as examples within the CI pipeline.

tools/: Utility scripts used to call the ZenML pipelines and inference code:

• run.py: Entry point script to run ZenML pipelines.
• ml_service.py: Starts the REST API inference server.
• rag.py: Demonstrates usage of the RAG retrieval module.
• data_warehouse.py: Used to export or import data from the MongoDB data warehouse through JSON files.

configs/: ZenML YAML configuration files to control the execution of pipelines and steps.

code_snippets/: Independent code examples that can be executed independently.

Installation

[!NOTE] If you are experiencing issues while installing and running the repository, consider checking the Issues GitHub section for other people who solved similar problems or directly asking us for help.

1. Clone the Repository

Start by cloning the repository and navigating to the project directory:

git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git
cd LLM-Engineers-Handbook 

Next, we have to prepare your Python environment and its adjacent dependencies.

2. Set Up Python Environment

The project requires Python 3.11. You can either use your global Python installation or set up a project-specific version using pyenv.

Option A: Using Global Python (if version 3.11 is installed)

Verify your Python version:

python --version  # Should show Python 3.11.x

Option B: Using pyenv (recommended)

1. Verify pyenv installation:

pyenv --version   # Should show pyenv 2.3.36 or later

1. Install Python 3.11.8:

pyenv install 3.11.8

1. Verify the installation:

python --version  # Should show Python 3.11.8

1. Confirm Python version in the project directory:

python --version
# Output: Python 3.11.8

[!NOTE]
The project includes a .python-version file that automatically sets the correct Python version when you're in the project directory.

3. Install Dependencies

The project uses Poetry for dependency management.

1. Verify Poetry installation:

poetry --version  # Should show Poetry version 1.8.3 or later

1. Set up the project environment and install dependencies:

poetry env use 3.11
poetry install --without aws
poetry run pre-commit install

This will:

• Configure Poetry to use Python 3.11
• Install project dependencies (excluding AWS-specific packages)
• Set up pre-commit hooks for code verification

4. Activate the Environment

As our task manager, we run all the scripts using Poe the Poet.

1. Start a Poetry shell:

poetry shell

1. Run project commands using Poe the Poet:

poetry poe ...

Alternative Command Execution

If you're experiencing issues with poethepoet, you can still run the project commands directly through Poetry. Here's how:

1. Look up the command definition in pyproject.toml
2. Use poetry run with the underlying command

Example:

Instead of:

poetry poe local-infrastructure-up

Use the direct command from pyproject.toml:

poetry run <actual-command-from-pyproject-toml>

Note: All project commands are defined in the [tool.poe.tasks] section of pyproject.toml

Now, let's configure our local project with all the necessary credentials and tokens to run the code locally.

5. Local Development Setup

After you have installed all the dependencies, you must create and fill a .env file with your credentials to appropriately interact with other services and run the project. Setting your sensitive credentials in a .env file is a good security practice, as this file won't be committed to GitHub or shared with anyone else.

1. First, copy our example by running the following:

cp .env.example .env # The file must be at your repository's root!

1. Now, let's understand how to fill in all the essential variables within the .env file to get you started. The following are the mandatory settings we must complete when working locally:

OpenAI

To authenticate to OpenAI's API, you must fill out the OPENAI_API_KEY env var with an authentication token.

OPENAI_API_KEY=your_api_key_here

→ Check out this tutorial to learn how to provide one from OpenAI.

Hugging Face

To authenticate to Hugging Face, you must fill out the HUGGINGFACE_ACCESS_TOKEN env var with an authentication token.

HUGGINGFACE_ACCESS_TOKEN=your_token_here

→ Check out this tutorial to learn how to provide one from Hugging Face.

Comet ML & Opik

To authenticate to Comet ML (required only during training) and Opik, you must fill out the COMET_API_KEY env var with your authentication token.

COMET_API_KEY=your_api_key_here

→ Check out this tutorial to learn how to get started with Opik. You can also access Opik's dashboard using this link.

6. Deployment Setup

When deploying the project to the cloud, we must set additional settings for Mongo, Qdrant, and AWS. If you are just working locally, the default values of these env vars will work out of the box. Detailed deployment instructions are available in Chapter 11 of the LLM Engineer's Handbook.

MongoDB

We must change the DATABASE_HOST env var with the URL pointing to your cloud MongoDB cluster.

DATABASE_HOST=your_mongodb_url

→ Check out this tutorial to learn how to create and host a MongoDB cluster for free.

Qdrant

Change USE_QDRANT_CLOUD to true, QDRANT_CLOUD_URL with the URL point to your cloud Qdrant cluster, and QDRANT_APIKEY with its API key.

USE_QDRANT_CLOUD=true
QDRANT_CLOUD_URL=your_qdrant_cloud_url
QDRANT_APIKEY=your_qdrant_api_key

→ Check out this tutorial to learn how to create a Qdrant cluster for free

AWS

For your AWS set-up to work correctly, you need the AWS CLI installed on your local machine and properly configured with an admin user (or a user with enough permissions to create new SageMaker, ECR, and S3 resources; using an admin user will make everything more straightforward).

Chapter 2 provides step-by-step instructions on how to install the AWS CLI, create an admin user on AWS, and get an access key to set up the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables. If you already have an AWS admin user in place, you have to configure the following env vars in your .env file:

AWS_REGION=eu-central-1 # Change it with your AWS region.
AWS_ACCESS_KEY=your_aws_access_key
AWS_SECRET_KEY=your_aws_secret_key

AWS credentials are typically stored in ~/.aws/credentials. You can view this file directly using cat or similar commands: