langchainrb

Langchain.rb

⚡ Building LLM-powered applications in Ruby ⚡

For deep Rails integration see: langchainrb_rails gem.

Available for paid consulting engagements! Email me.

Use Cases

• Retrieval Augmented Generation (RAG) and vector search
• Assistants (chat bots)

Table of Contents

• Installation
• Usage
• Unified Interface for LLMs
• Prompt Management
• Output Parsers
• Building RAG
• Assistants
• Evaluations
• Examples
• Logging
• Problems
• Development
• Discord

Installation

Install the gem and add to the application's Gemfile by executing:

bundle add langchainrb

If bundler is not being used to manage dependencies, install the gem by executing:

gem install langchainrb

Additional gems may be required. They're not included by default so you can include only what you need.

Usage

require "langchain"

Unified Interface for LLMs

The Langchain::LLM module provides a unified interface for interacting with various Large Language Model (LLM) providers. This abstraction allows you to easily switch between different LLM backends without changing your application code.

Supported LLM Providers

• Anthropic
• AWS Bedrock
• Azure OpenAI
• Cohere
• Google Gemini
• Google Vertex AI
• HuggingFace
• Mistral AI
• Ollama
• OpenAI
• Replicate

Usage

All LLM classes inherit from Langchain::LLM::Base and provide a consistent interface for common operations:

1. Generating embeddings
2. Generating prompt completions
3. Generating chat completions

Initialization

Most LLM classes can be initialized with an API key and optional default options:

llm = Langchain::LLM::OpenAI.new(
  api_key: ENV["OPENAI_API_KEY"],
  default_options: { temperature: 0.7, chat_model: "gpt-4o" }
)

Generating Embeddings

Use the embed method to generate embeddings for given text:

response = llm.embed(text: "Hello, world!")
embedding = response.embedding

Accepted parameters for embed()

• text: (Required) The input text to embed.
• model: (Optional) The model name to use or default embedding model will be used.

Prompt completions

Use the complete method to generate completions for a given prompt:

response = llm.complete(prompt: "Once upon a time")
completion = response.completion

Accepted parameters for complete()

• prompt: (Required) The input prompt for completion.
• max_tokens: (Optional) The maximum number of tokens to generate.
• temperature: (Optional) Controls randomness in generation. Higher values (e.g., 0.8) make output more random, while lower values (e.g., 0.2) make it more deterministic.
• top_p: (Optional) An alternative to temperature, controls diversity of generated tokens.
• n: (Optional) Number of completions to generate for each prompt.
• stop: (Optional) Sequences where the API will stop generating further tokens.
• presence_penalty: (Optional) Penalizes new tokens based on their presence in the text so far.
• frequency_penalty: (Optional) Penalizes new tokens based on their frequency in the text so far.

Generating Chat Completions

Use the chat method to generate chat completions:

messages = [
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "What's the weather like today?" }
  # Google Gemini and Google VertexAI expect messages in a different format:
  # { role: "user", parts: [{ text: "why is the sky blue?" }]}
]
response = llm.chat(messages: messages)
chat_completion = response.chat_completion

Accepted parameters for chat()

• messages: (Required) An array of message objects representing the conversation history.
• model: (Optional) The specific chat model to use.
• temperature: (Optional) Controls randomness in generation.
• top_p: (Optional) An alternative to temperature, controls diversity of generated tokens.
• n: (Optional) Number of chat completion choices to generate.
• max_tokens: (Optional) The maximum number of tokens to generate in the chat completion.
• stop: (Optional) Sequences where the API will stop generating further tokens.
• presence_penalty: (Optional) Penalizes new tokens based on their presence in the text so far.
• frequency_penalty: (Optional) Penalizes new tokens based on their frequency in the text so far.
• logit_bias: (Optional) Modifies the likelihood of specified tokens appearing in the completion.
• user: (Optional) A unique identifier representing your end-user.
• tools: (Optional) A list of tools the model may call.
• tool_choice: (Optional) Controls how the model calls functions.

Switching LLM Providers

Thanks to the unified interface, you can easily switch between different LLM providers by changing the class you instantiate:

# Using Anthropic
anthropic_llm = Langchain::LLM::Anthropic.new(api_key: ENV["ANTHROPIC_API_KEY"])

# Using Google Gemini
gemini_llm = Langchain::LLM::GoogleGemini.new(api_key: ENV["GOOGLE_GEMINI_API_KEY"])

# Using OpenAI
openai_llm = Langchain::LLM::OpenAI.new(api_key: ENV["OPENAI_API_KEY"])

Response Objects

Each LLM method returns a response object that provides a consistent interface for accessing the results:

• embedding: Returns the embedding vector
• completion: Returns the generated text completion
• chat_completion: Returns the generated chat completion
• tool_calls: Returns tool calls made by the LLM
• prompt_tokens: Returns the number of tokens in the prompt
• completion_tokens: Returns the number of tokens in the completion
• total_tokens: Returns the total number of tokens used

[!NOTE] While the core interface is consistent across providers, some LLMs may offer additional features or parameters. Consult the documentation for each LLM class to learn about provider-specific capabilities and options.

Prompt Management

Prompt Templates

Create a prompt with input variables:

prompt = Langchain::Prompt::PromptTemplate.new(template: "Tell me a {adjective} joke about {content}.", input_variables: ["adjective", "content"])
prompt.format(adjective: "funny", content: "chickens") # "Tell me a funny joke about chickens."

Creating a PromptTemplate using just a prompt and no input_variables:

prompt = Langchain::Prompt::PromptTemplate.from_template("Tell me a funny joke about chickens.")
prompt.input_variables # []
prompt.format # "Tell me a funny joke about chickens."

Save prompt template to JSON file:

prompt.save(file_path: "spec/fixtures/prompt/prompt_template.json")

Loading a new prompt template using a JSON file:

prompt = Langchain::Prompt.load_from_path(file_path: "spec/fixtures/prompt/prompt_template.json")
prompt.input_variables # ["adjective", "content"]

Few Shot Prompt Templates

Create a prompt with a few shot examples:

…

Save prompt template to JSON file:

prompt.save(file_path: "spec/fixtures/prompt/few_shot_prompt_template.json")

Loading a new prompt template using a JSON file:

prompt = Langchain::Prompt.load_from_path(file_path: "spec/fixtures/prompt/few_shot_prompt_template.json")
prompt.prefix # "Write antonyms for the following words."

Loading a new prompt template using a YAML file:

prompt = Langchain::Prompt.load_from_path(file_path: "spec/fixtures/prompt/prompt_template.yaml")
prompt.input_variables #=> ["adjective", "content"]

Output Parsers

Parse LLM text responses into structured output, such as JSON.

Structured Output Parser

You can use the StructuredOutputParser to generate a prompt that instructs the LLM to provide a JSON response adhering to a specific JSON schema:

…

Then parse the llm response:

llm = Langchain::LLM::OpenAI.new(api_key: ENV["OPENAI_API_KEY"])
llm_response = llm.chat(messages: [{role: "user", content: prompt_text}]).completion
parser.parse(llm_response)
# {
#   "name" => "Kim Ji-hyun",
#   "age" => 22,
#   "interests" => [
#     {
#       "interest" => "Organic Chemistry",
#       "levelOfInterest" => 85
#     },
#     ...
#   ]
# }

If the parser fails to parse the LLM response, you can use the OutputFixingParser. It sends an error message, prior output, and the original prompt text to the LLM, asking for a "fixed" response:

begin
  parser.parse(llm_response)
rescue Langchain::OutputParsers::OutputParserException => e
  fix_parser = Langchain::OutputParsers::OutputFixingParser.from_llm(
    llm: llm,
    parser: parser
  )
  fix_parser.parse(llm_response)
end

Alternatively, if you don't need to handle the OutputParserException, you can simplify the code:

# we already have the `OutputFixingParser`:
# parser = Langchain::OutputParsers::StructuredOutputParser.from_json_schema(json_schema)
fix_parser = Langchain::OutputParsers::OutputFixingParser.from_llm(
  llm: llm,
  parser: parser
)
fix_parser.parse(llm_response)

See here for a concrete example

Building Retrieval Augment Generation (RAG) system

RAG is a methodology that assists LLMs generate accurate and up-to-date information. A typical RAG workflow follows the 3 steps below:

1. Relevant knowledge (or data) is retrieved from the knowledge base (typically a vector search DB)
2. A prompt, containing retrieved knowledge above, is constructed.
3. LLM receives the prompt above to generate a text completion. Most common use-case for a RAG system is powering Q&A systems where users pose natural language questions and receive answers in natural language.

Vector search databases

Langchain.rb provides a convenient unified interface on top of supported vectorsearch databases that make it easy to configure your index, add data, query and retrieve from it.

Supported vector search databases and features:

Database Open-source Cloud offering Chroma ✅ ✅ Hnswlib ✅ ❌ Milvus ✅ ✅ Zilliz Cloud Pinecone ❌ ✅ Pgvector ✅ ✅ Qdrant ✅ ✅ Weaviate ✅ ✅ Elasticsearch ✅ ✅

Using Vector Search Databases

Pick the vector search database you'll be using, add the gem dependency and instantiate the client:

gem "weaviate-ruby", "~> 0.8.9"

Choose and instantiate the LLM provider you'll be using to generate embeddings

llm = Langchain::LLM::OpenAI.new(api_key: ENV["OPENAI_API_KEY"])

clien