chatgpt-retrieval-plugin

ChatGPT Retrieval Plugin

Build Custom GPTs with a Retrieval Plugin backend to give ChatGPT access to personal documents.

Introduction

The ChatGPT Retrieval Plugin repository provides a flexible solution for semantic search and retrieval of personal or organizational documents using natural language queries. It is a standalone retrieval backend, and can be used with ChatGPT custom GPTs, function calling with the chat completions or assistants APIs, or with the ChatGPT plugins model (deprecated). ChatGPT and the Assistants API both natively support retrieval from uploaded files, so you should use the Retrieval Plugin as a backend only if you want more granular control of your retrieval system (e.g. document text chunk length, embedding model / size, etc.).

The repository is organized into several directories:

Directory Description datastore Contains the core logic for storing and querying document embeddings using various vector database providers. docs Includes documentation for setting up and using each vector database provider, webhooks, and removing unused dependencies. examples Provides example configurations, authentication methods, and provider-specific examples. local_server Contains an implementation of the Retrieval Plugin configured for localhost testing. models Contains the data models used by the plugin, such as document and metadata models. scripts Offers scripts for processing and uploading documents from different data sources. server Houses the main FastAPI server implementation. services Contains utility services for tasks like chunking, metadata extraction, and PII detection. tests Includes integration tests for various vector database providers. .well-known Stores the plugin manifest file and OpenAPI schema, which define the plugin configuration and API specification.

This README provides detailed information on how to set up, develop, and deploy the ChatGPT Retrieval Plugin (stand-alone retrieval backend).

Table of Contents

• Quickstart
• About
  • Retrieval Plugin
  • Retrieval Plugin with custom GPTs
  • Retrieval Plugin with function calling
  • Retrieval Plugin with the plugins model (deprecated)
  • API Endpoints
  • Memory Feature
  • Security
  • Choosing an Embeddings Model
• Development
  • Setup
    • General Environment Variables
  • Choosing a Vector Database
    • Pinecone
    • Elasticsearch
    • MongoDB Atlas
    • Weaviate
    • Zilliz
    • Milvus
    • Qdrant
    • Redis
    • Llama Index
    • Chroma
    • Azure Cognitive Search
    • Azure CosmosDB Mongo vCore
    • Supabase
    • Postgres
    • AnalyticDB
  • Running the API Locally
  • Personalization
  • Authentication Methods
• Deployment
• Webhooks
• Scripts
• Limitations
• Contributors
• Future Directions

Quickstart

Follow these steps to quickly set up and run the ChatGPT Retrieval Plugin:

1. Install Python 3.10, if not already installed.
2. Clone the repository: git clone https://github.com/openai/chatgpt-retrieval-plugin.git
3. Navigate to the cloned repository directory: cd /path/to/chatgpt-retrieval-plugin
4. Install poetry: pip install poetry
5. Create a new virtual environment with Python 3.10: poetry env use python3.10
6. Activate the virtual environment: poetry shell
7. Install app dependencies: poetry install
8. Create a bearer token
9. Set the required environment variables:

…

1. Run the API locally: poetry run start
2. Access the API documentation at http://0.0.0.0:8000/docs and test the API endpoints (make sure to add your bearer token).

About

Retrieval Plugin

This is a standalone retrieval backend that can be used with ChatGPT custom GPTs, function calling with the chat completions or assistants APIs, or with the ChatGPT plugins model (deprecated).

It enables a model to carry out semantic search and retrieval of personal or organizational documents, and write answers informed by relevent retrieved context (sometimes referred to as "Retrieval-Augmented Generation" or "RAG"). It allows users to obtain the most relevant document snippets from their data sources, such as files, notes, or emails, by asking questions or expressing needs in natural language. Enterprises can make their internal documents available to their employees through ChatGPT using this plugin.

The plugin uses OpenAI's embeddings model (text-embedding-3-large 256 dimension embeddings by default) to generate embeddings of document chunks, and then stores and queries them using a vector database on the backend. As an open-source and self-hosted solution, developers can deploy their own Retrieval Plugin and register it with ChatGPT. The Retrieval Plugin supports several vector database providers, allowing developers to choose their preferred one from a list.

A FastAPI server exposes the plugin's endpoints for upserting, querying, and deleting documents. Users can refine their search results by using metadata filters by source, date, author, or other criteria. The plugin can be hosted on any cloud platform that supports Docker containers, such as Fly.io, Heroku, Render, or Azure Container Apps. To keep the vector database updated with the latest documents, the plugin can process and store documents from various data sources continuously, using incoming webhooks to the upsert and delete endpoints. Tools like Zapier or Make can help configure the webhooks based on events or schedules.

Retrieval Plugin with Custom GPTs

To create a custom GPT that can use your Retrieval Plugin for semantic search and retrieval of your documents, and even store new information back to the database, you first need to have deployed a Retrieval Plugin. For detailed instructions on how to do this, please refer to the Deployment section. Once you have your app URL (e.g., https://your-app-url.com), take the following steps:

1. Navigate to the create GPT page at https://chat.openai.com/gpts/editor.
2. Follow the standard creation flow to set up your GPT.
3. Navigate to the "Configure" tab. Here, you can manually fill in fields such as name, description, and instructions, or use the smart creator for assistance.
4. Under the "Actions" section, click on "Create new action".
5. Choose an authentication method. The Retrieval Plugin supports None, API key (Basic or Bearer) and OAuth. For more information on these methods, refer to the Authentication Methods Section.
6. Import the OpenAPI schema. You can either:
   • Import directly from the OpenAPI schema hosted in your app at https://your-app-url.com/.well-known/openapi.yaml.
   • Copy and paste the contents of this file into the Schema input area if you only want to expose the query endpoint to the GPT. Remember to change the URL under the -servers section of the OpenAPI schema you paste in.
7. Optionally, you might want to add a fetch endpoint. This would involve editing the /server/main.py file to add an endpoint and implement this for your chosen vector database. If you make this change, please consider contributing it back to the project by opening a pull request! Adding the fetch endpoint to the OpenAPI schema would allow the model to fetch more content from a document by ID if some text is cut off in the retrieved result. It might also be useful to pass in a string with the text from the retrieved result and an option to return a fixed length of context before and after the retrieved result.
8. If you want the GPT to be able to save information back to the vector database, you can give it access to the Retrieval Plugin's /upsert endpoint. To do this, copy the contents of this file into the schema area. This allows the GPT to store new information it generates or learns during the conversation. More details on this feature can be found at Memory Feature and in the docs here.

Remember: ChatGPT and custom GPTs natively support retrieval from uploaded files, so you should use the Retrieval Plugin as a backend only if you want more granular control of your retrieval system (e.g. self-hosting, embedding chunk length, embedding model / size, etc.).

Retrieval Plugin with Function Calling

The Retrieval Plugin can be integrated with function calling in both the Chat Completions API and the Assistants API. This allows the model to decide when to use your functions (query, fetch, upsert) based on the conversation context.

Function Calling with Chat Completions

In a call to the chat completions API, you can describe functions and have the model generate a JSON object containing arguments to call one or many functions. The latest models (gpt-3.5-turbo-0125 and gpt-4-turbo-preview) have been trained to detect when a function should be called and to respond with JSON that adheres to the function signature.

You can define the functions for the Retrieval Plugin endpoints and pass them in as tools when you use the Chat Completions API with one of the latest models. The model will then intelligently call the functions. You can use function calling to write queries to your APIs, call the endpoint on the backend, and return the response as a tool message to the model to continue the conversation. The function definitions/schemas and an example can be found here.

Function Calling with Assistants API

You can use the same function definitions with the OpenAI Assistants API, specifically the [function calling