react-llm

@react-llm/headless

Easy-to-use headless React Hooks to run LLMs in the browser with WebGPU. As simple as useLLM().

Live Demo

Features:

• Supports Vicuna 7B
• Use custom system prompts and "user:"/"assistant:" role names
• Completion options like max tokens and stop sequences
• No data leaves the browser. Accelerated via WebGPU.
• Hooks built to 'Bring your own UI'
• Persistent storage for conversations in browser storage. Hooks for loading and saving conversations.
• Model caching for faster subsequent loads

Installation

npm install @react-llm/headless

Packages in this repository

• @react-llm/model - The LLM model and tokenizer compiled for the browser
• @react-llm/retro-ui - Retro-themed UI for the hooks
• @react-llm/extension - Chrome Extension that uses the hooks
• @react-llm/headless - Headless React Hooks for running LLMs in the browser

useLLM API

Types

…

Hooks

import useLLM from '@react-llm/headless';

const MyComponent = () => {
  const {
    conversation,
    allConversations,
    loadingStatus,
    isGenerating,
    createConversation,
    setConversationId,
    deleteConversation,
    deleteAllConversations,
    deleteMessages,
    setConversationTitle,
    onMessage,
    setOnMessage,
    userRoleName,
    setUserRoleName,
    assistantRoleName,
    setAssistantRoleName,
    gpuDevice,
    send,
    init,
  } = useLLM();

  // Component logic...

  return null;
};

Provider

import { ModelProvider } from "@react-llm/headless";

export default function Home() {
  return (
    
      
    
  );
}

Packages

• @react-llm/headless - Headless React Hooks for running LLMs in the browser
• @react-llm/retro-ui - Retro-themed UI for the hooks

How does it work?

This library is a set of React Hooks that provide a simple interface to run LLMs in the browser. It uses Vicuna 13B.

• SentencePiece tokenizer (compiled for the browser via Emscripten)
• Vicuna 7B (transformed to Apache TVM format)
• Apache TVM and MLC Relax (compiled for the browser via Emscripten)
• Off-the-main-thread WebWorker to run the model (bundled with the library)

The model, tokenizer, and TVM runtime are loaded from a CDN (huggingface). The model is cached in browser storage for faster subsequent loads.

Example

See packages/retro-ui for the full demo code. This is a simple example of how to use the hooks. To run it, after cloning the repo,

cd packages/retro-ui
pnpm install
pnpm dev

License

MIT

The code under packages/headless/worker/lib/tvm is licensed under Apache 2.0.