modelfusion

ModelFusion

The TypeScript library for building AI applications.

Introduction | Quick Install | Usage | Documentation | Examples | Contributing | modelfusion.dev

Introduction

[!IMPORTANT] ModelFusion has joined Vercel and is being integrated into the Vercel AI SDK. We are bringing the best parts of modelfusion to the Vercel AI SDK, starting with text generation, structured object generation, and tool calls. Please check out the AI SDK for the latest developments.

ModelFusion is an abstraction layer for integrating AI models into JavaScript and TypeScript applications, unifying the API for common operations such as text streaming, object generation, and tool usage. It provides features to support production environments, including observability hooks, logging, and automatic retries. You can use ModelFusion to build AI applications, chatbots, and agents.

• Vendor-neutral: ModelFusion is a non-commercial open source project that is community-driven. You can use it with any supported provider.
• Multi-modal: ModelFusion supports a wide range of models including text generation, image generation, vision, text-to-speech, speech-to-text, and embedding models.
• Type inference and validation: ModelFusion infers TypeScript types wherever possible and validates model responses.
• Observability and logging: ModelFusion provides an observer framework and logging support.
• Resilience and robustness: ModelFusion ensures seamless operation through automatic retries, throttling, and error handling mechanisms.
• Built for production: ModelFusion is fully tree-shakeable, can be used in serverless environments, and only uses a minimal set of dependencies.

Quick Install

npm install modelfusion

Or use a starter template:

• ModelFusion terminal app starter
• Next.js, Vercel AI SDK, Llama.cpp & ModelFusion starter
• Next.js, Vercel AI SDK, Ollama & ModelFusion starter

Usage Examples

[!TIP] The basic examples are a great way to get started and to explore in parallel with the documentation. You can find them in the examples/basic folder.

You can provide API keys for the different integrations using environment variables (e.g., OPENAI_API_KEY) or pass them into the model constructors as options.

Generate Text

Generate text using a language model and a prompt. You can stream the text if it is supported by the model. You can use images for multi-modal prompting if the model supports it (e.g. with llama.cpp). You can use prompt styles to use text, instruction, or chat prompts.

generateText

import { generateText, openai } from "modelfusion";

const text = await generateText({
  model: openai.CompletionTextGenerator({ model: "gpt-3.5-turbo-instruct" }),
  prompt: "Write a short story about a robot learning to love:\n\n",
});

Providers: OpenAI, OpenAI compatible, Llama.cpp, Ollama, Mistral, Hugging Face, Cohere

streamText

import { streamText, openai } from "modelfusion";

const textStream = await streamText({
  model: openai.CompletionTextGenerator({ model: "gpt-3.5-turbo-instruct" }),
  prompt: "Write a short story about a robot learning to love:\n\n",
});

for await (const textPart of textStream) {
  process.stdout.write(textPart);
}

Providers: OpenAI, OpenAI compatible, Llama.cpp, Ollama, Mistral, Cohere

streamText with multi-modal prompt

Multi-modal vision models such as GPT 4 Vision can process images as part of the prompt.

import { streamText, openai } from "modelfusion";
import { readFileSync } from "fs";

const image = readFileSync("./image.png");

const textStream = await streamText({
  model: openai
    .ChatTextGenerator({ model: "gpt-4-vision-preview" })
    .withInstructionPrompt(),

  prompt: {
    instruction: [
      { type: "text", text: "Describe the image in detail." },
      { type: "image", image, mimeType: "image/png" },
    ],
  },
});

for await (const textPart of textStream) {
  process.stdout.write(textPart);
}

Providers: OpenAI, OpenAI compatible, Llama.cpp, Ollama

Generate Object

Generate typed objects using a language model and a schema.

generateObject

Generate an object that matches a schema.

…

Providers: OpenAI, Ollama, Llama.cpp

streamObject

Stream a object that matches a schema. Partial objects before the final part are untyped JSON.

…

Providers: OpenAI, Ollama, Llama.cpp

Generate Image

Generate an image from a prompt.

import { generateImage, openai } from "modelfusion";

const image = await generateImage({
  model: openai.ImageGenerator({ model: "dall-e-3", size: "1024x1024" }),
  prompt:
    "the wicked witch of the west in the style of early 19th century painting",
});

Providers: OpenAI (Dall·E), Stability AI, Automatic1111

Generate Speech

Synthesize speech (audio) from text. Also called TTS (text-to-speech).

generateSpeech

generateSpeech synthesizes speech from text.

import { generateSpeech, lmnt } from "modelfusion";

// `speech` is a Uint8Array with MP3 audio data
const speech = await generateSpeech({
  model: lmnt.SpeechGenerator({
    voice: "034b632b-df71-46c8-b440-86a42ffc3cf3", // Henry
  }),
  text:
    "Good evening, ladies and gentlemen! Exciting news on the airwaves tonight " +
    "as The Rolling Stones unveil 'Hackney Diamonds,' their first collection of " +
    "fresh tunes in nearly twenty years, featuring the illustrious Lady Gaga, the " +
    "magical Stevie Wonder, and the final beats from the late Charlie Watts.",
});

Providers: Eleven Labs, LMNT, OpenAI

streamSpeech

generateSpeech generates a stream of speech chunks from text or from a text stream. Depending on the model, this can be fully duplex.

import { streamSpeech, elevenlabs } from "modelfusion";

const textStream: AsyncIterable;

const speechStream = await streamSpeech({
  model: elevenlabs.SpeechGenerator({
    model: "eleven_turbo_v2",
    voice: "pNInz6obpgDQGcFmaJgB", // Adam
    optimizeStreamingLatency: 1,
    voiceSettings: { stability: 1, similarityBoost: 0.35 },
    generationConfig: {
      chunkLengthSchedule: [50, 90, 120, 150, 200],
    },
  }),
  text: textStream,
});

for await (const part of speechStream) {
  // each part is a Uint8Array with MP3 audio data
}

Providers: Eleven Labs

Generate Transcription

Transcribe speech (audio) data into text. Also called speech-to-text (STT).

import { generateTranscription, openai } from "modelfusion";
import fs from "node:fs";

const transcription = await generateTranscription({
  model: openai.Transcriber({ model: "whisper-1" }),
  mimeType: "audio/mp3",
  audioData: await fs.promises.readFile("data/test.mp3"),
});

Providers: OpenAI (Whisper), Whisper.cpp

Embed Value

Create embeddings for text and other values. Embeddings are vectors that represent the essence of the values in the context of the model.

import { embed, embedMany, openai } from "modelfusion";

// embed single value:
const embedding = await embed({
  model: openai.TextEmbedder({ model: "text-embedding-ada-002" }),
  value: "At first, Nox didn't know what to do with the pup.",
});

// embed many values:
const embeddings = await embedMany({
  model: openai.TextEmbedder({ model: "text-embedding-ada-002" }),
  values: [
    "At first, Nox didn't know what to do with the pup.",
    "He keenly observed and absorbed everything around him, from the birds in the sky to the trees in the forest.",
  ],
});

Providers: OpenAI, OpenAI compatible, Llama.cpp, Ollama, Mistral, Hugging Face, Cohere

Classify Value

Classifies a value into a category.

…

Classifiers: EmbeddingSimilarityClassifier

Tokenize Text

Split text into tokens and reconstruct the text from tokens.

const tokenizer = openai.Tokenizer({ model: "gpt-4" });

const text = "At first, Nox didn't know what to do with the pup.";

const tokenCount = await countTokens(tokenizer, text);

const tokens = await tokenizer.tokenize(text);
const tokensAndTokenTexts = await tokenizer.tokenizeWithTexts(text);
const reconstructedText = await tokenizer.detokenize(tokens);

Providers: OpenAI,