#678·parlant

[Enhancement] xllamacpp NLP adapter

Author: iwr-redmondCreated Dec 19, 2025Updated Jan 26, 2026
Labelsenhancement

Motivation

The main option for local inference is currently Ollama, which requires third-party installation and configuration. Additionally, Ollama does not fully support structured outputs; it merely proposes the structure to the LLM and validates the response after generation.

By comparison, xllamacpp can be installed using pip and includes LLGuidance, which constrains generation to required JSON schemas regardless of the quality of the model selected. This allows for complete installation management within Parlant itself as well as reliable JSON response generation for all potential LLMs.

Solution Proposal

  1. Create an NLP adapter for xllamacpp; I reckon this will have some differences to the standard OpenAI adapter, e.g. using the json_schema_to_grammar function to activate structured outputs and appending the JSON schema to each prompt to ensure the LLM gets as close as possible without the intervention of LLGuidance
  2. Select default models and code their GGUF files into the NLP adapter for ease of use, allowing for auto-downloading when these models are missing
  3. Create a [local] install group, which draws the xllamacpp package from the Vulkan index on Windows and Linux; this will allow for immediate accelerated inference regardless of whether the end user has an NVIDIA, AMD, Intel, or Apple GPU (Metal acceleration is available via the PyPI package)
  4. Clone the healthcare example to create a local inference copy

Discussion

A useful explanation of how constrained sampling works is available here.