[Enhancement]: Simplify Custom LLM provider setup through the web UI

Target Component

External Integrations (LLM/Search APIs)

Enhancement Description

Problem Statement

Current Situation

Custom LLM provider configuration is currently split across multiple places:

• .env / Docker Compose variables for API keys, endpoints, and connection settings
• mounted provider YAML files for model and agent configuration
• Settings → Providers for provider profiles and runtime options

This is difficult for users who want to connect an OpenAI-compatible endpoint such as vLLM, LiteLLM, OneAPI, OpenRouter, or a self-hosted gateway. Users must understand environment variables, mounted files, provider-specific YAML, and service restarts before they can create a usable provider profile in the web UI.

Use Case

As a self-hosted PentAGI user, I want to add and test a custom OpenAI-compatible LLM endpoint directly from the web UI, so that I can configure a provider without editing .env files or mounted YAML files for every new endpoint or model.

Proposed Solution

Feature Description

Add an optional web-based setup flow for Custom / OpenAI-compatible LLM providers.

The provider form could include:

• Provider profile name
• API base URL
• API key
• API compatibility type, initially OpenAI-compatible
• Model name, with manual input as a fallback
• Per-agent model assignment and existing runtime options

The UI should provide a “Test connection” action with actionable results, for example:

• endpoint unreachable
• authentication failed
• model not found
• unsupported API response format
• streaming or tool-calling unsupported

Optionally, the UI could retrieve models from the standard GET /v1/models endpoint after valid credentials are entered.

Credentials created through the UI should be stored encrypted at rest and returned to the frontend only as masked values. The existing .env and mounted YAML configuration should remain supported as the server-managed/deployment-oriented configuration method.

Expected Benefits

• Makes custom provider setup accessible to non-technical self-hosted users.
• Reduces configuration errors caused by splitting setup across .env, YAML, and the web UI.
• Makes it easier to test and switch between local and hosted OpenAI-compatible services.
• Preserves existing deployment automation and advanced configuration options.
• Enables a clearer separation between connection configuration and runtime/model configuration.

Example Scenario

A user runs a vLLM or LiteLLM endpoint on their local network.

1. They open Settings → Providers and select “Create Provider”.
2. They choose “OpenAI-compatible”.
3. They enter a name, base URL, API key, and model name.
4. They click “Test connection”.
5. PentAGI validates the endpoint and reports whether chat, streaming, and tool calling are available.
6. The user saves the provider and assigns its model to the desired agents.

No manual editing of .env, provider YAML files, Docker mounts, or container restart is required for this user-managed provider.

Technical Details

Implementation Approach

Architecture Changes

Introduce a distinction between:

1. Server-managed provider configuration
   Existing .env, Compose, and mounted YAML configuration remains unchanged.

2. User-managed provider configuration
   Provider connection settings created from the web UI are persisted in the database and loaded alongside existing provider profiles.

An initial implementation could limit this feature to the existing Custom/OpenAI-compatible provider path.

New Components

• A database model/table for encrypted provider connection credentials and endpoint settings.
• Server-side encryption/decryption using an administrator-provided master key.
• API/GraphQL mutations for creating, updating, testing, and deleting user-managed provider connection settings.
• A frontend provider setup wizard or expanded Custom provider form.
• A provider capability test service for:
  • basic chat completion
  • optional model discovery via GET /v1/models
  • streaming support
  • tool-calling support

Dependencies

No external dependency should be required initially. Existing Go cryptography packages and the existing provider abstraction should be sufficient.

Integration Points

AI Agents

The resulting provider profile should integrate with the existing per-agent model assignment mechanism. Existing runtime parameters, reasoning settings, pricing metadata, and provider tests should continue to work.

Memory System

No changes are required initially. Embedding-provider configuration can remain server-managed and out of scope for the first iteration.

Monitoring

Provider connection tests should be logged without exposing credentials. Existing observability/tracing should also redact API keys and authorization headers.

Security Considerations

• API keys must be encrypted at rest; plaintext credentials must never be returned by API or GraphQL responses.
• The encryption master key should be supplied only through server-side environment configuration.
• Credentials and provider records must be scoped to the owning user/tenant and protected by existing provider permissions.
• Logs, errors, traces, and test output must redact API keys and authorization headers.
• Existing .env and YAML configuration remains preferable for centrally managed production deployments.

Designs and Mockups

Suggested UI Flow

flowchart TD
    A[Settings → Providers] --> B[Create Provider]
    B --> C{Provider setup type}
    C -->|OpenAI-compatible| D[Enter name, URL, API key, model]
    C -->|Existing server-managed type| E[Use current provider profile flow]
    D --> F[Test connection]
    F -->|Success| G[Show available capabilities and optional model list]
    F -->|Failure| H[Show actionable diagnostic]
    G --> I[Save encrypted connection settings]
    I --> J[Configure per-agent model and runtime options]

Alternative Solutions

Alternative Solutions

Alternative Approaches

1. Improve documentation and examples only

• Pros: low implementation cost; no changes to credential storage.
• Cons: users still need to edit .env, YAML files, Docker mounts, and restart services.

2. Keep credentials in .env but add a UI generator

• Pros: avoids database credential storage.
• Cons: the application cannot safely write the host .env file in many deployments; users still need to apply changes and restart services.

3. Support only static provider YAML files

• Pros: suitable for centrally managed and infrastructure-as-code deployments.
• Cons: does not solve the usability issue for individual self-hosted users.

Reason for Preferred Solution

A web-managed provider flow provides the best user experience while retaining .env and YAML configuration for operators who prefer declarative, centrally managed deployment. Starting with OpenAI-compatible endpoints also keeps the initial scope limited and useful.

Verification

• I have checked that this enhancement hasn't been already proposed
• This enhancement aligns with PentAGI's goal of autonomous penetration testing
• I have considered the security implications of this enhancement
• I have provided clear use cases and benefits