[Bug]: `llmfit bench` silently substitutes an unrelated model when the requested tag isn't installed, instead of erroring

Author: stefanlasiewskiCreated Sep 14, 2026Updated Sep 15, 2026

Labelsbug

Bug description

If llmfit bench runs against a not-installed model, it will run a benchmark against a different model. I reproduced 3 times in a row with the same substitution before I identified the cause via ollama list.

qwen3.6 is not installed, but the test runs anyways.

% ollama list | grep qwen
qwen3-coder-next:latest     ca06e9e4087c    51 GB     4 hours ago
qwen2.5-coder:latest          dae161e27b0e    4.7 GB    10 days ago
%

% llmfit bench qwen3.6:latest --quality
  Quality benchmarking 1 model(s)...
  === Model: deepseek-r1:7b (Ollama) ===

  Role                 Quality       Speed  Composite
  ────────────────────────────────────────────────────
...
...
  Overall: Q:6.2  S:41.2 t/s  C:7.6
%

Environment: llmfit 1.1.14 OS: macOS (Darwin 25.6.0, Apple M3 Max), Ollama provider. Ubuntu 24.04, Ollama provider.

Impact: benchmark results can silently be attributed to the wrong model, with no indication anything went wrong — undermines confidence in any bench result unless the user manually cross-checks the printed model name against what they intended to test every time.

Suggested fix: if the requested model name doesn't exactly match an installed Ollama tag, error out immediately rather than falling back to a different installed model.

Expected behavior

An error indicating the requested model isn't installed, ideally listing available tags.

Actual behavior

llmfit runs a full quality benchmark against a different, unrelated installed model (deepseek-r1:7b in this case) with no warning that the requested name wasn't found. The output header correctly names the model that actually ran (=== Model: deepseek-r1:7b (Ollama) ===), but nothing flags the mismatch against the requested name — easy to miss, and the resulting scores can be mistaken for the requested model's performance.

Steps to reproduce

Confirm a model name is NOT in ollama list (e.g., qwen3.6:latest when only qwen2.5-coder:latest is actually installed).
Run: llmfit bench qwen3.6:latest --quality

Diagnostic report (`llmfit doctor`)

# llmfit doctor report

- llmfit version: 1.1.15
- OS: macos (aarch64)

## Detected by llmfit

SystemSpecs {
    total_ram_gb: 128.0,
    available_ram_gb: 122.68576049804688,
    total_cpu_cores: 16,
    cpu_name: "Apple M3 Max",
    has_gpu: true,
    gpu_vram_gb: Some(
        128.0,
    ),
    total_gpu_vram_gb: Some(
        128.0,
    ),
    gpu_available_gb: Some(
        107.52000427246094,
    ),
    gpu_name: Some(
        "Apple M3 Max",
    ),
    gpu_count: 1,
    unified_memory: true,
    backend: Metal,
    gpus: [
        GpuInfo {
            name: "Apple M3 Max",
            vram_gb: Some(
                128.0,
            ),
            backend: Metal,
            count: 1,
            unified_memory: true,
        },
    ],
    cluster_mode: false,
    cluster_node_count: 0,
}


## nvidia-smi (extended query)

(not available: No such file or directory (os error 2))


## nvidia-smi (standard query)

(not available: No such file or directory (os error 2))


## rocm-smi --showmeminfo vram

(not available: No such file or directory (os error 2))


## rocm-smi --showproductname

(not available: No such file or directory (os error 2))


## system_profiler SPDisplaysDataType

Graphics/Displays:

    Apple M3 Max:

      Chipset Model: Apple M3 Max
      Type: GPU
      Bus: Built-In
      Total Number of Cores: 40
      Vendor: Apple (0x106b)
      Metal Support: Metal 4
      Displays:
        DELL S2722QC:
          Resolution: 3840 x 2160 (2160p/4K UHD 1 - Ultra High Definition)
          UI Looks like: 1920 x 1080 @ 60.00Hz
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Rotation: Supported
        DELL U2719D:
          Resolution: 1440 x 2560
          UI Looks like: 720 x 1280 @ 60.00Hz
          Mirror: Off
          Online: Yes
          Rotation: 90


## vulkaninfo --summary

(not available: No such file or directory (os error 2))


## npu-smi info

(not available: No such file or directory (os error 2))


## Provider app installs

LM Studio installed: false
Docker Desktop installed: false
ollama on PATH: true

llmfit version

llmfit 1.1.15

Operating system

macOS (Apple Silicon)

Affected component

Model database (missing/incorrect model data)

Mode used

CLI (--cli flag)

GPU details (if relevant)

Additional context

No response

Source: AlexsJones/llmfit

View original on GitHub View discussion on GitHub