Support Prism ML Hadamard packs (Ternary Bonsai 2 27B, MLX 2-bit): dispatch to the pack's bundled loader + register its MODEL_CONFIG format

Is your feature request related to a problem? Please describe.

prism-ml/Ternary-Bonsai-2-27B-mlx-2bit cannot be loaded. oMLX correctly classifies it as a VLM (its config.json carries vision_config), then both the VLM path and the LLM fallback fail:

Model 'Ternary-Bonsai-2-27B-mlx-2bit' failed to load:
  VLM load failed: Model type prism_hadamard_qwen35 not supported.
   Error: No module named 'mlx_vlm.speculative.drafters.prism_hadamard_qwen35';
  LLM fallback also failed: Model type prism_hadamard_qwen35 not supported.
  Reload models after fixing the files to retry.

That not supported is expected and correct: the pack must be loaded by the loader it bundles in runtime/. What is missing is oMLX wiring that loader in.

Why we cannot simply make the model type resolvable

The pack stores its language projections in a rotated basis: every matrix is blockwise Hadamard-transformed (block 1024, explicit ±1 signs, declared in hadamard.json) before the ternary {−1, 0, +1} assignment, and the runtime must apply the matching transform to activations plus the inverse to the embedding lookup. Stock mlx-lm / mlx-vlm do neither, and crucially they do not raise — they return wrong output.

Prism ML's own demo therefore refuses to serve these packs over MLX servers (scripts/start_mlx_server.sh in PrismML-Eng/Bonsai-demo):

"No MLX server for Bonsai 2 yet. Its MLX pack needs the loader bundled in the pack, which mlx_lm.server and mlx_vlm.server do not use; serving through them would return wrong output."

So the fix has to be "load through the bundled loader", not "let the model type resolve". I verified the wrong road fails loudly for the wrong reason, too: normalizing model_type to qwen3_5 and feeding it to the stock path immediately trips the pack's own guard (Unsupported packed model schema) — and any loader that bypasses that guard silently produces garbage.

There is a second, independent failure that survives a working loader

Even once the model loads, mlx_vlm.prompt_utils.MODEL_CONFIG has no prism_hadamard_qwen35 entry, so get_message_json() raises Unsupported model. That aborts oMLX's _format_messages_for_vlm_template() as a whole and falls back to the generic formatter, which emits no image placeholders — while oMLX still extracts the images. Every vision request then dies with:

ValueError: Image features and image tokens do not match: tokens: 0, features 64

This is the same failure mode oMLX already documents for glm5_next in omlx/patches/mlx_vlm_glm5_next_compat/__init__.py, and the same fix applies: register the type against the base model's MessageFormat (qwen3_5 → LIST_WITH_IMAGE_FIRST).

Describe the solution you'd like

Follow the existing mlx_vlm_*_compat convention and dispatch through the extension point that paroquant / qwen38_modelopt_mixed already use, i.e. omlx/utils/model_loading.py::maybe_load_custom_quantization():

1. New omlx/patches/mlx_vlm_prism_hadamard/__init__.py:
   • is_supported_config(config) — identifies the pack from metadata the pack itself declares (model_type == prism_hadamard_qwen35, components.vision), so a renamed directory or a future Bonsai size still works
   • load(dir) — imports <pack>/runtime/ and calls its own vision_artifact.load_vl_model()
   • apply() — registers prism_hadamard_qwen35 in MODEL_CONFIG (copying qwen3_5's format)
   • sha256-pins runtime/*.py against the revision Prism ML publishes in Bonsai-demo/scripts/bonsai2-runtime.sha256, refusing anything that does not match. Because the loader applies weight-basis transforms, an unreviewed runtime/ does not fail — it silently produces wrong numbers, which is worth the pin.
2. One 21-line insertion in model_loading.py (the diff is below).

Non-Prism models are untouched: the branch is only entered when model_type == prism_hadamard_qwen35.

Two implementation details worth knowing

• <pack>/runtime/ has no __init__.py and its modules import each other unqualified (from codec import transcode, from runtime import Packed) — it is designed to be put straight on sys.path. A plain package import therefore raises ModuleNotFoundError. The loader builds a synthetic namespace package (types.ModuleType + __path__), then import_module("<pkg>.<name>"), plus bare-name aliases.
• Reuse the pack's own build_processor(). It already routes around the torch-gated AutoProcessor, which cannot resolve our model_type through transformers' AutoConfig (this is exactly the problem _patch_torch_free_image_processor handles for glm_ocr / dots_ocr).

Describe alternatives you've considered

• Just map the model type to qwen3_5 — rejected, see above: it either fails the pack's schema guard or silently yields wrong output.
• Vendor runtime/ into omlx/patches/.../vendor/, as mlx_vlm_inkling_compat does — viable, but Prism ML version-pins and revises that runtime, so vendoring would need re-syncing. I would rather reference the pack's own copy with a hash pin; happy to do it your way if you prefer vendoring.

Additional context

I have a working implementation and tested it locally against main's model_loading.py (v0.7.0.dev3) — the anchor for the insertion is present and the patch applies with patch -p0. Verified on oMLX 0.6.4 / build 260830015308-macos26-27, macOS 26.5.1, Apple Silicon, mlx 0.32.0 / mlx-lm 0.31.3 / mlx-vlm 0.6.3:

The dispatch diff (against main):

@@ -1328,7 +1348,28 @@ def maybe_load_custom_quantization(
         return None
+
+    from ..patches.mlx_vlm_prism_hadamard import (
+        is_supported_config as _is_prism_hadamard_pack,
+        load as _load_prism_hadamard_pack,
+    )
 
+    if is_vlm and _is_prism_hadamard_pack(config):
+        # Prism ML "Hadamard" packs (Ternary Bonsai 2) store their language
+        # projections in a rotated basis ... mlx-lm / mlx-vlm do not apply the
+        # matching activation transform, so they return *wrong output rather
+        # than an error*.  Use the pack's bundled loader instead.
+        logger.info(
+            "Prism Hadamard pack detected for %s; using its bundled loader",
+            model_name,
+        )
+        return _load_prism_hadamard_pack(model_name)
+
     quant_config = config.get("quantization_config")

Note oMLX already ships omlx/custom_kernels/bonsai/ and omlx/patches/bonsai_t5_load.py, i.e. the Bonsai family is already known here — this is the missing member of that family. I am happy to open a PR with the patch module + a regression test if you want it; just tell me whether you prefer the bundled-loader-with-hash-pin form or a vendored copy.