[Feature Request] Tracking first-class embedding serving path in MLC-LLM

Feature

Introduce a first-class embedding serving/runtime path in MLC LLM.

The current /v1/embeddings API support is already in place, but the execution path is still relatively lightweight and does not yet have a dedicated embedding runtime/serving architecture underneath. I would like to propose making embedding a more explicit and general path inside MLC LLM, while keeping the normal chat model serving path unchanged.

The goal is to support embedding models more naturally inside MLC LLM, instead of treating embedding inference mainly as a thin serving wrapper on top of the existing runtime assumptions.

Motivation

I have been working on local agent-style systems recently, and that made me look more closely at the embedding path in MLC LLM.

For agent workloads, embedding is a core primitive for retrieval, memory, indexing, and ranking. In practice, embedding latency and throughput can matter just as much as generation performance. Part of why I want to build on MLC LLM is that it is a hardware-aware local serving stack, so it seems like a good place to support more optimized embedding serving as well.

Earlier, I contributed some of the initial /v1/embeddings support. At the time, because of time and engineering scope, I intentionally kept the implementation lightweight. That was enough to establish the API contract, but it did not yet introduce a task-specific embedding serving engine or a first-class embedding execution path.

From my current understanding, the main gap is no longer the API itself, but that embedding is still not modeled as a first-class serving/runtime path. This makes it harder to:

• support both encoder and decoder-only embedding models cleanly
• introduce optimizations that depend on embedding model structure
• add backend-specific optimizations in a general way later
• keep the embedding stack aligned with the broader serving/runtime architecture of MLC LLM

This proposal is intended to be aligned with the earlier tracking issue for generic sentence embedding models in MLCEngine: #2324.

Additional context

My current implementation plan is to break this into several smaller steps instead of trying to do everything in one PR:

1. Metadata abstraction
   Branch: feature/embedding/metadata-abstraction
   Make embedding traits explicit in metadata/config instead of relying on function-name-based detection.

2. Dedicated TVM-native embedding runtime
   Branch: feature/embedding/runtime-tvm-native
   Introduce a clearer embedding runtime boundary while keeping the existing TVM-native path as the default implementation.

3. First-class encoder embedding path
   Branch: feature/embedding/encoder-first-class
   Make encoder embedding models first-class, instead of relying on Python-side pooling/runtime glue.

4. First-class decoder-only embedding path (starting from qwen3-embedding)
   Branch: feature/embedding/decoder-first-class-qwen3
   Start with qwen3-embedding as the first concrete decoder-only embedding family, while keeping the overall design general.

5. Backend registry for future hardware-specific optimization
   Branch: feature/embedding/backend-registry
   Add a cleaner extension point for backend-specific optimization later.

A few important non-goals for the initial stages:

• no change to the /v1/embeddings API contract
• no disruption to the normal chat model serving path
• no requirement to solve every backend/hardware-specific optimization in the first step

If this direction sounds reasonable, I plan to use this issue as the main tracking thread and link the staged PRs back here.