关于 InternVL3-9B 架构的说明:语言模型代码库 (InternLM2 与 InternLM3) 和 Wqkv 布局

Dear InternVL Team, First, thank you for releasing the impressive InternVL3 series. Our team is currently developing a research framework for Chain-of-Thought (CoT) Vector extraction, which involves high-precision hooks into the internal Attention mechanisms of the model to analyze activation shifts. Observation (观察结果) The official documentation states that InternVL3-9B uses InternLM3-8B-Instruct as its language backbone. However, upon inspecting the Hugging Face repository (OpenGVLab/InternVL3-9B), we noticed: The repository contains modeling_internlm2.py instead of modeling_internlm3.py. The model configuration uses trust_remote_code=True, loading the local InternLM2 codebase. The InternLM2Attention implementation uses a merged wqkv linear layer with an interleaved GQA layout, whereas the standalone internlm3-8b-instruct repo uses independent q_proj, k_proj, and v_proj layers. Hypothesis (假设) We hypothesize that for InternVL3-9B, the team performed a weight fusion (merging the independent weights of InternLM3 into the wqkv matrix format) to maintain compatibility with the existing InternVL framework codebase. ❓ Requests for Confirmation (申请确认) To ensure our custom Hook logic is mathematically correct, could you please confirm the following: Architecture Truth: Does InternVL3-9B strictly follow the modeling_internlm2.py implementation (using the merged wqkv layer) during runtime? Wqkv Slicing Logic: In InternVL3-9B, is the wqkv tensor layout still interleaved (as defined in InternLM2) for Grouped Query Attention? (e.g., slicing into [q1, q2, ..., q_group, k, v] for each head group). Weight Source: Are the underlying weights indeed distilled/transferred from the InternLM3-8B-Instruct pre-training? Why this matters to us: Our code must manually slice the wqkv output to inject specific task vectors. If we assume independent q/k/v projections for an InternLM3 backbone but the model actually runs on a fused wqkv (InternLM2 code), our research results will be invalid.