[FEATURE] Video (vision) input support for gemini_mllm_python — regression from legacy gemini_v2v_python

Description

The legacy TEN-Agent extension gemini_v2v_python supported realtime vision: it consumed RTC video_frame input, converted frames to JPEG, and forwarded them to the Gemini Live API alongside audio (this was a flagship demo — "realtime vision and realtime screenshare detection").

The current generation gemini_mllm_python (v0.2.1) appears to be audio-only:

• extension.py only implements the audio ingress path (send_realtime_input(audio=...)); there is no video_frame handling.
• The shared MLLM interface (ten_ai_base/api/mllm-interface.json) only defines audio_frame_in / audio_frame_out — no video input channel. The same applies to openai_mllm_python and glm_mllm_python.

Gemini Live natively supports continuous video input (processed at 1 fps server-side), so this looks like a capability regression introduced by the v2v → mllm rewrite rather than a provider limitation.

Use case

We are building a realtime voice + video call agent on TEN (agora_rtc + gemini_mllm_python). Voice works out of the box, but the camera track cannot reach Gemini because the MLLM interface has no video path.

Request

1. Add a video input channel to the MLLM interface (ten_ai_base);
2. Restore video frame forwarding in gemini_mllm_python, similar to what the legacy gemini_v2v_python did (see PR #474).

Questions

• Is video support for the mllm extensions already on the roadmap?
• If not, would a PR be welcome? Guidance on how you'd like the ten_ai_base interface change to be split would help.

Severity

Major

Additional Information

No response