[Bug]: RichPromptTemplate crashes on embedded audio/video files and corrupts embedded documents
Bug Description
RichPromptTemplate.format_messages converts banks media blocks by stuffing input_*.data into the llama_index block's url field:
elif bank_block.type == BanksContentBlockType.audio:
llama_blocks.append(AudioBlock(url=bank_block.input_audio.data))
elif bank_block.type == BanksContentBlockType.video:
llama_blocks.append(VideoBlock(url=bank_block.input_video.data))
elif bank_block.type == BanksContentBlockType.document:
llama_blocks.append(DocumentBlock(url=bank_block.input_document.data))
But banks' | audio, | video, and | document filters store embedded files as raw base64 in data (only URL inputs are kept as URLs). Two failure modes:
- Audio and video files crash.
AudioBlock.url/VideoBlock.urlareAnyUrl, so raw base64 fails pydantic validation:ValidationError: ... Input should be a valid URL, relative URL without a base. Any chat template embedding an audio or video file is unusable. - Document files are silently corrupted.
DocumentBlock.urlis a plain string, so the base64 blob "succeeds" and every downstream consumer receives a document block whoseurlis a few kilobytes of base64 instead of a URL or the document bytes.
(Images are unaffected: banks embeds local images as data: URLs, which are valid URLs.)
Proposed fix: convert by payload shape - payloads that are URLs keep going to url, base64 payloads are decoded into the block's bytes field (audio / video / data) with the format banks detected carried over (format, video_mimetype, document_mimetype). I have the patch and regression tests ready - happy to send the PR.
Version
llama-index-core 0.14.24 (also verified on current main, fd4a517); banks 2.1.1
Steps to Reproduce
import os
from llama_index.core.prompts import RichPromptTemplate
os.environ["BANKS_MEDIA_ROOT"] = "/tmp" # let banks read the test file
open("/tmp/clip.mp3", "wb").write(b"\xff\xf3" + b"\x00" * 100)
t = RichPromptTemplate("{% chat role='user' %}Describe this: {{ p | audio }}{% endchat %}")
t.format_messages(p="/tmp/clip.mp3")
Output:
pydantic_core._pydantic_core.ValidationError: 1 validation error for AudioBlock
url
Input should be a valid URL, relative URL without a base [type=url_parsing, input_value='//MAAAAAAAAAAAAAAAAAAAA...', input_type=str]
Same crash for {{ p | video }} with an mp4. For {{ p | document }} with a pdf there is no crash, but the resulting block is DocumentBlock(url='JVBERi0xLjQK...') - base64 in the url field.
Source: run-llama/llama_index