TopkRetriever crashes on transformers 5.x: tokenizer.encode_plus was removed

Author: rrrxxx0510Created Sep 8, 2026Updated Sep 8, 2026

encode_plus was removed in transformers 5.x, but DatasetEncoder.init_dataset still calls it (opencompass/openicl/icl_dataset_reader.py):

python
tokenized_data = self.tokenizer.encode_plus(data, truncation=True, return_tensors='pt', verbose=False)

Any run using TopkRetriever with a recent transformers dies with AttributeError: Qwen2Tokenizer has no attribute 'encode_plus'. Same class of breakage as #2573 (batch_encode_plus in the HF model backends).

The direct call API is a drop-in replacement — for a single string it returns the same [1, L] tensors, and verbose is still accepted. I have a fix ready and can send a PR.

Source: open-compass/opencompass