knnCollector is faster than topNCollector

when search knn with text filter, it uses EligibleCollector first, and does knn query by filtered result from EligibleCollector, but when only search text with the same text condition, it uses TopNCollector, which is much slower. So how can i make text search faster than KNNWithFilter SearchRequest?