flaky test TestIssue54055

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

     index_lookup_join_test.go:201: 
        	Error Trace:	pkg/executor/join/index_lookup_join_test.go:201
        	Error:      	Expected value not to be nil.
        	Test:       	TestIssue54055 

https://tiprow.hawkingrei.com/view/gs/pingcapprow/pr-logs/pull/pingcap_tidb/58612/fast_test_tiprow/1873652532308348928#1:build-log.txt%3A2305

4. What is your TiDB version? (Required)