disable pull to refresh when item clicking or touching?

Hi, i have some issue. when i click recyclerview item, pull to refresh will be pull down if my finger try to click and swipe down a little bit, that makes pull to refresh will cancel my item click event. This can happen when im of the top of recyclerview and checkCanDoRefresh return true.

how can i avoid this? thank you.