How to achieve whitelist mode?

Is it possible to invert how it normally works and trap all ports except for real ssh (and other tools)? How? And if yes, is it good idea?

I noticed that very few login attempts go to 22. Instead they go to random high ports. So trapping only one port which is never targeted seems not very useful