Even better don't use default ports. Or have a second ssh that doesn't except any IP address at port 22 and than have your non-standard port with keys and limited user names and if possible a white list of your IP addresses.
If you do this, keep it in the privileged range (< 1024) or you run the risk of your ssh server crashing and some malicious normal user binds to your unprivileged port with a fake sshd and grabs your root password.
Not using default ports will mildly confuse automated scans and do absolutely nothing to a determined attacker. Or somebody with nmap, which is not the same thing.
If you're whitelisting IPs, you may as well run it on port 22.
No it makes it harder and more of a pain. Trust me I have a friend who loves breaking into my personal server. That one trick two ssh running on different ports screwed with him for a long, long time. He is a genius of a hacker and has been doing it for a living for years. When he finally got in he was so pissed that threw him.
You are describing an anecdotal instance of a person whose capabilities are not established being thrown by something that nmap will catch on a normal scan.
Color me skeptical. I shall decline to "trust you."
Not my competence it his competence I trust and I got him good with that one since it never occurred to him that one stupid trick messed with him for so long. Lie 5 minutes a month.
One small benefit of using a non default port is that it keeps down the noise from automated scans. So any "real" suspicious activity will now stand out as it is not drowned out by the noise anymore.
