It is trivial if you actually know what the port number is and you are only talking to one server and the non-22 port is static. Now imagine you have code that is connecting to thousands of different servers on different ports and they can change at any time if someone who owns one of those servers decides to rebuild their machine and changes the SSH port. Nothing is trivial at scale. That's not say that is always a valid consideration but standards and conventions exist for a reason.
