server/: Introduce accessor functions for reading and writing port numbers.

There are still some raw accesses to port numbers outside of address-
family switches, but these are all concerned with conversions between
address structures and strings, and this will be fixed in a later
change.