While we're doing a hostkey reorg, store port numbers as well