In my experience ssh has a slightly irksome habit of not exiting cleanly if 'something' is still running on the remote system. E.g. started in the background. You can reproduce this by:
ssh <server>
while true; do sleep 60; done&
exit
Your ssh will log out, but won't actually close the session - until the remote process exits (which it won't, because it's a 'while true' loop). It may be something similar is happening - your session has a 'stuck' process that's being spawned by ssh. The port remains in use, and therefore it cannot be re-used by your local process.
