Follow-Up: It looks like the rapid series of disconnects coinciding with a few months of running each server is probably coincidental and just served to reveal the actual problem. The reason it failed to reconnect is almost certainly due to the AliveInterval values (kasperd's answer). Using the ExitOnForwardFailure option should allow the timeout to occur properly before reconnecting, which should solve the problem in most cases. MadHatter's suggestion (the kill script) is probably the best way to make sure that the tunnel can reconnect even if everything else fails.
I have a server (A) behind a firewall that initiates a reverse tunnel on several ports to a small DigitalOcean VPS (B) so I can connect to A via B's IP address. The tunnel has been working consistently for about 3 months, but has suddenly failed four times in the last 24 hours. The same thing happened a while back on another VPS provider - months of perfect operation, then suddenly multiple rapid failures.
I have a script on machine A that automatically executes the tunnel command (ssh -R *:X:localhost:X address_of_B for each port X) but when it executes, it says Warning: remote port forwarding failed for listen port X.
Going into the sshd /var/log/secure on the server shows these errors:
bind: Address already in use
error: bind: Address already in use
error: channel_setup_fwd_listener: cannot listen to port: X
Solving requires rebooting the VPS. Until then, all attempts to reconnect give the "remote port forwarding failed" message and will not work. It's now to the point where the tunnel only lasts about 4 hours before stopping.
Nothing has changed on the VPS, and it is a single-use, single user machine that only serves as the reverse tunnel endpoint. It's running OpenSSH_5.3p1 on CentOS 6.5. It seems that sshd is not closing the ports on its end when the connection is lost. I'm at a loss to explain why, or why it would suddenly happen now after months of nearly perfect operation.
To clarify, I first need to figure out why sshd refuses to listen on the ports after the tunnel fails, which seems to be caused by sshd leaving the ports open and never closing them. That seems to be the main problem. I'm just not sure what would cause it to behave this way after months of behaving as I expect (i.e. closing the ports right away and allowing the script to reconnect).
