Server problems
Phoo.starkast.net stopped responding to requests the night between Sunday and Monday. Don’t really know what happened, when I woke up my SSH-sessions were dead and I couldn’t reconnect. HTTP and IMAP was dead too. I didn’t have time to look into it at the time, as I had school on the morning and work later.
On Monday dentarg mailed the administrator of the firewall that sits in front of the server, explaining the situation. He checked the firewall logs and said that Phoo was flooding some packets and the firewall thought it was an internal DoS and blocked it. He then added some firewall rules that allowed my IP to connect without matching any firewall rules, but I couldn’t initiate a SSH-session. When I looked at the traffic, Phoo only managed to respond with a SYN+ACK to my SYN packet, and then kept quiet. The TCP/IP-stack on Phoo seems to be working, but the sshd can’t respond. The weird thing is that my IRC-client was still connected to the IRC-networks on the morning, and Luddes IRC-client was connected for 48h and then got a “Ping timeout”. I’ve also noticed that the DNS-server is still responding.
This is a bit annoying, since I do my developing on Phoo, and some of us has our mail there. Regarding mail, our other server Beaver.starkast.net is a backup MX for all the domains, and I’ve changed Postfix so that all the aliases that forwards to external addresses can do so right away from Beaver. My two main e-mail accounts are forwarding all new mail to my Gmail-account now.
One thing I haven’t figured out with Postfix, is how to re-run all the mails that’s in the relay queue. New mails will be forwarded to my Gmail-account, but the mail recieved before I added the virtual aliases still tries to be delivered to Phoo since the mail has already been added to the relay queue. It would be great if I could pull out the mails and send it to the pickup process (IIRC).
We’re probably forced to visit the co-location facility and check what’s wrong. We could pay 800 SEK for someone at the facility to press the reset button on Phoo, but since we don’t know what’s wrong, it just feels like a waste of money.
Wonder if using watchdog would have helped.
Update
We have decided not to pay 800 SEK and reboot Phoo, we will try and visit Phoo in the co-location facility. When the visit will take place we do not know, probably in a few weeks from now.