July 2016 Server Failure
Posted: Fri Jul 22, 2016 11:31 am
There was a hardware failure with the old server that ran the main site, forums, wiki, server list, etc. I'm on a new server now, but I had to roll the system back to the 18th as that was the last successful backup, so there will be some minimal data loss.
On Friday the 15th, I noticed the server was performing really poorly, so I started to investigate. I was watching iostat numbers and the second hard drive was showing high wait times for reads and writes. There were also some errors about that drive in the kernel logs. I think that drive ended up falling off the RAID array on the 17th. But then the other drive started showing high wait times and the same kinds of errors in the kernel log. So, either both drives started to fail at the same time, or there was something deeper in the system failing (power supply, motherboard, SATA backplane, etc).
If anyone notices anything wrong now that I've migrated stuff, let me know.
On Friday the 15th, I noticed the server was performing really poorly, so I started to investigate. I was watching iostat numbers and the second hard drive was showing high wait times for reads and writes. There were also some errors about that drive in the kernel logs. I think that drive ended up falling off the RAID array on the 17th. But then the other drive started showing high wait times and the same kinds of errors in the kernel log. So, either both drives started to fail at the same time, or there was something deeper in the system failing (power supply, motherboard, SATA backplane, etc).
If anyone notices anything wrong now that I've migrated stuff, let me know.