vz1 is currently unresponsive/down and is being investigated
Update: Looks like its just under high load, waiting for onsite hands to connect KVM.
Update 2: One particular VPS was maxing out CPU causing high load. This same VPS has caused issues twice before, and has now been removed from the server. Appologies for the inconvenience.
Approx downtime: 10min
cp1 our main shared/reseller cPanel system has crashed and is rebooting.
Its currently doing an automatic FSCK of the filesystem as it’s last reboot was over 250 days ago. Will be up soon.
Update: Server is now up and load is slowly coming down. Will keep an eye on this for the next 24hrs.
xn5 appears to be down and is being checked
Update: Looks like a VPS is under attack and is causing high packet loss. Working to get this under control ASAP.
Update 2: Service restored and we will continue to monitor. This was actually caused by a FreeBSD VM which had run out of disk space and was maxing out CPU and disk, very unusual!
Downtime: Approx 4mins
xn2 appears to be down and is being checked.
The server is now back and VPS are starting
One of the new Western Digital drives fitted only last week has failed in VZ2 and caused the host filesystem to go read-only. We are working on this and service will be restored ASAP.
Update: We have been waiting over 15mins so far and no response from our datacenter (iomart). Further updates will follow…
Update 2: We now have a USB DVD drive connected at last and are working to restore the machine.
Update 3: Since /lib is missing and a few other parts of the OS are resolving to the wrong files we are going to take the fastest route which is to reload the OS on the host. Customer data and configs are on a seperate partition so your data should be safe. Reload is currently 50% so ETA around 20 minutes and we will be up.
Update 4: OS reload complete. We are now doing a quick OS update, installing OpenVZ, rebooting and getting VPS up. SolusVM will be restored shortly after.
Update 5: Server is now booting into OpenVZ
Update 6: VPS are now booting up one by one and we are restoring SolusVM access to this node.
Update 7: SolusVM now restored, VPS are booting up.
Two drives have failed in XN12 at the same time causing the RAID array to go offline and the system crash.
Update 1: We have forced the array back online and will attempt repair and recovery of data.
Update 2:The machine will not boot due to filesystem issues on the host filesystem. We are currently working to restore service ASAP.
Update 3:We have almost finished repairing the damage to / on the host machine and will be rebooting shortly.
Update 4: It seems there are more damaged files, going back into rescuemode to repair + replace.
Update 5: Attempts to repair the host filesystem has failed, the system boots but we are unable to login. We are going to backup VM configuration files and reinstall the machine. From there we will assess what data is available.
Update 6: Host OS is now reinstalled, we are going to check the RAID status and begin checking of VPS filesystems.
Update 7: VPS are being restored one by one.
A disk has failed in XN11 and caused the host root operating system / to go read only. Server is rebooting to restore service.
XN2 has crashed approx 8.42PM and is being checked.
Update 8.45PM: Currently waiting on getting a KVM attached to the machine.
Update 8.56PM: Onsite staff are currently working ont he machine
Update 9.03PM: The server is now up and VPS are booting. The RAID array on this node is currently verifying, please be patient and allow your VPS to boot up rather than just rebooting. I/O performance will be a little slow until the rebuild completes.
Thanks
A drive has failed in XN11, has been replaced and is booting up now.
Update: Server has failed to come back up. Sorry about this folks, waiting for our KVM to be moved to it now so we can see what the hold up is.
Update 2: There are some filesystem issues which we are repairing, service will be restored ASAP
Update 3: The machine is now online and all VPS are running. We are going through each VPS to complete a FSCK of their individual filesystems to ensure they are clean. If your VPS goes down, please do not try to restart it. It will be restarted automatically once the FSCK completes. Thanks
We are currently seeing some intermittent packet loss affecting all of our UK infrastructure. As soon we we have more information from the datacentre we will post it here.
Appologies for the inconvenience.
Update: Although the outage was only a few minutes, we are seeing some sporadic DNS issues which are being worked on.
Update: Total downtime was less than 5 minutes, this was caused by some issues with one of our datacenter’s upstream providers.