xn11 has crashed and is being checked
Update: A drive has failed in the RAID10 array, unfortunatley the RAID card was unable to handle the transfer resulting in the filesystem going to read only mode
Update 2: A replacement drive has been fitted, xn11 is now back online and vps are starting up one by one.
Please expect some reduction in I/O performance for the next few hours while the RAID array rebuilds to an optimal state.
Appologies for the inconvenience.
There is currently a critical network outage affecting all of our equipment in Spectrum House. We are in contact with our Upstream now and further updates will follow.
Update: Connectivity has been restored, downtime less than 5 minutes. This appears to have been a blip from one of the Core Routers.
RFO From Our Upstream: On Friday 16/07/10 our monitoring systems detected a connectivity problem affecting elements of the network. The issue was quickly escalated and once traced to certain malicious activity, senior network engineers immediately took action to halt the offending traffic.
xn12 and xn13 are currently experiencing stability issues and are being worked on.
xn12 Update: This machine appears to be suffering from hardware issues, due to a number of recent crashes. We will be replacing the RAM in due course and customers will be notified by email in advance of this planned maintainence task.
xn13 Update: This has been isolated to a particular VPS which has been suspended pending investigation.
Our monitoring has just picked up that xn14 has gone down. This is being checked
Update: We are waiting for some onsite hands.
Update 2: Ping requests are intermittently coming through, its looking like a network based attack against the machine or more likely a VPS.
Update 3: Connectivity has been restored to the machine and all VPS, we are now investigating further.
We are currently aware of issues with the XN12 Server, we are working with onsite staff to get this resolved as quick as possible.
Updates will follow here when we have them.
Update @ 11:51 BST – This machine is now back online, it will be monitored very closely over the next few hours and a full investigation into the cause will follow.
Update @ 11:44 BST – This is looking like a hardware failure, its not confirmed yet however if so we have onsite stock of parts should the worst happen.
xn12 appears to have crashed about 5 minutes ago. We are just waiting for some remote hands and will provide updates shortly.
Update: The server is now up and VPS are starting up.
We are currently facing a network outage in our UK DataCentre, once we have more information it will be posted here.
Update @ 01:49 am – At approximately 00:20 a network issue was detected affecting connectivity to the North cluster.
Our on call engineers were contacted and the problem has been resolved. Service was restored at approximately 01:35. Unfortunately we do not currently have any further details available at this time but will issue them as and when they become available.
We would like to express our sincere apologies for the inconvenience this will have caused. If you are experiencing any problems as a result of this please do not hesitate to contact our support team immediately and we will look into this for you.
Update @ 01:35 am – Connectivity has just come back, still awaiting any news as to what caused this.
Update @ 01:13 am – Just a quick update to say that we still cant get in touch with the DataCentre by any means, we are continuing to call them but as of yet all phone lines are completely dead.
cp1 has crashed (hence why we are upgrading) and is being checked.
We will update you ASAP
Update: The machine is doing an automatic forced FSCK on the filesystem. It’s already 43% complete so we will let this run its course.
Sincere appologies for the inconvenience caused, it couldnt happen at worse timing.
cp1 is currently having high load issues which means some customers might see their services down.
We are working to restore service ASAP
Additional Info: The current CP1 will be retiring in the next 2 weeks, as we have a new replacement machine on order based on the latest Xeon Woodcrest platform to put a stop to these load issues. We really appreciate your patience.
Update: We are currently still waiting on on the DC for our KVM to be moved. Appologies for the delay.
Update 2: The machine is failing to boot, we are waiting for further information from onsite staff, on-call staff have been notified and are ready to go to site if required.
Update 3: Things are now back under control and the load will return to normal shortly.
xn9 appears to be suffering some filesystem issues. Currently getting some remote eyes.
Update: For some reason disk 3 decided it would belong to a new array, and caused the filesystem to go read only. This has been corrected and the RAID is rebuilding. We will continue to monitor.