From 8PM on 02/09/11 I am going to be onsite doing some work on a few machines.
VZ2 will be down while we troubleshoot a RAID controller issue and replace two faulty disks.
XN2 will be rebooted as the RAID controller has not detected a new disk correctly which was hot-swapped in.
If your VPS is currently down we appologise for the inconvenience, but its important we return your host machine to its fully working state with an Optimal RAID10 disk array.
Thanks
As per an email sent out to all customers last week, we are carrying out some network upgrades on Saturday 27th August at 10.00PM (GMT+1). This is to reconfigure our network with a Cisco HSRP setup for improved redundancy incase of any router issues.
There will be a short outage between 10-20 minutes affecting all of our UK servers, except XN15 and XN16 which are in different racks.
Updates will be posted here throughout.
21.50: We are now preparing for the upgrade
22.03: Upgrade has started. Network is now down
22.11: Conectivity is restored. All finished!
Two drives have failed in XN12 at the same time causing the RAID array to go offline and the system crash.
Update 1: We have forced the array back online and will attempt repair and recovery of data.
Update 2:The machine will not boot due to filesystem issues on the host filesystem. We are currently working to restore service ASAP.
Update 3:We have almost finished repairing the damage to / on the host machine and will be rebooting shortly.
Update 4: It seems there are more damaged files, going back into rescuemode to repair + replace.
Update 5: Attempts to repair the host filesystem has failed, the system boots but we are unable to login. We are going to backup VM configuration files and reinstall the machine. From there we will assess what data is available.
Update 6: Host OS is now reinstalled, we are going to check the RAID status and begin checking of VPS filesystems.
Update 7: VPS are being restored one by one.
A disk has failed in XN11 and caused the host root operating system / to go read only. Server is rebooting to restore service.
XN2 has crashed approx 8.42PM and is being checked.
Update 8.45PM: Currently waiting on getting a KVM attached to the machine.
Update 8.56PM: Onsite staff are currently working ont he machine
Update 9.03PM: The server is now up and VPS are booting. The RAID array on this node is currently verifying, please be patient and allow your VPS to boot up rather than just rebooting. I/O performance will be a little slow until the rebuild completes.
Thanks