Categories
Outages

[Complete] XN7 Issues

XN7 is currently having issues affecting some VPS customers in the 95.154.207.xx range. It’s very rare this happens but both drives in one half of the RAID10 array are showing signs of failure and its causing intermittent filesystem issues.

We are working to resolve this as quickly as possible, while maintaining integrity of all customer data.

Update: Our datacentre is being very slow doing anything for us at the moment, appologies for the delay.
Update 2: We are still waiting on our datacentre…
Update 3: It looks like the motherboard may have failed in the machine. We do have a spare onsite but are confirming this now.
Update 4: System is back online and VPS are booting up. The RAID array is rebuilding and we will continue to monitor this. Hopefully the rebuild completes and we can then swap out the failing drives immediatley.

Update 5: The rebuild has failed. Server has been restored for now with the rebuild stopped. We are going to shortly backup all data, put in new disks and reload this server.

Update 6: Senior staff will begin maintainence on this server at the datacentre, starting approximatley 10.00PM (GMT+1).

Update 7: 1.53AM. 12 out of the 18 VPS on this node have now been backed up. This gives us a rough ETA that the backing up will be completed at 4AM in around 2 hours time.

Update 8: 3.21AM. The last VPS Is backing up now, then we will reload the machine.

Update 9: We are restoring data

Update 10: We are about to boot all VM’s

XN7 has now been restored and all VPS are booting up. We will continue to monitor closely.