Categories
Outages

[Resolved] xn9 Issue

xn9 is currently having issues, although the system & VPS are responding to ping it looks like I/O or RAID related issues. Just waiting on the DC to attach KVM now

Update 1: We have confirmed this is a RAID failure and we are currently working to restore the array.

Update 2: Waiting on DC again at the moment, sorry for the delay.

Update 3: Unfortunatley it appears the RAID array has collapsed and is unusable instead of just being degraded due to a bad disk. We are working through our options to restore the array.

Update 4: We have now re-assembled both sides of the RAID set and the system is now booted off a degraded RAID-10 volume, VPS are currently starting up. We will now proceed to inspect the integrity of the array and hot-swap out any suspect disks.

Update 5: Disks p1 and p2 make up the first half of the RAID-10 array (Or together  a RAID-1 set), disk p2 is bad and a read error caused the system to hang. The system is currently running off the bad disk, p2 while p1 rebuilds itself. Once this has been completed we will immediatley take p2 offline and replace the disk. At this point the array is fragile however we have no reason to believe the current rebuild will not complete successfully. Many thanks for your patience.

Update 6: Full redundancy and performance has now been restored to the array.