Categories
Outages

[Resolved] XN12 Issue

Two drives have failed in XN12 at the same time causing the RAID array to go offline and the system crash.

Update 1: We have forced the array back online and will attempt repair and recovery of data.

Update 2:The machine will not boot due to filesystem issues on the host filesystem. We are currently working to restore service ASAP.

Update 3:We have almost finished repairing the damage to / on the host machine and will be rebooting shortly.

Update 4: It seems there are more damaged files, going back into rescuemode to repair + replace.

Update 5: Attempts to repair the host filesystem has failed, the system boots but we are unable to login. We are going to backup VM configuration files and reinstall the machine. From there we will assess what data is available.

Update 6: Host OS is now reinstalled, we are going to check the RAID status and begin checking of VPS filesystems.

Update 7: VPS are being restored one by one.