Status Blog

We are currently facing a 5GB/s inbound DDoS attack, we are working to mitigate this and update will be provided here.

Update: We are still working on mitigating this attack, appologies for any inconvenience. Further updates will be provided as soon as possible.

Update 2: Connectivity has been restored to all but 1 server at the moment.

Update 3: Connectivity has now been restored to all servers. The remaining malicious traffic has terminated, and traffic through our network has returned to normal levels. We are very sorry for the amount of time it has taken to mitigate this, the largest timekiller was isolating multiple targets. Some customers will have had a longer amount of downtime/packet loss than others.

Following this outage we will continue to monitor the network vigilently for the next 48 hours, and we are also going to draw up our options for implementing additional filtering into our network to improve our resilienece to malicious traffic.

xn7 appears to be down although its responding to SSH. We are currently waiting for a KVM to be attached.

Update: The datacentre appears to be non-responsive at the moment. Appologies for the delay.

Update 2: This was due to the first disk in the RAID array failing, the server has now been rebooted and VPS are starting up. We will hot-swap the disk out shortly.

Update 3: We have hot-swapped out the bad disk and the array has now almost finished rebuilding.

We have been alerted by our monitoring systems that there is currently a high amount of packet loss in our UK Data Centre, we are in contact with the Data Centre and will update here once we have any more information.

Update 2: This has now been resolved and all traffic is flowing normally.

Update: This is a LINX (London Internet Exchange) Issue, it has just dropped over 500GB/s of traffic:

During a maintenance window, one of the DWDM units lost power, this caused temporary packet loss for approx 2-3 Minutes however everything is now online and normal service has been resumed.

xn9 is currently having issues, although the system & VPS are responding to ping it looks like I/O or RAID related issues. Just waiting on the DC to attach KVM now

Update 1: We have confirmed this is a RAID failure and we are currently working to restore the array.

Update 2: Waiting on DC again at the moment, sorry for the delay.

Update 3: Unfortunatley it appears the RAID array has collapsed and is unusable instead of just being degraded due to a bad disk. We are working through our options to restore the array.

Update 4: We have now re-assembled both sides of the RAID set and the system is now booted off a degraded RAID-10 volume, VPS are currently starting up. We will now proceed to inspect the integrity of the array and hot-swap out any suspect disks.

Update 5: Disks p1 and p2 make up the first half of the RAID-10 array (Or together  a RAID-1 set), disk p2 is bad and a read error caused the system to hang. The system is currently running off the bad disk, p2 while p1 rebuilds itself. Once this has been completed we will immediatley take p2 offline and replace the disk. At this point the array is fragile however we have no reason to believe the current rebuild will not complete successfully. Many thanks for your patience.

Update 6: Full redundancy and performance has now been restored to the array.

We are currently facing a 1+GB/s DDoS Attack on our systems, we are working to mitigate this as quickly as possible and all update will be provided here.

Update 2: Traffic is now flowing normally again and now consider this resolved.

Update: This appears to be under control, we have nulled the IPs in question and are monitoring the situation very closely.