We have just been notified of an issue at LINX (London Internet Exchange), they have just dropped 600GB/s of traffic so UK Clients may notice some interuptions to service however this is completely outside of our control.
Update: Traffic flow appears to have returned to normal again now.
We are currently facing an issue with CP1, the machine has loaded up on the MySQL Service and we are looking into it now.
Update: This has now been resolved, apologies for the short MySQL Downtime.
We currently have malicious network traffic incoming affecting connectivity to the below servers:
xn4, xn5, xn6, xn8
It seems only 109.169.51.0/24 is targeted, systems on other subnets are fine.
Update: After some process of elimination, this looks like a possible malfunction with the NIC on xn4, which is causing a traffic storm on the subnet. XN5, XN6 + XN8 are restored.
Update 2: We are going to move to the 2nd NIC on XN4 with a new cable + switchport.
Update 3: The traffic has returned to xn5/6/7 and we are working to mitigate.
Update 4: XN5/Xn6/Xn8 restored. XN4 being worked on
Update 5: All systems are now up and we will be following this up with our datacentre in the morning.
We are currently facing a 5GB/s inbound DDoS attack, we are working to mitigate this and update will be provided here.
Update: We are still working on mitigating this attack, appologies for any inconvenience. Further updates will be provided as soon as possible.
Update 2: Connectivity has been restored to all but 1 server at the moment.
Update 3: Connectivity has now been restored to all servers. The remaining malicious traffic has terminated, and traffic through our network has returned to normal levels. We are very sorry for the amount of time it has taken to mitigate this, the largest timekiller was isolating multiple targets. Some customers will have had a longer amount of downtime/packet loss than others.
Following this outage we will continue to monitor the network vigilently for the next 48 hours, and we are also going to draw up our options for implementing additional filtering into our network to improve our resilienece to malicious traffic.
We are currently not experiencing any issues.
xn7 appears to be down although its responding to SSH. We are currently waiting for a KVM to be attached.
Update: The datacentre appears to be non-responsive at the moment. Appologies for the delay.
Update 2: This was due to the first disk in the RAID array failing, the server has now been rebooted and VPS are starting up. We will hot-swap the disk out shortly.
Update 3: We have hot-swapped out the bad disk and the array has now almost finished rebuilding.
We have been alerted by our monitoring systems that there is currently a high amount of packet loss in our UK Data Centre, we are in contact with the Data Centre and will update here once we have any more information.
Update 2: This has now been resolved and all traffic is flowing normally.
Update: This is a LINX (London Internet Exchange) Issue, it has just dropped over 500GB/s of traffic:
During a maintenance window, one of the DWDM units lost power, this caused temporary packet loss for approx 2-3 Minutes however everything is now online and normal service has been resumed.