XN22 is currently down due to filesystem/RAID issues and is being worked on. We apologise for the delay on this caused by a number of factors including a faulty KVM and response times with our datacenter.
Further updates will follow shortly.
Update @ 21:15: VPS Should now be booting back up
Our billing and support systems are currently undergoing scheduled maintenance and will be back online shortly.
0315 GMT – This has now been completed on time.
As per the email sent yesterday, at approx 9PM (GMT+1) tonight we will be powering down, and relocating these servers due to a potential PDU issue. Downtime will be kept to a minimum but we expect approx 20-30mins.
Updates will be posted here as we carry out the work. Thanks
8.50PM – VPS are shutting down.
8.55PM – VPS are now shutdown and work will start shortly.
9.16PM – Work still in progress
9.36PM – VPS are booting up now.
We will be carrying out scheduled audits and node reboots for the below host machines on Friday 27th June starting approx 8PM BST. We estimate downtime for VPS will be between 10-20 minutes.
This affects any customers with Xen VPS on the following host machines:
Further updates will be posted here during the maintenance task.
Thank You.
newvz / vz2 is currently failing to boot and hanging initialising device-mapper, we are working on it and further updates will be posted soon.
Update: Repaired and also updated to new OpenVZ kernel at the same time.
xn5 is currently showing some service issues although it’s still pinging. Getting the datacentre to check.
Once again there appears to be a huge DDoS going on at the datacentre which is affecting all of our equipment even going through different router clusters.
Further updates will follow shortly.
Update: This appears to be fixed though we’ve not heard anything from the DC yet.
At approximately 18:40 GMT our network was targeted by a large scale 50GB/s+ DDoS attack which caused a loss of connectivity for some of our clients, the attack has now been mitigated and we are remaining vigilant and monitoring all network traffic very carefully for the next 24 hours however we do not suspect that the attack will return.
All services should be now back up and running, if there are any still offline please open a ticket and we will take a look into this as a priority.
web1.cpservers.info is currently having some intermittent timeout issues due to a sophisticated peice of malware which goes after the Apache binary.
This malware takes control of httpd and inserts code to interrupt or redirect requests, for the last few days we have been mitigating the symptoms as best as possible.
We believe this has come across on a recent migration from another server, and as it’s incredibly difficult to pinpoint we will be reinstalling the system to ensure its fully removed. There is no risk to the security of individual accounts or the integrity of your data.
21/01/14 20:15 – We started generating a new backup set containing all account data at approx 6.30PM, and there are now only around 15% of accounts remaining. Assuming this completes within the next 45-60mins we will be proceeding with the reinstall.
21/01/14 20:47 – Backup completed, reinstall in progress.
21/01/14 21:26 – cPanel install is now well underway, should be approx 45mins then we can begin account restoration.
21/01/14 22:37 – The server has now been fully configured, and account restorations have been running for around 30mins. It will take several hours for all accounts to restore.
22/01/14 18:09 – Restores have almost completed, unfortunately this has taken a while as cPanel’s new backup system will only restore accounts one by one on a queue based system. The majority of clients should already be up at this point.
Restoration has been completed.
All servers in the 95.154.208.0/24 subnet are currently experiencing packet loss – even those in different racks, so this appears to be a routing issue with our upstream provider.
We have logged a ticket with them and updates will follow shortly.
Update: This turned out to be a client under a large DDoS attack, his account has now been closed due to suspected illegal activity.