Due to a new SSHD rootkit which is circulating we are ensuring all of our systems are fully updated and booted into the latest kernels. At the moment we are not aware of any of our VPS host machines or managed customers being compromised but it’s good preventative maintenance nonetheless.
There may be a brief outage while your host machine is restarted.
Apologies for the lack of notice on this.
Thanks
We have an inbound 1Gbps+ DDoS attack against a VPS on this machine, we are working on isolating the target to restore service.
Update: The target IP has been nullrouted and service restored. If you are seeing any further connectivity issues please open a ticket.
There appears to be none or partial connectivity to all of our equipment at the datacentre currently depending on your ISP and location, it looks like it could be an issue with one of our main carriers – Level 3 from doing a quick traceroute on my mobile but will get more details shortly
Thanks
Update: The issue appears to have resolved itself as soon as I finished writing this post! If you have any further issues please let us know.
Message recieved from our datacentre:
We are currently working with one of our upstream providers, who have notified us about a network issue as a result of them making a configuration change. We had not been notified of the change.
Our core networking teams are doing what they can to mitigate the issue and looking into re-routing traffic. This will be performed in the next few minutes, when connectivity should improve.
We will keep you updated on this matter and sincerely apologise for any inconvenience this is causing.
If you continue to experience issues please let our support teams know and we will investigate as a matter of urgency.
xn17 currently has high packet loss because another server in the rack is under a DDoS attack. This is a shared colo rack so the datacentre is working on it.
Update: Connectivity is restored
Update2: Just got back from the datacentre, it looks like the motherboard failed or at least ACPI functionailty had completely stopped working. We’ve moved the drives into a different chassis (inc motherboard, psu, raid card etc) and service is now restored.
Thanks
The powersupply in XN3 failed this morning and has been replaced but the server still has startup issues.
We are waiting on an update from the datacentre but will probably go to site shortly and swap the drives into another chassis.
Further updates will follow.
cp1 has crashed and is currently doing an automated FSCK of / to correct any fileystem errors. Its currently at 13% so ETA is about 20 minutes to service restoration.
Appologies for the inconvenience.
The issues on xn18 are back, as the load on the machine is too high to get in via SSH the DC is nullrouting IP’s individually until we identify the cause for these issues.
Updates will follow shortly.
A VPS on XN18 is currently being attacked and we are working on mitigating the traffic. If your VPS is down please do not try and reboot it.
Update We believe this is now resolved but will continue to monitor.
Today we are upgrading our Support desk to the latest version of Kayako fusion, and once the conversion is completed will be moving our main websites, billing system and the support desk to a new server.
If you have any urgent support issues please email tempsupport@pcsmarthosting.net while the helpdesk is offline
Thanks
xn3 is currently having issues with the RAID array and is failing to boot, its looking like multiple drive failures in one half of the RAID10 set. This also hosts the VPS for our website and helpdesk so that’s currently offline.
We are working on this and updates will be posted here shortly.
Update: We have recovered the array and it’s now rebuilding onto some fresh drives. Disk performance will be poor for a few hours while it rebuilds and restores full redundancy to the array. If your VPS is currently down, its booting up and may be doing an automated filesystem check.
Thanks