A drive has failed in the RAID-10 array on vz1 and needs to be replaced.
20.35: Drive has been hot-swapped and the array is rebuilding. Once this completes we will reboot the machine.
00.20: The server is now restarting
00:30: Automatic forced FSCK of /vz is now running and 17% complete.
00:40: Now 31% complete
00:50: Now 74% complete
01:00: FSCK has finished and passed. Now rebooting back into OpenVZ kernel and VM’s will boot up.
Our datacentre, RapidSwitch/iomart has taken the decision to relocate all full rack customers into a new, more resilient area of the datacentre. As mentioned on emails sent out earlier in the week, this means we will be relocating all of our servers into new racks.
This maintainence will be starting at midnight (00:00 British Summer Time, GMT+1) on Thursday 21/04/11 and we aim to be completed before start of the business day.
Updates will be provided here as often as possible.
10.10 – All servers are online and we are working through any remaining issues now
09.50 – Just arrived back and it looks like XN8 has lost network connectivity, investigating now.
08:06 – This has now been completed.
00:00 – Work has now begun.
20:18 – The new racks are now fully cabled up and ready to go, we will now be taking a well earned break and returning to start the move at 00:00 BST (3hrs 22 Minutes from now).
17.37 – The preparation and Cabling of the new racks is almost complete. We are finishing up soon and will then be returning at midnight 00:00 to begin migration of hardware
We are currently aware of a MysQL issue on CP1 our shared/reseller server which occurs when the daily backups run at around 4AM in the morning. It is normal for MySQL databse tables to lock while a particular database is being backed up, however we have recieved reports of MySQL being inaccessible for extended amounts of time during the backup cycle.
We are working diligently to resolve this, and have a Senior Admin monitoring the situation again tonight around the time this has been occuring.
Thanks
Tonight at 8PM we are doing maintainence on our last server affected by the kernel driver bug, XN1.
Updates will be provided here as usual.
XN1 (Complete)
8.00PM – Going down for reboot now
8.04PM – System is back
8.16PM – All VPS have been restarted and are booting up now
The maintainence has now been completed. Many thanks for your patience.
Tonight we are continuing with maintainence on XN12 and XN3.
Updates will be provided here throughout.
XN12 (Complete)
8.00PM – Going down for reboot now
8.03PM – System is up and starting rolling restarts of VPS
8.10PM – All VPS have been restarted
XN3 (Complete)
8.24PM – Going down for reboot now
8.26PM – System has failed to boot into the new kernel, investigating now
8.37PM – This server appears to have a bootloader (grub) issue which we are working to resolve
8.53PM – Looks like this isn’t such a simple issue. We are still working on it and will restore service ASAP.
9.04PM – Server is now booting up on the old kernel and we are investigating why the new kernel failed to boot. It’s the same hardware as our other servers.
9.22PM – Issue resolved and the system is now up on the new kernel. Restarts of VPS to follow
9.38PM – All VPS have been restarted
Today’s maintainence is now complete. Appologies for the delay on XN3, however this was absolutly mandatory to ensure the machine is not vulnerable to a driver bug which has affected two other systems earlier this week.
We are rebooting XN11 and XN14 tonight as per the earlier email to patch against a driver issue affecting two servers yesterday.
Updates will be provided here throughout.
XN11 (Done)
8.00PM – Going down for restart now
8.04PM – Small issue related to the BIOS on the machine, sorting now
8.06PM – Server is booting now
8.10PM – Server is up and starting rolling restarts of VPS now
8.24PM – All VPS have been restarted and should be up. Please be aware the RAID array on this machine is now doing a verification and I/O wait will be a little higher than usual until this completes. Shouldn’t be more than 30mins
XN14
8.35PM – Going down for restart now
8.39PM – Server is up and starting rolling restarts of VPS now
8.49PM – All VPS are up and we are going through them individualyl to ensure they are on the correct kernel + the matching kernel modules are copied inside VPS’s.
9.15PM – There are about 5 VPS left to restart
9.20PM – All VPS have been restarted and should be up. Please be aware the RAID array on this machine is now doing a verification and I/O wait will be a little higher than usual until this completes. With this being a larger server it will take a few hours for the load to settle and the RAID verification to complete.
Realtime updates (As close as possible) on the UK datacentre maintainence will be posted below. Watch this space!
19:00 01/10/09 – Servers are being prepared for powerdown, and having the latest updates installed including a kernel released today. Don’t worry we are also rolling out automated kernel module updates so you won’t need to copy them in manually.
00:00 02/10/09 – Servers are now being cleanly shutdown. If you have a dedicated server and have not provided us with your login details, we strongly advise powering down your machine now.
01:00 02/10/09 – All servers + power is now down.
05:14 02/10/09 – The following servers are now online: VZ1, XN1, XN2, XN7, XN10
05:39 02/10/09 – All servers are now online, we are working through each node to ensure VPS come up ok.
06:05 02/10/09 – We have issued the all-clear. All of our VPS nodes + customer servers are online and passed sanity checks. Some VPS nodes are doing routine quota checks and RAID verifications so you may find your VPS sluggish for a few hours until things settle, or you VPS is not online yet.
If your VPS is not up, please allow at least 1 hour from now for it to sort itself out, if it still doesn’t come up you can reboot by SolusVM or open a support ticket.
This is a copy of an email we sent out on 09/09/10 due to some planned maintainence on 02/10/2010 for mandatory power maintainence at the UK based datacentre we use.
—
Maintanence Task:
Controlled power outage to the Spectrum House facility, allowing for essential repairs to the main power systems.
Scheduled For:
Saturday 02/10/2010 between the hours of 00:00 and 08:00 GMT. We are expecting/hoping to be up sooner than 08.00GMT and senior staff will be in-office for the duration.
Further Details:
The datacentre recently installed a fourth 500kva UPS into a modular system, to add additional resilience and capacity for further growth. Having been designed to be modular, the appropriate connections were already available on the UPS systems. During installation, a fault was identified with the panel which meant the additional UPS could not be connected.
In order to resolve this fault, the panel needs to be electrically isolated, as it cannot be worked on while live for safety reasons. The work is being carried out by a team of engineers from the manufacturer, and the datacentre will be manned with additional staff for the duration, as well as our senior staff on standby for the duration.
It goes without saying that we sincereley appologise for any inconvenience this may cause you, and we are equally as frustrated about the circumstances of a power outage of this length to our equipment.
All servers will be powered down cleanly, if you are a dedicated server customer please contact us so we can store your login details and initiate a graceful shutdown prior to the power loss. Although the window for the maintainence is 00:00 to 08:00, every effort is being taken to reduce the downtime to a minimum.
We will keep you updated as much as possible via our offsite status blog, accessed at http://pcsmarthosting.net during the maintainence.
Please don’t hesitate to ask if you have any further questions.
Planned maintainence work is currently being carried out on XN12, affecting a small portion of customers on the 95.154.207.xx range.
This is a mandatory maintainence window in order to resolve ongoing stability issues.
Update: This has been completed, total downtime approximatley 20 minutes.
Migrations have now completed! In the following 24 hours we will be doing some in-depth optimization of the machine in order to further improve performance.
Thanks.