• "If a job is worth doing, it's worth doing well."
  • 20PERCENT20% Off For Life Coupon:

Announcements

[SOLVED] or3 availability

Monday, February 26, 2018

Monday, 26 February 2018, 03:12 - It has been brought to our attention that or3 node is experiencing availability problems. Relevant personnel at the datacenter have been alerted and we're awaiting resolution. We do not have a specific ETA at this stage.

Monday, 26 February 2018, 10:17 - the following notification has been sent to all affected users, providing a timeline of events:

====

We are writing to you regarding your active service on our Winity OR3 node.

At 2:55 AM CST this morning, we received an alert that this server had gone down. We engaged our remote access and determined that the server was powered off, but not able to power it back on remotely.

We then engaged the datacenter to investigate, and they determined that the redundant power supplies in this server had failed and were preventing the server from powering on. The power supplies were replaced, and the server was able to power on.

However on boot, it was discovered that due to the power supply failure, our RAID volumes were marked as offline due to two of the SSDs in one span of the RAID array being marked as failed.
The datacenter technicians are working to bring the volumes online as soon as possible.

For further updates, please reference the following announcement in our client area: https://winity.io/announcements.php?id=26

We know that you rely on our services and we apologize for any inconveience caused by this outage. If you have any questions or comments, please open a ticket for inquiry and a member of our team will respond directly.

-Winity Team

====

Monday, 26 February 2018, 14:26 - we have been unsucessful in onlining the volumes in place, and are working on copying the failed drives to new drives to insert into the array and see if they take. This will take several hours, and we'll update this announcement as we know more. 

Monday, 26 February 2018, 23:39 - we've been briefed that the copy operation has succeeded in at least one of the two failed drives. We are now attempting to force the array online if possible, and then rebuild from a degraded state.

Tuesday, 27 February 2018, 00:15 - we've received confirmation from the datacenter that the rebuild process has started. Further updates as they become available.

Tuesday, 27 February 2018, 01:47 - the array has been successfully rebuilt, however it appears that there is corruption on the OS partition. Windows refuses to boot. We're currently discussing internally how to get the system back to a working state.
Tuesday, 27 February 2018, 07:34 - Recovery remains underway. 
 
Tuesday, 27 February 2018, 09:06 - running through a SFC scan now. This process will take some time. 

Tuesday, 27 February 2018, 11:30 - the host node is booting to Windows now, and we're working on getting VPS online

Tuesday, 27 February 2018, 13:51 - VPS are running, although the server should NOT yet be considered stable, as there's still work to do on the host node. You may find that your VPS is running and accessible, but please wait for our update before considering the VPS production-stable

Tuesday, 27 February 2018, 19:47 - work continues to progress on this situation. 

Tuesday, 27 February 2018, 22:07 - server is online and most VPS are internet-accessible too. Some VPS were corrupted with the power distributor issue and are currently not bootable. We're working on getting them back online. Any running VPS should not yet be considered production-stable and may go offline at any time due to reboot actions or the like. 

Wednesday, 28 February 2018, 03:12 - 95% of the VPS's on the node have been restored and are online at the moment. However, unfortunately, some of the VPS's disk image have been corrupted. We have purchased a filesystem salvage software to save the important files off the filesystem of the corrupted ones. We are contacting the corrupted VPS owners at the moment to provide them with the salvaged files.

« Back