Readynas X-Raid failure

Discussions on using the professional data recovery program R-STUDIO for RAID re-construction, NAS recovery, and recovery of various disk and volume managers: Windows storage spaces, Apple volumes, and Linux Logical Volume Manager.
Guest

Readynas X-Raid failure

Post by Guest » Sun Oct 21, 2012 4:48 am

Hi all

Let me start by explaining the setup and series of events: I have a Readynas NV+ box with 4x1tb drives in x-raid format and a couple of weeks ago I got a system alert saying:

1 - Access to the disk on channel (??) is producing I/O errors. Although the array is still redundant, please replace this drive as soon as possible, as it is likely to fail soon.

(the above message came through a few times and then:)

2 - A SATA reset has been performed on one or more of your disks that may have affected the RAID parity integrity. It is recommended that you perform a RAID volume resync from the RAID Settings tab ( accessible in the Volumes page => Volume tab in FrontView ). The resync process will run in the background, and you can continue to use the ReadyNAS in the meantime.

(followed by, because I followed the instructions)

3 - RAID sync started on volume C.

Then the system ground to a halt, couldn't access it at all, I waited nearly an hour and then pulled the plug so then

4 - Improper shutdown detected., To ensure data integrity, a filesystem check should be performed as soon as possible.

(so I did that)

5 - The paths for the shares listed below could not be found. Typically, this occurs when the ReadyNAS is unable to access the data volume.

(ok panic!)

6 - The X-RAID engine has failed to start., To ensure data integrity, please refrain from using the NAS and contact Support immediately.

(double panic and frantic research on the web. I followed some suggestions from netgear forums about booting from usb, reflashing firmware etc etc but with no joy and then when I worked out that the failing drive was the 4th drive I pulled it out, inserted a blank drive in the hope that the raid would rebuild...but no joy instead I got this message)

7 - A SATA reset has been performed on one or more of your disks that may have affected the RAID parity integrity. It is recommended that you perform a RAID volume resync from the RAID Settings tab ( accessible in the Volumes page => Volume tab in FrontView ). The resync process will run in the background, and you can continue to use the ReadyNAS in the meantime.


This is when I powered down, took the readynas box and made haste for Kroll Ontrack....Two weeks later and with a possible bill for GBP 4000 they showed me what the could recover - it was missing most of the files that I needed so I thanked them but no thanks.

Their report indicated that the 4th Drive was only imaging to 85% and seems to have "magnetic issue" rather than hardware or electronics. But the strange thing was the 3 remaining drives were absolutely fine but they were not seeing them as RAID but Striped. X-Raid is glorified RAID 5 so I thought I would give this a shot myself as I couldn't fathom why they would see the 3 drives as striped.

Got R-Studio Network (under advisement of a Tech guru friend) and started imaging the drives.

Drive 4 failed to image (as expected) using the R Studio Compressed image and so I just did a byte by byte image just for the sake of it. When I put the 4th disk in the USB cradle I did notice that the drive was read as having:

1 Empty and 4 volumes varying in size from a few kb to 500gb ( I assume that this is the way x-raid does things). Also the byte by byte image created doesn't show these volumes at all.

Then I Imaged Drive 1 and that was all fine again this showed contents as being 1 Empty and 4 volumes varying in size.

Now I am Imaging Drive 2 and this one is showing NO volumes...So this got me thinking that this disk might have been initialized and that is why it is not showing volumes, and probably why Kroll Ontrack were not able to see it as RAID.

I know I am jumping the gun as I haven't imaged the 3rd drive and tried the RAID rebuild, but I am guessing that I have to recover Drive 2 to a previous state when it had the volume information and presumably RAID info.

If anyone has any ideas, experience or advice it would be very much appreciated. I am a novice at data recovery and understanding RAID parity etc...but I am learning quickly. Also Netgear are useless at support or disclosing any info regarding their system so no help there (as many others have found).

Thanks Frank

frankDMI
Posts: 4
Joined: Sun Oct 21, 2012 4:05 am

Re: Readynas X-Raid failure

Post by frankDMI » Mon Oct 22, 2012 8:33 am

Update

I have now imaged all 4 drives. As I said before:

Drive 1 is ok and shows 4 volumes
Drive 2 is ok but shows no volumes
Drive 3 is ok and shows 4 volumes
Drive 4 is partially corrupt but shows 4 volumes

I have tried to use the R Studio virtual RAID and obviously I am getting nowhere because I don't know what the RAID parameters are but it is painfully obvious that if I had the volume set for Disk 2 I could possibly guess the RAID parameters and using a blank drive as my 4th drive I could start rebuilding the RAID and get my files.

If anyone has any experience in this I would very much appreciate guidance/help.

Thanks

Frank

Alt
Site Moderator
Posts: 3129
Joined: Tue Nov 11, 2008 2:13 pm
Contact:

Re: Readynas X-Raid failure

Post by Alt » Tue Oct 23, 2012 11:28 am

Kroll Ontrack are really professionals, so prospects for do-it-yourself data recovery look pretty much grim.
I can't say much more without looking closer at the actual data on the disks.
We at R-TT don't do actual data recovery, but I can arrange such data inspection using remote data recovery over the Internet with our partner data recovery company in Europe. This article gives the overview of what is to be done: R-Studio: Data Recovery over Network. If you agree, I think, then it would be better to discuss this via e-mail: raptorbck (at) gmail (dot) com

magicm
Posts: 9
Joined: Thu Sep 27, 2012 10:20 am

Re: Readynas X-Raid failure

Post by magicm » Thu Oct 25, 2012 12:46 pm

Frank, I just went thru a similar situation. I suggest you look at http://forum.r-tt.com/netgear-x-raid2-r ... t8261.html.

frankDMI
Posts: 4
Joined: Sun Oct 21, 2012 4:05 am

Re: Readynas X-Raid failure

Post by frankDMI » Sun Oct 28, 2012 1:24 pm

Thanks for your suggestion Alt and also magiccm for your link...I did read your post and am eagerly awaiting your response on your success (good luck).

One of the issues that I have is that one of my RAID disks has lost its volume information and combining that with a 4th disk that is partially corrupted, it isnt making life easy.

Tomorrow I have a friend that will log into my ubuntu and he is going to try and rebuild the missing data volumes on disk 2, by copying the layout of the other disks, and if he succeeds then in theory I could rebuild the system...here is me wishing.

If this fails I will take you up on your offer Alt.

Thanks for helping a noob.

Cheers

Frank

Alt
Site Moderator
Posts: 3129
Joined: Tue Nov 11, 2008 2:13 pm
Contact:

Re: Readynas X-Raid failure

Post by Alt » Mon Oct 29, 2012 8:24 am

OK, just send me an email, if necessary.

frankDMI
Posts: 4
Joined: Sun Oct 21, 2012 4:05 am

Re: Readynas X-Raid failure

Post by frankDMI » Mon Oct 29, 2012 9:46 am

Hi all, I have a quick question, specially for people with readynas x-raid experience. If a drive get Initialized does that mean that all the data is wiped on the disk or just the MBR (or linux equivalent)...I have a horrible suspicion that one of the good drives was initialized by the readynas box.

Alt
Site Moderator
Posts: 3129
Joined: Tue Nov 11, 2008 2:13 pm
Contact:

Re: Readynas X-Raid failure

Post by Alt » Mon Oct 29, 2012 10:01 am

frankDMI wrote:Hi all, I have a quick question, specially for people with readynas x-raid experience. If a drive get Initialized does that mean that all the data is wiped on the disk or just the MBR (or linux equivalent)...I have a horrible suspicion that one of the good drives was initialized by the readynas box.
Most like not, but that depends. Just browse through the disk using R-Studio's editor. If it all 00 or FF, the data is gone.

magicm
Posts: 9
Joined: Thu Sep 27, 2012 10:20 am

Re: Readynas X-Raid failure

Post by magicm » Mon Oct 29, 2012 7:39 pm

Frankdmi, I was successful with getting all the data back. I would highly recommend ReclaiMe to find the RAID parameters needed by R-Studio to recover the data.

frankDMI
Posts: 4
Joined: Sun Oct 21, 2012 4:05 am

Re: Readynas X-Raid failure

Post by frankDMI » Wed Oct 31, 2012 4:59 am

Joy of Joy - Successful Recovery of data from Netgear ReadyNas NV+ running X-Raid

Thanks to some digging around, logic and a failed attempt by Kroll Ontrack to recover data we have managed to recover all my data using Ubuntu and R-Studio.

Let me recap the original situation:

1 - ReadyNas NV+ Box running 4x1Tb drives in X-Raid Mode started to grind to a halt.
2 - Since the RadyNas Box didn't specify what drive was failing and ground to a halt I had no choice (foolishly with hindsight) but to try and pull out one drive at a time to see if the Box would reeboot and work out which drive was faulty.
3 - Because of this action the ReadNas NV+ Box did some "funny things" and resulted in showing me the Volumes but telling me that they were not available. The screen at the front told me 0/0...I THOUGHT THAT WAS IT
4 - Took my ReadyNas box to Kroll Ontrack and told them that the system was running on X-Raid (which I took to mean a form of RAID 5)
5 - Kroll Ontrack confirmed that one of the drives was partially corrupt and they could only image it at 85% but the other 3 drives were fine
6 - After 2 weeks they came back and said that they only managed to recover 1/3 of the files that I was supposed to have and told me that they were seeing the Disks as Striped NOT as Raid 5??? Anyways the data they could recover was useless to me so I thanked them and only paid for the diagnostic. At this point I would like to say that Kroll Ontrack were extremely professional, were always available to answer my questions and gave me regular updates...just didn't succeed with recovering all the data.
7 - Having got the RadyNas NV+ Back and under the suggestion of Kroll Ontrack, I made Images of all 4 drives using R-Studio Network and used the Byte by Byte Image copy to copy the 4 disks onto 2 3TB USB external drives. The R-Studio Image copy for the 4th drive I put the parameters of 20 attempts to read sector failures. Each drive took around 6 hours to image.
6 - R-Studio showed me that 3 of the 4 drives had 4 volumes and one drive had 0 volumes (the drive with 0 volumes was not the corrupt 4th drive but one of the good ones), also I could see that this second drive had data on it!
7 - Doing a lot of research on the internet (I had 2 weeks of twiddling my thumbs) I found that the ReadyNas NV+ box is bascally running linux and that people had success in connectig the drives and viewing content on Ubuntu.
8 - I called my friend Steve at Regenology and since they do hosting I thought that he might be able to help as he knows a lot about Linux for obvious reasons.
9 - The first thing he did was to copy the Volume Table from one of the images that had a Volume Table to the image of the drive that didn't have the Volume Table (we saw that the volume tables were identical for all 3 drives). The loss of this volume table on drive 2 was probably caused by the ReadyNas NV+ box Initializing the disk when I was doing the drive swap test!
10 - The next thing we did was to mount these images in Ubuntu, including the partially corrupt one, and using the "kpartx -a" command in the terminal (this is where it all got too technical for me) followed by a "vgscan" VOLUME C appeared!!!! This has all the DATA!!!. This was the guiding article - http://home.bott.ca/webserver/?p=306
11 - After listing the folders in this volume and determining the size (just to make sure that data was there) we "rsync" all the data onto an external Drive

Next thing I will try is to do the volume table rebuild on the the physical drive 2 from the ReadyNas box, see if it boots with the 3 good drives and then throw in a 4th drive (in place of the corrupt one) and let it do what it is supposed to do and re-sync all the data - Even though I seem to have all the data now anyways I have learnt my lesson and doubling on everything.

Also I haven't been able to determined if the corrupted Disk 4 data has affected any of the files that I have recovered via this method. But in theory the ReadyNas box should rebuild everything....but now I am not afraid to try!!

A big thanks to R-Studio for giving me to tool to clone the drives (in an easy way) and also cloning the corrupt drive.
A big thanks to Kroll Ontrack, even though they didn't manage to get the data, they still gave me information that allowed me to think of this...and didn't have to spend a fortune!
A big thanks to the forums and the internet
A big thanks for nothing to Netgear for being USELESS in their support
A big thanks to this forum - Alt and magicm
And finally a huge thanks to Steve at Regenology http://regenology.co.uk/ for doing the terminal work in Ubuntu (we used Google Chrome Remote Desktop App for him to access my computer)

I will let you know how I get on with the ReadyNas box when I do the physical rebuild....see if it works.

The main lesson learnt is BACKUP BACKUP BACKUP...Cloud, Physical, Remote...DO NOT RELY on just one medium

Post Reply