Recovering lvm over RAID5 Linux using R-Studio

A forum on data recovery using the professional data recovery software R-STUDIO.
kellogs
Posts: 12
Joined: Thu Sep 01, 2011 3:59 pm

Recovering lvm over RAID5 Linux using R-Studio

Post by kellogs » Thu Sep 01, 2011 4:31 pm

Dear all,

First many thanks, R-studio saved my life.
Here is how:

I am using a Debian GNU/Linux system:
* RAID 5 , four 1Tb discs
* lvm over RAID. lvm has 2 partitions root and home.

This system has all photos and films from our family and company accounting (old data).
Let's say it is a crucial system.

There used to be a backup on a second RAID5.

But our house got broken in a few weeks ago, but by chance the RAID systems were not stolen.
Therefore I decided to reformat the backup using RAID and encrypted LVM.
During format, I did a low level erase writing zeros.

I formated the backup and started copying main RAID system to backup RAID.

And this is how the system died:
during copy of main to backup, something went wrong on the first RAID server.
The RAID got degraded, I rebooted but nothing worked as expected, as the system got stuck on grub.

No backup (it was zeroed)
This is Murphy's law!

Hopefully, I was on holidays and could work a full week trying to resolve this crisis.

Here is how I did it :

1) Rebuilding the RAID
When one of the discs died, I could not rebuild the system easily.
The reason is that I did a hard reset while the RAID was unsynchronised.

Never EVER do a hard reset when a RAID system is degraded.

I tried like hell the usual recovery:
* Buy a new disc
* Format with same partitions
* mdadm --assemble /dev/md1 /dev/sda6 /dev/sdb6 /dev/sdc6 /dev/sdd6
* mdadm --manage /dev/md1 --add /dev/sdd6

It did not work, I had to rebuild the RAID array (like explained in several howtos):
* mdadm --create /dev/md1 -n4 -l5 /dev/sda6 /dev/sdb6 /dev/sdc6 /dev/sdd6

It worked.

2) lvm revovery
I could rebuild the array, but no lvm information.
lvm information should always be saved to backup but I did not know it.

Because I had created a new RAID array, the lvm information was missing from headers.
I look at dd dumps, no trace of lvm configuration.

I was quite desperate until I found R-studio for Linux.
After testing it, I bought it.

3) Using R-Studio
I created a virtual array with the four drives.
I am currently searching all files in the array and some partitions have been found.
I can extract individual files, all seems fine.

The process should take 14 hours.

After testing 10 minutes, it seems like my partitions /root and /home have been found.

So my question is :
* Are you able to recover a whole lvm partition (formatted with ext3)?
* Or are you rebuilding the partitions from scratch using a virtual file system?

Finally, is there a way to dump the whole content of a recovered partition to a large 2To disc?

Congratulations for this wonderful software and many thanks.
Kind regards,

Kellogs
Last edited by kellogs on Fri Sep 02, 2011 2:43 am, edited 1 time in total.

kellogs
Posts: 12
Joined: Thu Sep 01, 2011 3:59 pm

Re: Recovering lvm over RAID5 Linux using R-Studio

Post by kellogs » Fri Sep 02, 2011 2:12 am

xxx
Last edited by kellogs on Fri Sep 02, 2011 2:27 am, edited 1 time in total.

kellogs
Posts: 12
Joined: Thu Sep 01, 2011 3:59 pm

Re: Recovering lvm over RAID5 Linux using R-Studio

Post by kellogs » Fri Sep 02, 2011 2:16 am

Hello again,

R-Studio was able to create virtual partitions.
So I should be able to extract the content of my lvm partitions.
But it should take hours.

I am not sure to go this way, as my RAID array is synchronized.
All data is there, I only need a working lvm configuration.

The lvm backup file should be in /etc/lvm/backup/
Is R-Studio able to recover this folder?
It could be a cool feature...

Then I would run:
vgcfgrestore -f lvmbackupfile
vgscan
pvscan

kellogs
Posts: 12
Joined: Thu Sep 01, 2011 3:59 pm

Re: Recovering lvm over RAID5 Linux using R-Studio

Post by kellogs » Fri Sep 02, 2011 3:20 am

Files in /etc/lvm/backup have no extension.
How to recover them? ;)

kellogs
Posts: 12
Joined: Thu Sep 01, 2011 3:59 pm

Re: Recovering lvm over RAID5 Linux using R-Studio

Post by kellogs » Fri Sep 02, 2011 3:46 am

I need to search for the lvm backup file.
It should look like:

Code: Select all

        physical_volumes {

                pv0 {
                        id = "rVNFt0-YYRQ-cz2n-igAB-MD3f-Edni-5x2Qqt"
                        device = "/dev/sda5"    # Hint only

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 770015232    # 367,172 Gigabytes
                        pe_start = 384
                        pe_count = 93995        # 367,168 Gigabytes
                }
        }

        logical_volumes {

                root {
                        id = "qsCZ5D-ydWo-T3NA-UYop-lWos-RmnM-Q1kQlu"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 9536     # 37,25 Gigabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }
                                                                                                                                   54,1-8        25%

kellogs
Posts: 12
Joined: Thu Sep 01, 2011 3:59 pm

Re: Recovering lvm over RAID5 Linux using R-Studio

Post by kellogs » Fri Sep 02, 2011 5:06 am

I searched for "logical_volumes {" string using text/hexadecimal editor and quickly found the file.
There are several versions of the same file.
The trick is to find the lasted backup.
It is quite easy as the text includes the date in Unix format.

I copied the text file and pasted it.

Unfortunately, the text contains hexa and text and utf8.

It would be nice to add a "copy special" feature to allow copy only text result.

IMHO this could be automated in R-Studio to recover a lost lvm with no backup of lvm structure.
Look at Google, there are hundreds of such cases.

kellogs
Posts: 12
Joined: Thu Sep 01, 2011 3:59 pm

Re: Recovering lvm over RAID5 Linux using R-Studio

Post by kellogs » Fri Sep 02, 2011 6:14 am

Here is the file that I could recover:

Code: Select all

contents = "TextFormat Volume Group"
version = 1
description = "Created *before* executing 'lvresize -L +50G /dev/lvm/home'"
creation_host = "supermicro" # Linux supermicro 2.6.30-2-486 #1 Sat Sep 26 00:03:46 UTC 2009 i686

creation_time = 1260353201. #Wed Dec  9 11:06:41 2009

lvm {
	id = "pa5YGU-T1JG-vx7W-iolk-Hvrg-gJdm-AtMnRH"
	seqno = 8
	status = ["RESIZEABLE", "READ", "WRITE"]
	flags = []
	extent_size = 8192 # 4 Megabytes
	max_lv = 0
	max_pv = 0

	physical_volumes {

		pv0 {

		id = "TT2YkN-9q19-yyPk-gU0e-qZBC-yhMx-6XNbAm"
		device = "/dev/md1" #Ht only
		
		status = ["ALLOCATABLE"]
		flags = []
		dev_size = 5848799616 # 2.72356 Terabytes
		pe_start = 384
		pe_count = 713964 # 2.72356 Terabytes
		}
	}

	logical_volumes {

		root 	{
			id = "JaFCuw-nqDg-G3sS-qRMs-Mcjw-JHSX-CIA5MN"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			segment_count =1

			segment1 {
			start_extent = 0
			extent_count = 7152 #27.9375 Gigabytes
			
			type = "striped"
			stripe_count = 1 # linear
			
			
			stripes = [
			        "pv0", 0
			                ]
			}
		}

		home 	{
			id = "wX28KJ-ziic-Uy5c-0iCT-N54b-q2Hj-0ltSf2"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			segment_count = 1
			segment1 {
			start_extent = 0
			extent_count = 699940 # 2.670
			
			type = "striped"
			stripe_count = 1 # linear

			stripes =[
			        "pv0", 7152
			]
			}
		}

	}
}
vgcfgrestore -d -vvv -f /media/DataTravel/lvm.conf lvm
Processing: vgcfgrestore -d -vvv -f /media/DataTravel/lvm.conf lvm
O_DIRECT will be used
Setting global/locking_type to 1
Setting global/wait_for_locks to 1
File-based locking selected.
Setting global/locking_dir to /var/lock/lvm
Locking /var/lock/lvm/V_lvm WB
_do_flock /var/lock/lvm/V_lvm:aux WB
_do_flock /var/lock/lvm/V_lvm WB
_undo_flock /var/lock/lvm/V_lvm:aux
Locking /var/lock/lvm/P_orphans WB
_do_flock /var/lock/lvm/P_orphans:aux WB
_do_flock /var/lock/lvm/P_orphans WB
_undo_flock /var/lock/lvm/P_orphans:aux
Opened /media/DataTravel/lvm.conf RO
Closed /media/DataTravel/lvm.conf
Can't process text format file - unrecognised contents field.
Unlocking /var/lock/lvm/P_orphans
_undo_flock /var/lock/lvm/P_orphans
Unlocking /var/lock/lvm/V_lvm
_undo_flock /var/lock/lvm/V_lvm
Restore failed.
Any idea what is wrong?

kellogs
Posts: 12
Joined: Thu Sep 01, 2011 3:59 pm

Re: Recovering lvm over RAID5 Linux using R-Studio

Post by kellogs » Fri Sep 02, 2011 7:39 am

Before of a poor copy, there were errors in the configuration file.
It is highly important that R-Studio could allow text copy.

Here is the right configuration:

Code: Select all

contents = "Text Format Volume Group"
version = 1

description = "Backup"

creation_host = "supermicro-formalib"
creation_time = 1260353201

lvm {
	id = "pa5YGU-T1JG-vx7W-iolk-Hvrg-gJdm-AtMnRH"
	seqno = 8
	status = ["RESIZEABLE", "READ", "WRITE"]
	flags = []
	extent_size = 8192 # 4 Megabytes
	max_lv = 0
	max_pv = 0
        metadata_copies = 0

	physical_volumes {

		pv0 {

		id = "TT2YkN-9q19-yyPk-gU0e-qZBC-yhMx-6XNbAm"
		device = "/dev/md1" #Hint only
		
		status = ["ALLOCATABLE"]
		flags = []
		dev_size = 5848799616 # 2.72356 Terabytes
		pe_start = 384
		pe_count = 713964 # 2.72356 Terabytes
		}
	}

	logical_volumes {

		root 	{
			id = "JaFCuw-nqDg-G3sS-qRMs-Mcjw-JHSX-CIA5MN"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			segment_count = 1

			segment1 {
			start_extent = 0
			extent_count = 7152 #27.9375 Gigabytes
			
			type = "striped"
			stripe_count = 1 # linear
			
			
			stripes = [
			        "pv0", 0
			                ]
			}
		}

		home 	{
			id = "wX28KJ-ziic-Uy5c-0iCT-N54b-q2Hj-0ltSf2"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			segment_count = 1

			segment1 {
			start_extent = 0
			extent_count = 699940 # 2.670
			
			type = "striped"
			stripe_count = 1 # linear

			stripes =[
			        "pv0", 7152
			]
			}
		}

	}
}
vgcfgrestore -vvv -f /media/DataTravel/lvm.conf lvm

Code: Select all

      Setting global/locking_type to 1
      Setting global/wait_for_locks to 1
      File-based locking selected.
      Setting global/locking_dir to /var/lock/lvm
      Locking /var/lock/lvm/V_lvm WB
      Locking /var/lock/lvm/P_orphans WB
      /dev/loop0: No label detected
      /dev/md1: No label detected
      /dev/sda5: No label detected
      /dev/sdb5: No label detected
      /dev/sdc5: No label detected
      /dev/sdd5: No label detected
      /dev/sde1: No label detected
      /dev/md126: No label detected
      Setting devices/ignore_suspended_devices to 0
      Setting devices/cache_dir to /etc/lvm/cache
      Setting devices/write_cache_state to 1
    Wiping cache of LVM-capable devices
      /dev/loop0: size is 2110896 sectors
      /dev/loop0: size is 2110896 sectors
      /dev/loop0: No label detected
      /dev/sda: size is 1953525168 sectors
      /dev/loop1: size is 0 sectors
      /dev/sda1: size is 1959867 sectors
      /dev/sda1: size is 1959867 sectors
      /dev/md1: size is 5848793088 sectors
      /dev/md1: size is 5848793088 sectors
      /dev/md1: No label detected
      /dev/loop2: size is 0 sectors
      /dev/sda2: size is 2 sectors
      /dev/loop3: size is 0 sectors
      /dev/loop4: size is 0 sectors
      /dev/loop5: size is 0 sectors
      /dev/sda5: size is 1959867 sectors
      /dev/sda5: size is 1959867 sectors
      /dev/sda5: No label detected
      /dev/loop6: size is 0 sectors
      /dev/sda6: size is 1949600142 sectors
      /dev/sda6: size is 1949600142 sectors
      /dev/loop7: size is 0 sectors
      /dev/sdb: size is 1953525168 sectors
      /dev/sdb1: size is 1959867 sectors
      /dev/sdb1: size is 1959867 sectors
      /dev/sdb2: size is 2 sectors
      /dev/sdb5: size is 1959867 sectors
      /dev/sdb5: size is 1959867 sectors
      /dev/sdb5: No label detected
      /dev/sdb6: size is 1949600142 sectors
      /dev/sdb6: size is 1949600142 sectors
      /dev/sdc: size is 1953525168 sectors
      /dev/sdc1: size is 1959929 sectors
      /dev/sdc1: size is 1959929 sectors
      /dev/sdc2: size is 2 sectors
      /dev/sdc5: size is 1959929 sectors
      /dev/sdc5: size is 1959929 sectors
      /dev/sdc5: No label detected
      /dev/sdc6: size is 1949600204 sectors
      /dev/sdc6: size is 1949600204 sectors
      /dev/sdd: size is 3907029168 sectors
      /dev/sdd1: size is 1959867 sectors
      /dev/sdd1: size is 1959867 sectors
      /dev/sdd2: size is 2 sectors
      /dev/sdd5: size is 1959867 sectors
      /dev/sdd5: size is 1959867 sectors
      /dev/sdd5: No label detected
      /dev/sdd6: size is 3903104204 sectors
      /dev/sdd6: size is 3903104204 sectors
      /dev/sde: size is 7555528 sectors
      /dev/sde1: size is 7550487 sectors
      /dev/sde1: size is 7550487 sectors
      /dev/sde1: No label detected
      /dev/md126: size is 1959680 sectors
      /dev/md126: size is 1959680 sectors
      /dev/md126: No label detected
  Couldn't find device with uuid TT2YkN-9q19-yyPk-gU0e-qZBC-yhMx-6XNbAm.
  Cannot restore Volume Group lvm with 1 PVs marked as missing.
      Unlocking /var/lock/lvm/P_orphans
      Unlocking /var/lock/lvm/V_lvm
  Restore failed.

kellogs
Posts: 12
Joined: Thu Sep 01, 2011 3:59 pm

Re: Recovering lvm over RAID5 Linux using R-Studio

Post by kellogs » Fri Sep 02, 2011 12:40 pm

This was a UID problem, I created a new pv.
Then I could import the lvm configuration file.
And rebuild volumes.

Then I mounted the volumes, but the ext3 filesystem was not recognised.
Do you any tool or method to recover a destroyed lvm?
The RAID5 data is still there.

Alt
Site Moderator
Posts: 3129
Joined: Tue Nov 11, 2008 2:13 pm
Contact:

Re: Recovering lvm over RAID5 Linux using R-Studio

Post by Alt » Sat Sep 03, 2011 4:40 am

Unfortunately, R-Studio doesn't currently support lvm. I think, the easiest way to recover your files is to copy the entire content of the partition to another disk. You see, experimenting with the disks where the original data resides is highly dangerous to the data.
Then you may create a new RAID+LVM and copy the data back.

Post Reply