2 Disks failed simultaneously on a RAID 5 array - Disk, controller or software?

[Log in to get rid of this advertisement]
Hi there. My 1st post, so please be gentle.

I have a home server running Openfiler 2.3 x64 with 4x1.5TB software RAID 5 array (more details on the hardware and OS later). All was working well for two years until several weeks ago, the array failed with two faulty disks at the same time.

Well, those thing could happen, especially if one is using desktop-grade disks instead of enterprise-grade ones (way too expensive for a home server). Since is was most likely a false positive, I've reassembled the array:
Code:

# mdadm --assemble --force /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: forcing event count in /dev/sdb1(0) from 110 upto 122
mdadm: forcing event count in /dev/sdc1(1) from 110 upto 122
mdadm: /dev/md0 has been started with 4 drives.

and a reboot later all was back to normal.



-------------------

http://en.wikipedia.org/wiki/Mdadm

Recovering from a loss of raid superblock

There are superblocks on the drives themselves and on the raid (apparently). If you have a power failure, hardware failure, that does not include the drives themselves, and you cannot get the raid to recover in any other way, and wish to recover the data, proceed as follows:

Get a list of the devices in the raid in question:

mdadm --detail /dev/md[x]

Result something like this:

/dev/md127:
Version : 1.2
Creation Time : Sun Aug 21 23:35:28 2011
Raid Level : raid6
Array Size : 7814047744 (7452.06 GiB 8001.58 GB)
Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
Raid Devices : 6
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Sun Jan 1 11:43:17 2012
State : clean, degraded
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : clop:1 (local to host clop)
UUID : 7ee1e93a:1b011f80:04503b8d:c5dd1e23
Events : 62
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
2 8 81 2 active sync /dev/sdf1
3 8 65 3 active sync /dev/sde1
4 0 0 4 removed
5 0 0 5 removed

RaidDevice order (sdc1,sdd1,sdf1,sde1) and Chunk Size are critical

Record all your raid member parameters:

mdadm --examine /dev/sd[abcde...]1 | egrep 'dev|Update|Role|State|Chunk Size'

Look carefully at the Update time. If you have raid members attached to the motherboard and others attached to a raid card, and the card fails, but leaves enough members to keep the raid alive, you want to make a note of that. Look at Array State and Update Time. For example:

/dev/sdc1:
Update Time : Wed Jun 15 00:32:35 2011
Array State : AAAA.. ('A' == active, '.' == missing)
/dev/sdd1:
Update Time : Thu Jun 16 21:49:27 2011
Array State : .AAA.. ('A' == active, '.' == missing)
/dev/sde1:
Update Time : Thu Jun 16 21:49:27 2011
Array State : .AAA.. ('A' == active, '.' == missing)
/dev/sdf1:
Update Time : Thu Jun 16 21:49:27 2011
Array State : .AAA.. ('A' == active, '.' == missing)
/dev/sdk1:
Update Time : Tue Jun 14 07:09:34 2011
Array State : ....AA ('A' == active, '.' == missing)
/dev/sdl1:
Update Time : Tue Jun 14 07:09:34 2011
Array State : ....AA ('A' == active, '.' == missing)

Devices sdc1, sdd1, sde1 and sdf1 are the last members in the array and will rebuild correctly. sdk1 and sdl1 left the array (in my case due to a raid card failure).

Also note the raid member, starting with 0, the raid needs to be rebuilt in the same order. Chunk size is also important.

Zero the drive superblocks

mdadm --stop /dev/md0 # to halt the array
mdadm --remove /dev/md0 # to remove the array
mdadm --zero-superblock /dev/sd[cdefkl]1

Reassemble the raid

mdadm --create /dev/md1 --chunk=4096 --level=6 --raid-devices=6 /dev/sdc1 /dev/sdd1 /dev/sdf1 /dev/sde1 missing missing

'missing' tell the create command to rebuild the raid in a degraded state. sdk1 and sdl1 can be added later

Edit /etc/mdadm.conf and add an ARRAY line with a UUID. First get the UUID for your raid:

mdadm -D /dev/md

then:

nano /etc/mdadm.conf

and add something similar to the file (notice there is no # in front of the active line you are adding

#ARRAY /dev/md0 UUID=3aaa0122:29827cfa:5331ad66:ca767371
#ARRAY /dev/md1 super-minor=1
#ARRAY /dev/md2 devices=/dev/hda1,/dev/hdb1
ARRAY /dev/md1 UUID=7ee1e93a:1b011f80:04503b8d:c5dd1e23

Save with

Last, mark the array possilbly dirty with:

mdadm --assemble /dev/md1 --update=resync

Monitor the rebuild with

watch -n 1 cat /proc/mdstat

All your data should be recovered!

Birden fazla dosya uzantısını ile arama yapmak ve bunlarla işlem yapmak

How to find multiple file types using linux's "find"
I've always found *nix's "find" and "grep" rather hard to use. Not only are there different flavors of regular expressions to use, but mainly different syntax. For find, the directory you're searching for comes first. For grep, it comes last. To find the negation of something, you'd use "-not" and for grep it's "-v". pain.

Anyway, been trying to learn my tools better, and I found out how to grep and replace expressions across multiple files through emacs. Since rails uses all sorts of file extensions, naturally, I wanted to grep for find different files types. I had thought the -name options took regexs (it doesn't), so I had tried it in regex (no go)...only to find that it's something like this:


find . -name "*.rb" -o -name "*.rhtml"


the -o is the equivalent of a boolean "or". small tip...




------------


I was trying to find a solution todo a find & replace across multiple files which was purely command line based. There are plenty of scripts out there which will accomplish this but I needed a single line command. After some google searches and some experimentation I came up with this snippet.

find . -name "*.php" -print | xargs sed -i 's/foo/bar/g'

It looks a bit complicated but its quite simple. There are three components to the command:

find . -name "*.php" -print – Find all files (recursively) which has “.php” in the file and print them out. This will give you output like this:

./file.php
./includes/test.php
./classes/class.php

xargs- This command is used when you want to pass a lot of arguments to one command. xargs will combine the single line output of find and run commands with multiple
arguments, multiple times if necessary to avoid the max chars per line limit. In this case we combine xargs with sed
sed -i 's/foo/bar/g' – aka Stream Editor is a tool which should be in every sys admin’s toolkit. In this case every occurence of “foor” is replaced by “bar” in all the files found using the “find” command. Sed simply parses input and applies certain text transformations to it. There’s a lot to say about sed, you can find more at this tutorial.

This pretty much covers the core of the find & replace command. You could also open up a particular folder in an IDE and use it’s find and replace feature. But find + sed is quite fast and powerful.



Reply
JohnMay 29, 2008 10:17 AM

You can always use the -regextype to make regex play nice. The problem with using -o is that if you want to pass all of your results to a program as a list with -exec you can't because there needs to be a seperate -exec before each -o.

For instance I wanted to search my music directory or videos and pass them all to mplayer to be played in a random order but I had multiple file types in multiple locations, this is what I came up with:

find ./ -regextype posix-awk -regex "(.*.mpg|.*.avi|.*.wmv|.*.mpeg)" -exec mplayer -shuffle -loop 0 {} +