Common invocations of MegaCLI
Get an Overview of controllers and attached disk arrays
./MegaCli64 -AdpAllInfo -aALL (extracted info below)
./MegaCli64 -CFGDsply -aALL shows same info, with same verbosity
Product Name : PERC H800 Adapter Serial No : 25B0088
Link Speed : 0 Number of Frontend Port: 0 Device Interface : PCIE Number of Backend Port: 8 Port : Address 0 500c04f2e958eebf 1 500c04f2e958ee3f 2 0000000000000000 3 0000000000000000 4 0000000000000000 5 0000000000000000 6 0000000000000000 7 0000000000000000 Device Present ================ Virtual Drives : 1 Degraded : 0 Offline : 0 Physical Devices : 13 Disks : 12 Critical Disks : 0 Failed Disks : 0
Product Name : PERC 6/i Integrated
Serial No : 1122334455667788
Device Present ================ Virtual Drives : 1 Degraded : 0 Offline : 0 Physical Devices : 3 Disks : 2 Critical Disks : 0 Failed Disks : 0 Settings ================ Current Time : 12:16:19 6/18, 2013 Predictive Fail Poll Interval : 300sec Rebuild Rate : 30% Reconstruction Rate : 30% Cache Flush Interval : 4s Max Drives to Spinup at One Time : 2 Delay Among Spinup Groups : 12s Restore HotSpare on Insertion : Disabled Expose Enclosure Devices : Disabled Maintain PD Fail History : Disabled Maximum number of direct attached drives to spin up in 1 min : 0 Auto Enhanced Import : No Capabilities ================ The last paragraph above mentions many unusual RAID levels. Here is a brief explanation of the more exotic. RAID Level Supported : RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50, RAID60, PRL 11, PRL 11 with spanning, SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span Supported Drives : SAS, SATA
— Explanations of RAID Levels
This PDF explains secondary level raid etc: http://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf
- RAID6 – Dual Parity RAID:
Striped stripes: Break up a file into stripes and stripe it across all drives. Need multiple raid controllers and large files for this to provide benefit.
- PRL11-RLQ0 layout : aka RAID1E Integrated Adjacent Stripe Mirroring
Each block is written twice. The location of the mirror/parity block is defined is being in the same extent, but on the next adjacent disk. Spanning is needed when you have different sized Basic Virtual Drives (subdisks).
- SRL3: Spanned Secondary Raid Level 3:
(less protection) : data stripes are allocated pretty much where ever they can be – striped or concatenated from any available subdisks (aka bvd). The subdisk may or may not be raid-protected.
- RAID Level Qualifier:
This little guy says nothing about the level of protection, only that there are several ways to “skin a disk”. RLQ assigns numbers to them.
Examine drives on one adapter
root@auspcrpbak03:MegaCli> ./MegaCli64 -LdPdInfo -a0
Notes: There are 12 drives of about 1.8TB. We’re running RAID6, so we expect to lose 2 drives to parity and get 18TB.
Adapter #0 Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id: 0) Name :Virtual Disk 0 RAID Level : Primary-6, Secondary-0, RAID Level Qualifier-3 Size : 18.188 TB ( This is size presented to OS ) Sector Size : 512 Parity Size : 3.637 TB (In addition to the 18T. For RAID6 we have two) State : Optimal Number Of Drives : 12 Current Cache Policy : WriteThrough, ReadAdaptive, Direct, No Write Cache if Bad BBU Number of PDs: 12
Explanation of Raid Levels:
[‘Primary-0, Secondary-0, RAID Level Qualifier-0’] = RAID-0
[‘Primary-1, Secondary-0, RAID Level Qualifier-0’] = RAID-1
[‘Primary-5, Secondary-0, RAID Level Qualifier-3’] = RAID-5
[‘Primary-6, Secondary-0, RAID Level Qualifier-3’] = RAID-6
[‘Primary-1, Secondary-3, RAID Level Qualifier-0’] = RAID-10
--- PD: 0 Information Enclosure Device ID: 16 Slot Number : 0 Drive's position : DiskGroup: 0, Span: 0, Arm: 0 Enclosure position: N/A Device Id : 13 WWN: 5000C500558ADE08 Sequence Number: 2 Media Error Count : 0 PD Type : SAS Raw Size: 1.819 TB [0xe8e088b0 Sectors] (This is real 1024M size, not 1000M. It's sold as a 2TB drive) Firmware state: Online, Spun Up Device SAS Address(0): 0x5000c500558ade09 SAS Address(1): 0x5000c500558ade0a Connected Port Number: 1(path0) 0(path1) Inquiry Data: SEAGATE ST32000645SS RS0FZ1K0AV9E Drive has flagged a S.M.A.R.T alert : No PD: 1 Information Enclosure Device ID: 16 Slot Number: 1 Drive's position: DiskGroup: 0, Span: 0, Arm: 1 Device Id: 12 WWN: 5000C500558AEBA4 Sequence Number: 2 Media Error Count: 0 PD Type: SAS
< and much more >
Optional toppings for most LD configurations:
- WT : WriteThrough safer. Only returns once data is written to disk.
- WB: WriteBack faster. Returns as soon as data is is written to cache
- NORA : No Read Ahead vs
- RA: ReadAhead vs
- ADRA : Adaptive ReadAhead where if the previous two requests were sequential, it pre-loads the next in sequence.
- Cached: Cache reads.
- Direct: Only the previous read is cached.
- -strpszM : Stripe size so -strpsz64 means 64kb stripe size.
- Hsp[E0:S0] : Choose this drive to be a hot-spare
Dump the Event Log for the Adapter
./MegaCli64 -AdpEventLog -GetLatest 20 -f /tmp/events.txt -a0
Build a Virtual Disk
MegaCli -CfgLdAdd -rX [E0:S0,E1:S1,…] [WT|WB] [NORA|RA|ADRA] [Direct|Cached] [CachedBadBBU|NoCachedBadBBU] [-szXXX [-szYYY …]]
[-strpszM] [-Hsp[E0:S0,…]] [-AfterLdX] | [Secure]
[-Default| -Automatic| -None| -Maximum| -MaximumWithoutCaching] [-Cache] [-enblPI] [-Force]-aN
rX : Raid Level X. You can see your options by looking at the raid adapter settings (top of this page).
In this example we can choose 0,1,5,6,00,10,50,60 and more
[E0:S1,E0:S3] – Enclosure0:Slot1, Enclosure0:Slot3 : 2 Drives to use in the VD. Include the square brackets.
Configure disks as JBOD
MegaCli64 -AdpSetProp -EnableJBOD 1 -a0
Configure RAID 10
MegaCli64 -CfgSpanAdd -R10 -Array0 [E0:S1,E0:S2,E0,S3] -Array1 [E1:S0,E1:S1,E1:S2] + Toppings above
Delete the first logical drive on the first controller
MegaCli64 -CfgLdDel -L0 -a0
Replace a Failed Drive
MegaCli64 -PDOffline -PhysDrv [32:4] -a0
MegaCli64 -PDPrpRmv -PhysDrv [32:4] -a0
Pull the drive, insert a new one
MegaCli64 -Pdgetmissing -a0
No. Array Row Size Expected
0 1 4 1907200 MB
MegaCli64 -PdReplaceMissing -PhysDrv [32:4] -Array1 -row4 -a0
Rebuild the Array
MegaCli64 -PDRbld -Start -PhysDrv [32:4] -a0
Show Battery Backup Info
MegaCli64 -AdpBbuCmd -aALL
RAID Rebuild Rate – Increase to 90 Percent
MegaCli64 -AdpSetProp BgiRate 90 -aALL
Save configuration to filesystem
MegaCli64 -CfgSave -fFilename -a0
When you replace controllers, you can duplicate the previous config. In BIOS look for ‘Add Saved Configuration’. Say ‘Y’ to the warning to import it from the disks. (Be sure that your disk array is healthy first.)
RAID 10 and variants
Due to Dawid’s comment below, I did some more research, refreshed my recollections about RAID10 and similar configurations and have the following comments.
RAID10 is shorthand for RAID1+0 which is a stripe across mirrors. (Take RAID1/mirrored disks and RAID0/stripe them).
RAID01 is shorthand for RAID0+1 which mirrors disks that are striped. (Take RAID0 disks and RAID1/mirror them).
RAID01 is a less common config because RAID10 offers more redundancy (a big reason for why we RAID) and few disadvantages compared to RAID01.
In the case of four disks (which is a common config), I don’t (personally) see a big difference, but things get interesting when you have six or more disks. So let’s start there.
At this point, an analogy may make things more interesting, even if most readers already fully understand what I am saying. When I was younger, I played a game called Battleships. Assuming you have to carry eight squares of cargo, which would you prefer: two four-unit battleships or four two-unit destroyers? When the bombs start falling, you would much prefer to have your cargo in four different locations. While this is true when a malevolent opponent is trying to locate and destroy your ships, it’s also true when chaos sets in on your disk arrays. Forcing Murphy to precisely take out both disks in the same mirror (for example both yellows from the left picture), rather than one disk from each stripe is a better strategy.
- Illustration of RAID01 in a 2-by-4 config versus RAID10 in a 4-by-2 config
The image also clarifies somewhat the “Primary” and “Secondary” listings in the output from megacli. Primary is the “RAID1” part of “RAID1+0” (say the two yellow squares). It is the fundamental building block of the array. The “Secondary” field is how you organize the primal units you first created. For a RAID1+0, we stripe them (RAID 0).
Dawid’s situation was that he has 4 disks, but they show as (Pri-1, Sec-0, Qual-0) which is how megacli’s represents a plain mirror. I suspect that he manually created the striped disks then mirrored them and that megacli did not keep track of all the steps. I expect that it will work out just fine, but it would be more obvious and faster if next time he ran ‘megacli -CfgLdAdd -r10 …’