Hardware Raid Setup using MegaCli

From Linux Raid Wiki
(Difference between revisions)
Jump to: navigation, search
Line 195: Line 195:
 
This is the slot number the physical drive is connected to.
 
This is the slot number the physical drive is connected to.
  
Actually this slot number will be used to represent this physical drive in other commands.
+
Actually this slot number will be used to represent this physical '''ID''' drive in other commands.
  
 
==Gather info about Virtual drives==
 
==Gather info about Virtual drives==

Revision as of 00:00, 15 July 2016


Contents

MegaCli

MegaCli introduced by LSI as a command line administration of LSI MegaRaid controllers .

With megacli we can create physical raids, gather info about raids and monitor raids.


Download and Install MegaCli and other supportive tools

Centos

yum install MegaCli

Command location: /opt/MegaRAID/MegaCli/MegaCli64

make an alias for easier use:

alias megacli='/opt/MegaRAID/MegaCli/MegaCli64'

Ubuntu

apt-get install megacli

Command Location: /usr/sbin/megacli, no need alias.

Extra tools

MegaCli not providing all the information we need like mapping to linux devices and raid level (readable), so we are going to use some extra tools.

Centos

yum install sg3_utils

Ubuntu

apt-get install sg3-utils megactl

Megacli Concepts

Adapters, Physical Drives and Virtual Drives

Before we go through megacli commands we need to follow megacli concepts.

Adapter - The physical controller which we are going to use, represented by id (usually 0).

Enclosure - The physical chassis the physical drives attached to, represented by id, such as 254,252 etc.

Physical Drives - Physical Hard Disks attached to controller, represented by id, 0,1,2,3 etc.

Virtual Drives - Those drives contains Physical Drives and equal to Raid Devices, represented by id, 0,1,2,3 etc.

For example if we have RAID 0 over 3 Physical Drives, we get:

Physical Drives ids: 0,1,2

Virtual Drive id: 0

Virtual Drive contains physical drives 0,1 and 2 and includes the settings of the raid device such as raid level, strip size etc.

We can see configurations in megacli like Virtual Drive with RAID 0 over one physical drive, we have this settings because physical drives attached to raid controller, and for representing the device to the system we must set it in megacli.

As a default we will see all the physical devices part of virtual drive in raid 0.


Gather the information we need

Gather info about raid controller

Make sure you have raid controller attached to your server, using lspci.

lspci | grep -i raid

Output example:

81:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] (rev 05)

Now that we know we have MegaRaid controller we can go on and work with megacli CLI.

Gather info about raid adapters:

megacli -AdpGetPciInfo -aAll

Output example:

PCI information for Controller 0
--------------------------------
Bus Number : 5
Device Number : 0
Function Number : 0
Exit Code: 0x00

We can see here the adapters information, we will use the adapter id later on, adapter id is 0.

Gather info about the enclosure

megacli -EncInfo -a0

Output example:

Number of enclosures on adapter 0 -- 1
Enclosure 0:
Device ID : 245
Number of Slots : 24
Number of Power Supplies : 0
Number of Fans : 1
Number of Temperature Sensors : 3
Number of Alarms : 0
Number of SIM Modules : 0
Number of Physical Drives : 10
Status : Normal
Position : 1
Connector Name : Unavailable
Enclosure type : SGPIO
VendorId is LSI CORP and Product Id is Bobcat
VendorID and Product ID didnt match
FRU Part Number : N/A
Enclosure Serial Number : N/A
ESM Serial Number : N/A
Enclosure Zoning Mode : N/A
Partner Device Id : 65535
Inquiry data :
Vendor Identification : LSI CORP
Product Identification : Bobcat
Product Revision Level : 0504
Vendor Specific : x36-25.5.4.0
Exit Code: 0x00

Lets look over some of the values:

Device ID : 245

This id represent the enclosure and will be used in other commands.


Number of Slots : 24

The maximum physical drives we can connect to this enclosure.


Gather info about physical drives

Adapter id is 0.

megacli -LdPdInfo -a0

Output example:

PD: 0 Information
Enclosure Device ID: 245
Slot Number: 0
Drive's postion: DiskGroup: 0, Span: 0, Arm: 0
Enclosure position: 0
Device Id: 20
WWN: 5000C5006B1ECCA8
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 1.090 TB [0x8bba0cb0 Sectors]
Non Coerced Size: 1.090 TB [0x8baa0cb0 Sectors]
Coerced Size: 1.090 TB [0x8baa0000 Sectors]
Firmware state: Online, Spun Up
Is Commissioned Spare : NO
Device Firmware Level: 0002
Shield Counter: 0
Successful diagnostics completion on : N/A
SAS Address(0): 0x5000c5006b1ecca9
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: SEAGATE ST1200MM0017 0002S3L02PGK
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :34C (93.20 F)
PI Eligibility: No
Drive is formatted for PI information: No
PI: No PI
Drive's write cache : Disabled
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Port-1 :
Port status: Active
Port's Linkspeed: Unknown
Drive has flagged a S.M.A.R.T alert : No
Lets look over some of the properties:
Enclosure Device ID: 245
The enclosure id that the physical drive connected to.

Slot Number: 0

This is the slot number the physical drive is connected to.

Actually this slot number will be used to represent this physical ID drive in other commands.

Gather info about Virtual drives

megacli -LDInfo -Lall -a0

Output example:

Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 1.090 TB
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
Is VD Cached: No
Target Id: 0

This id represent the virtual drive its created within the raid creation.


Size : 1.090 TB

Size of the raid device, this size is the actual allocatable size.

For two devices that each size is 1TB at RAID 1, the virtual drive size will be 1TB, because the drives in mirror raid.

About physical drives inside virtual drives

As we look for simple output that shows the virtual drives structure, its hard to get that with megacli.

One way is to manipulate the output of the above command:

megacli -LdPdInfo -a0 | grep -E "Virtual Drive:|Slot Number:" | xargs | sed -r 's/(Slot Number:)(\s[0-9]+)/\2,/g' | sed 's/(Target Id: .)/Physical Drives ids:/g' | sed 's/Virtual Drive:/\nVirtual Drive:/g'

Output example:

Virtual Drive: 0 Physical Drives ids: 0,
Virtual Drive: 1 Physical Drives ids: 1,
Virtual Drive: 2 Physical Drives ids: 2, 3, 4, 5, 6, 7, 8, 9,

This manipulation is ugly and unnecessary since we can simply use megasasctl.

megasasctl

Output example:

a0 LSI MegaRAID SAS 9260-16i encl:1 ldrv:3 batt:FAULT, unknown charge state
a0d0 1TiB RAID 0 1x1 optimal
a0d1 1TiB RAID 0 1x1 optimal
a0d2 4TiB RAID 10 4x2 optimal
a0e245s0 1TiB a0d0 online
a0e245s1 1TiB a0d1 online
a0e245s2 1TiB a0d2 online
a0e245s3 1TiB a0d2 online
a0e245s4 1TiB a0d2 online
a0e245s5 1TiB a0d2 online
a0e245s6 1TiB a0d2 online
a0e245s7 1TiB a0d2 online
a0e245s8 1TiB a0d2 online
a0e245s9 1TiB a0d2 online

Lets look over one of the lines representing virtual drive:

a0d0 1TiB RAID 0 1x1 optimal

Explanation:

Adapter: 0

Virtual Drive: 0

Raid Size: 1TB

Raid level: 0

Line that represent physical drive:

a0e245s0 1TiB a0d0 online

Explanation:

Adapter: 0

Enclosure id: 245

Slot: 0

Size: 1TB

And this physical drive is part of Adapter 0 Virtual Drive 0 (a0d0).

Gather info about raid level

As we familiar with megasasctl command we can get information about raid level easily.

MegaCli providing information about the raid level also:

megacli -LDInfo -L2 -a0 | grep -i raid

Output example:

RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0

We can see here Primary raid level 1 and secondary 0. It means actually raid 0 over raid 1 that is RAID 10.

We find this output hard to understand this is why prefer using megasasctl. As explained above:

a0d2 4TiB RAID 10 4x2 optimal

Mapping virtual drives to linux devices

Eventually we want to get the information about the mapping between linux devices and virtual drives (raid devices).

We already know that virtual drives represented by ids.

We will use sg_map command:

sg_map -x

Output example:

/dev/sg0 0 1 117 0 13
/dev/sg1 0 2 0 0 0 /dev/sda
/dev/sg2 0 2 1 0 0 /dev/sdb
/dev/sg3 0 2 2 0 0 /dev/sdc

Explanation:

 sg_device_name  host_number  bus  scsi_id  lun  scsi_type  linux_device_name
 /dev/sg1        0            2    0        0    0          /dev/sda

scsi_id - Equals to virtual drive id, this is all we need to know the mapping between virtual drives to linux devices as shown above.

Create RAID Device (Virtual Drive)

Before we create raid device we need to gather some information as I explained above.

Lets use this information for the example:

Adpater id: 0
Enclosure id: 245
Physical Drive ids: 3,4
Raid Level: 0

The command syntax is:

megacli -CfgLdAdd -rX[enclosure_id:physical_id,enclosure_id:physical_id] -aN

X= Raid level

N= adapter id

Example:

megacli -CfgLdAdd -r0[245:3,245:4] -a0

Output example:

Adapter 0: Created VD 2
Adapter 0: Configured the Adapter!!
Exit Code: 0x00

We can see that virtual drive 2 created.

Create RAID 10 Device (Virtual Drive)

Creating raid 10 device is different because we have to write the exact pairs for RAID 1 and over them there is going to be RAID 0.

We also using other flag of megacli command.

Example:

megacli -CfgSpanAdd -r10 -Array0[245:2,245:3] -Array1[245:4,245:5] -a0

Delete Raid Device (Virtual Drive)

In case we want to delete raid device we need to determine the Virtual Drive id and adapter id.

Make sure there is no data on the disk before deletion, also device needs to be unmounted and out of /etc/fstab .

Lets delete Virtual Drive 2.

Example:

megacli -CfgLdDel -L2 -a0

Output example:

Adapter 0: Deleted Virtual Drive-2(target id-2)
Exit Code: 0x00
Personal tools