Hardware Raid Setup using MegaCli

From Linux Raid Wiki
(Difference between revisions)
Jump to: navigation, search
 
Line 380: Line 380:
 
  Adapter 0: Deleted Virtual Drive-2(target id-2)
 
  Adapter 0: Deleted Virtual Drive-2(target id-2)
 
  Exit Code: 0x00
 
  Exit Code: 0x00
 +
 +
=Cache Policy=
 +
 +
Cache Policy's are how the raid card uses on board RAM to collect data before writing out to disk or to read data before the system asks for it.
 +
 +
Write cache is used when we have a lot of data to write and it is faster to write data sequentially to disk instead of writing small chunks.
 +
 +
Read cache is used when the system has asked for some data and the raid card keeps the data in cache in case the system asks for the same data again.
 +
 +
It is always faster to read and write to cache then to access spinning disks. Understand that you should only use caching if you have good UPS power to the system.
 +
 +
If the system looses power and does not flush the cache it is possible to loose data. No one wants that. Lets look at each cache policy LSI raid card use.
 +
 +
*WriteBack uses the card's cache to collect enough data to make a series of long sequential writes out to disk. This is the fastest write method.
 +
 +
*WriteThrough tells the card to write all data directly to disk without cache. This method is quite slow by about 1/10 the speed of WriteBack, but is safer as no data can be lost that was in cache when the machine's power fails.
 +
 +
*ReadAdaptive uses an algorithm to see if when the OS asks for a bunch of data blocks sequentially, if we should read a few more sequential blocks because the OS _might_ ask for those too. This method can lead to good speed increases.
 +
 +
*ReadAheadNone tells the raid card to only read the data off the raid disk if it was actually asked for. No more, no less.
 +
 +
*Cached allows the general use of the cards cache for any data which is read or written. Very efficient if the same data is accessed over and over again.
 +
 +
*Direct is straight access to the disk without ever storing data in the cache. This can be slow as any I/O has to touch the disk platters.
 +
 +
*Write Cache OK if Bad BBU tells the card to use write caching even if the Battery Backup Unit (BBU) is bad, disabled or missing. This is a good setting if your raid card's BBU charger is bad, if you do not want or can't to replace the BBU or if you do not want WriteThrough enabled during a BBU
 +
relearn test.
 +
 +
*No Write Cache if Bad BBU if the BBU is not available for any reason then disable WriteBack and turn on WriteThrough. This option is safer for your data, but the raid card will switch to WriteThrough during a battery relearn cycle.
 +
 +
*Disk Cache Policy: Enabled Use the hard drive's own cache. For example if data is written out the drives this option lets the drives themselves cache data internally before writing data to its platters.
 +
 +
*Disk Cache Policy: Disabled does not allow the drive to use any of its own internal cache.

Latest revision as of 11:02, 15 July 2016


Contents

[edit] MegaCli

MegaCli introduced by LSI as a command line administration of LSI MegaRaid controllers .

With megacli we can create physical raids, gather info about raids and monitor raids.


[edit] Download and Install MegaCli and other supportive tools

[edit] Centos

yum install MegaCli

Command location: /opt/MegaRAID/MegaCli/MegaCli64

make an alias for easier use:

alias megacli='/opt/MegaRAID/MegaCli/MegaCli64'

[edit] Ubuntu

apt-get install megacli

Command Location: /usr/sbin/megacli, no need alias.

[edit] Extra tools

MegaCli not providing all the information we need like mapping to linux devices and raid level (readable), so we are going to use some extra tools.

[edit] Centos

yum install sg3_utils

[edit] Ubuntu

apt-get install sg3-utils megactl

[edit] Megacli Concepts

[edit] Adapters, Physical Drives and Virtual Drives

Before we go through megacli commands we need to follow megacli concepts.

Adapter - The physical controller which we are going to use, represented by id (usually 0).

Enclosure - The physical chassis the physical drives attached to, represented by id, such as 254,252 etc.

Physical Drives - Physical Hard Disks attached to controller, represented by id, 0,1,2,3 etc.

Virtual Drives - Those drives contains Physical Drives and equal to Raid Devices, represented by id, 0,1,2,3 etc.

For example if we have RAID 0 over 3 Physical Drives, we get:

Physical Drives ids: 0,1,2

Virtual Drive id: 0

Virtual Drive contains physical drives 0,1 and 2 and includes the settings of the raid device such as raid level, strip size etc.

We can see configurations in megacli like Virtual Drive with RAID 0 over one physical drive, we have this settings because physical drives attached to raid controller, and for representing the device to the system we must set it in megacli.

As a default we will see all the physical devices part of virtual drive in raid 0.


[edit] Gather the information we need

[edit] Gather info about raid controller

Make sure you have raid controller attached to your server, using lspci.

lspci | grep -i raid

Output example:

81:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] (rev 05)

Now that we know we have MegaRaid controller we can go on and work with megacli CLI.

[edit] Gather info about raid adapters:

megacli -AdpGetPciInfo -aAll

Output example:

PCI information for Controller 0
--------------------------------
Bus Number : 5
Device Number : 0
Function Number : 0
Exit Code: 0x00

We can see here the adapters information, we will use the adapter id later on, adapter id is 0.

[edit] Gather info about the enclosure

megacli -EncInfo -a0

Output example:

Number of enclosures on adapter 0 -- 1
Enclosure 0:
Device ID : 245
Number of Slots : 24
Number of Power Supplies : 0
Number of Fans : 1
Number of Temperature Sensors : 3
Number of Alarms : 0
Number of SIM Modules : 0
Number of Physical Drives : 10
Status : Normal
Position : 1
Connector Name : Unavailable
Enclosure type : SGPIO
VendorId is LSI CORP and Product Id is Bobcat
VendorID and Product ID didnt match
FRU Part Number : N/A
Enclosure Serial Number : N/A
ESM Serial Number : N/A
Enclosure Zoning Mode : N/A
Partner Device Id : 65535
Inquiry data :
Vendor Identification : LSI CORP
Product Identification : Bobcat
Product Revision Level : 0504
Vendor Specific : x36-25.5.4.0
Exit Code: 0x00

Lets look over some of the values:

Device ID : 245

This id represent the enclosure and will be used in other commands.


Number of Slots : 24

The maximum physical drives we can connect to this enclosure.


[edit] Gather info about physical drives

Adapter id is 0.

megacli -LdPdInfo -a0

Output example:

PD: 0 Information
Enclosure Device ID: 245
Slot Number: 0
Drive's postion: DiskGroup: 0, Span: 0, Arm: 0
Enclosure position: 0
Device Id: 20
WWN: 5000C5006B1ECCA8
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 1.090 TB [0x8bba0cb0 Sectors]
Non Coerced Size: 1.090 TB [0x8baa0cb0 Sectors]
Coerced Size: 1.090 TB [0x8baa0000 Sectors]
Firmware state: Online, Spun Up
Is Commissioned Spare : NO
Device Firmware Level: 0002
Shield Counter: 0
Successful diagnostics completion on : N/A
SAS Address(0): 0x5000c5006b1ecca9
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: SEAGATE ST1200MM0017 0002S3L02PGK
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :34C (93.20 F)
PI Eligibility: No
Drive is formatted for PI information: No
PI: No PI
Drive's write cache : Disabled
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Port-1 :
Port status: Active
Port's Linkspeed: Unknown
Drive has flagged a S.M.A.R.T alert : No
Lets look over some of the properties:
Enclosure Device ID: 245
The enclosure id that the physical drive connected to.

Slot Number: 0

This is the slot number the physical drive is connected to.

Actually this slot number will be used to represent this physical ID drive in other commands.

[edit] Gather info about Virtual drives

megacli -LDInfo -Lall -a0

Output example:

Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 1.090 TB
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
Is VD Cached: No
Target Id: 0

This id represent the virtual drive its created within the raid creation.


Size : 1.090 TB

Size of the raid device, this size is the actual allocatable size.

For two devices that each size is 1TB at RAID 1, the virtual drive size will be 1TB, because the drives in mirror raid.

[edit] About physical drives inside virtual drives

As we look for simple output that shows the virtual drives structure, its hard to get that with megacli.

One way is to manipulate the output of the above command:

megacli -LdPdInfo -a0 | grep -E "Virtual Drive:|Slot Number:" | xargs | sed -r 's/(Slot Number:)(\s[0-9]+)/\2,/g' | sed 's/(Target Id: .)/Physical Drives ids:/g' | sed 's/Virtual Drive:/\nVirtual Drive:/g'

Output example:

Virtual Drive: 0 Physical Drives ids: 0,
Virtual Drive: 1 Physical Drives ids: 1,
Virtual Drive: 2 Physical Drives ids: 2, 3, 4, 5, 6, 7, 8, 9,

This manipulation is ugly and unnecessary since we can simply use megasasctl.

megasasctl

Output example:

a0 LSI MegaRAID SAS 9260-16i encl:1 ldrv:3 batt:FAULT, unknown charge state
a0d0 1TiB RAID 0 1x1 optimal
a0d1 1TiB RAID 0 1x1 optimal
a0d2 4TiB RAID 10 4x2 optimal
a0e245s0 1TiB a0d0 online
a0e245s1 1TiB a0d1 online
a0e245s2 1TiB a0d2 online
a0e245s3 1TiB a0d2 online
a0e245s4 1TiB a0d2 online
a0e245s5 1TiB a0d2 online
a0e245s6 1TiB a0d2 online
a0e245s7 1TiB a0d2 online
a0e245s8 1TiB a0d2 online
a0e245s9 1TiB a0d2 online

Lets look over one of the lines representing virtual drive:

a0d0 1TiB RAID 0 1x1 optimal

Explanation:

Adapter: 0

Virtual Drive: 0

Raid Size: 1TB

Raid level: 0

Line that represent physical drive:

a0e245s0 1TiB a0d0 online

Explanation:

Adapter: 0

Enclosure id: 245

Slot: 0

Size: 1TB

And this physical drive is part of Adapter 0 Virtual Drive 0 (a0d0).

[edit] Gather info about raid level

As we familiar with megasasctl command we can get information about raid level easily.

MegaCli providing information about the raid level also:

megacli -LDInfo -L2 -a0 | grep -i raid

Output example:

RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0

We can see here Primary raid level 1 and secondary 0. It means actually raid 0 over raid 1 that is RAID 10.

We find this output hard to understand this is why prefer using megasasctl. As explained above:

a0d2 4TiB RAID 10 4x2 optimal

[edit] Mapping virtual drives to linux devices

Eventually we want to get the information about the mapping between linux devices and virtual drives (raid devices).

We already know that virtual drives represented by ids.

We will use sg_map command:

sg_map -x

Output example:

/dev/sg0 0 1 117 0 13
/dev/sg1 0 2 0 0 0 /dev/sda
/dev/sg2 0 2 1 0 0 /dev/sdb
/dev/sg3 0 2 2 0 0 /dev/sdc

Explanation:

 sg_device_name  host_number  bus  scsi_id  lun  scsi_type  linux_device_name
 /dev/sg1        0            2    0        0    0          /dev/sda

scsi_id - Equals to virtual drive id, this is all we need to know the mapping between virtual drives to linux devices as shown above.

[edit] Create RAID Device (Virtual Drive)

Before we create raid device we need to gather some information as I explained above.

Lets use this information for the example:

Adpater id: 0
Enclosure id: 245
Physical Drive ids: 3,4
Raid Level: 0

The command syntax is:

megacli -CfgLdAdd -rX[enclosure_id:physical_id,enclosure_id:physical_id] -aN

X= Raid level

N= adapter id

Example:

megacli -CfgLdAdd -r0[245:3,245:4] -a0

Output example:

Adapter 0: Created VD 2
Adapter 0: Configured the Adapter!!
Exit Code: 0x00

We can see that virtual drive 2 created.

[edit] Create RAID 10 Device (Virtual Drive)

Creating raid 10 device is different because we have to write the exact pairs for RAID 1 and over them there is going to be RAID 0.

We also using other flag of megacli command.

Example:

megacli -CfgSpanAdd -r10 -Array0[245:2,245:3] -Array1[245:4,245:5] -a0

[edit] Delete Raid Device (Virtual Drive)

In case we want to delete raid device we need to determine the Virtual Drive id and adapter id.

Make sure there is no data on the disk before deletion, also device needs to be unmounted and out of /etc/fstab .

Lets delete Virtual Drive 2.

Example:

megacli -CfgLdDel -L2 -a0

Output example:

Adapter 0: Deleted Virtual Drive-2(target id-2)
Exit Code: 0x00

[edit] Cache Policy

Cache Policy's are how the raid card uses on board RAM to collect data before writing out to disk or to read data before the system asks for it.

Write cache is used when we have a lot of data to write and it is faster to write data sequentially to disk instead of writing small chunks.

Read cache is used when the system has asked for some data and the raid card keeps the data in cache in case the system asks for the same data again.

It is always faster to read and write to cache then to access spinning disks. Understand that you should only use caching if you have good UPS power to the system.

If the system looses power and does not flush the cache it is possible to loose data. No one wants that. Lets look at each cache policy LSI raid card use.

  • WriteBack uses the card's cache to collect enough data to make a series of long sequential writes out to disk. This is the fastest write method.
  • WriteThrough tells the card to write all data directly to disk without cache. This method is quite slow by about 1/10 the speed of WriteBack, but is safer as no data can be lost that was in cache when the machine's power fails.
  • ReadAdaptive uses an algorithm to see if when the OS asks for a bunch of data blocks sequentially, if we should read a few more sequential blocks because the OS _might_ ask for those too. This method can lead to good speed increases.
  • ReadAheadNone tells the raid card to only read the data off the raid disk if it was actually asked for. No more, no less.
  • Cached allows the general use of the cards cache for any data which is read or written. Very efficient if the same data is accessed over and over again.
  • Direct is straight access to the disk without ever storing data in the cache. This can be slow as any I/O has to touch the disk platters.
  • Write Cache OK if Bad BBU tells the card to use write caching even if the Battery Backup Unit (BBU) is bad, disabled or missing. This is a good setting if your raid card's BBU charger is bad, if you do not want or can't to replace the BBU or if you do not want WriteThrough enabled during a BBU

relearn test.

  • No Write Cache if Bad BBU if the BBU is not available for any reason then disable WriteBack and turn on WriteThrough. This option is safer for your data, but the raid card will switch to WriteThrough during a battery relearn cycle.
  • Disk Cache Policy: Enabled Use the hard drive's own cache. For example if data is written out the drives this option lets the drives themselves cache data internally before writing data to its platters.
  • Disk Cache Policy: Disabled does not allow the drive to use any of its own internal cache.
Personal tools