Poor write performance on Fusion MPT SAS RAID controllers

From lxadm | Linux administration tips, tutorials, HOWTOs and articles
Jump to: navigation, search

If you have a hardware RAID controller using mptsas module on your Linux server, you may notice your write performance is extremely poor – i.e. some database workloads can be 30x slower than on other servers (i.e. using Linux software RAID).

An example card below – lspci shows:

02:05.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068 PCI-X Fusion-MPT SAS (rev 01)

How can you fix it? See below.

The issue happens because write caching is by default disabled on drives behind the RAID – and there is no way to enable it in controller firmware / BIOS, or by running scsi-config command.

To enable write caching, you may use below commands in your startup scripts:

# create a /dev/sg0 node first
mknod /dev/sg0 c 21 0

# this one should create /dev/sg1 and /dev/sg2 nodes, too (you'll find sg_map in sg3-utils package, or, sg3_utils, if you're using RedHat / CentOS)
sg_map -i -x

# finally, enable write cache - assuming you have two drives; /dev/sg2 would be the controller itself
sdparm -s WCE=1 /dev/sg0
sdparm -s WCE=1 /dev/sg1

That’s it! Performance should be back to normal. Of course you have to be aware of the dangers of write-caching and potential data loss on unexpected power outage.

Other, useful commands:

# modprobe mptctl

# mpt-status
ioc0 vol_id 0 type IM, 2 phy, 231 GB, state OPTIMAL, flags ENABLED
ioc0 phy 1 scsi_id 8 ATA      ST3250310AS      A   , 232 GB, state ONLINE, flags NONE
ioc0 phy 0 scsi_id 1 ATA      ST3250310AS      A   , 232 GB, state ONLINE, flags NONE

# lsscsi
[4:0:0:0] disk ATA ST3250310AS A -
[4:0:1:0] disk ATA ST3250310AS A -
[4:1:0:0] disk LSILOGIC Logical Volume 3000 /dev/sda

You can also add /dev/sg0 and /dev/sg1 to monitoring by smartmontools (/etc/smartd.conf).