Testing RAM on a server without rebooting to run memtest86+

From lxadm | Linux administration tips, tutorials, HOWTOs and articles
Jump to: navigation, search

Sometimes, applications on your server crash in mysterious ways, or your server hangs without any apparent reason. You suspect that the RAM may be broken, so memtest86+ would be ideal tool to check RAM – however, there are some problems involved with it:

  • running memtest86+ means server downtime,
  • it may not be possible to run memtest86+ on a remote server without KVM-IP, iLO or similar access.

What to do in such situations? User-space memtester to the rescue!

The obvious disadvantage is that you will not be able to test all RAM in the server, but sometimes, it’s not needed to determine if RAM is good or not.

The below command will try to allocate ~15GB RAM and make one test pass – if you see errors like below, your RAM is hosed!

Note that with the exact command line shown below you should have at least 15 GB of free, unallocated memory, otherwise, you’re likely to hang your server, or at least cause a serious downtime or OOM-killer. Please also note that memtester will slow down your server a lot, since it will move lots of data around in RAM.

# memtester 15000 1
memtester version 4.0.8 (64-bit)
Copyright (C) 2007 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 15000MB (15728640000 bytes)
got  15000MB (15728640000 bytes), trying mlock ...locked.
Loop 1/1:
  Stuck Address       : testing   0FAILURE: possible bad address line at offset 0x5d922ec8.
Skipping to next test...
  Random Value        : ok
FAILURE: 0x7f40f12dd82264da != 0x7f40f52dd82264da at offset 0x22fa2d27.
FAILURE: 0xe800dc254863429d != 0xe8005c254863429d at offset 0x22fa2de7.
FAILURE: 0x95005ca5bc351fd0 != 0x95007ca5bc351fd0 at offset 0x22fa2e47.
FAILURE: 0x16cc8129822daa5 != 0x16c4a129822daa5 at offset 0x22fa2e5f.
FAILURE: 0xce8945572424d79a != 0xce89c5572424d79a at offset 0x22fa2f1f.
FAILURE: 0x14054d669820e595 != 0x1405cd669820e595 at offset 0x22fa306f.
FAILURE: 0x8d283c130c69dc25 != 0x8d28bc130c69dc25 at offset 0x22fa309f.
FAILURE: 0x19285abae23f881 != 0x19287abae23f881 at offset 0x22fa30b7.
FAILURE: 0x55302b281ab13a51 != 0x55302f281ab13a51 at offset 0x22fa31bf.
FAILURE: 0x48b8ae1715e5084b != 0x48b8aa1715e5084b at offset 0x22fa31d7.
FAILURE: 0x28a2b64b9039074a != 0x28a2164b9039074a at offset 0x22fa5db7.
FAILURE: 0xc6080cfa18200698 != 0xc6080efa18200698 at offset 0x22fa5dcf.
FAILURE: 0x2b30fefcd69a347 != 0x2b38fefcd69a347 at offset 0x22fa5de7.
FAILURE: 0xbc27b13358294271 != 0xbc27b33358294271 at offset 0x22fa5e2f.
FAILURE: 0xc17e0d24dc21fe7e != 0xc17e0f24dc21fe7e at offset 0x22fa5e47.
FAILURE: 0x29ac181590911ae3 != 0x29ac1a1590911ae3 at offset 0x22fa5f07.
FAILURE: 0x233001ba38a53ccd != 0x233021ba38a53ccd at offset 0x22fa5f37.
FAILURE: 0x7542cd4b38a3ea9f != 0x7542c94b38a3ea9f at offset 0x22fa5f7f.
FAILURE: 0x86c89d49992151c8 != 0x86c8bd49992151c8 at offset 0x22fa5ff7.
FAILURE: 0x16208f9b584969b3 != 0x16200f9b584969b3 at offset 0x22fa609f.
FAILURE: 0x6a08171eb0a18751 != 0x6a08151eb0a18751 at offset 0x22fa60b7.
FAILURE: 0x8214f12e9a254014 != 0x8214f72e9a254014 at offset 0x22fa60cf.
FAILURE: 0x8005ab68021ad9f != 0x800dab68021ad9f at offset 0x22fa612f.
FAILURE: 0x884e76469ba12421 != 0x884e72469ba12421 at offset 0x22fa61a7.
FAILURE: 0x30509c65d890299e != 0x30509e65d890299e at offset 0x22fa62af.
  Compare XOR         : FAILURE: 0x3f81821efe3ced18 != 0x3f81861efe3ced18 at offset 0x22fa2d27.
FAILURE: 0xa8416d166e7dcadb != 0xa840ed166e7dcadb at offset 0x22fa2de7.
FAILURE: 0x5540ed96e24fa80e != 0x55410d96e24fa80e at offset 0x22fa2e47.
FAILURE: 0xc1ad5903be3d62e3 != 0xc1acdb03be3d62e3 at offset 0x22fa2e5f.
FAILURE: 0x8ec9d6484a3f5fd8 != 0x8eca56484a3f5fd8 at offset 0x22fa2f1f.
FAILURE: 0xd445de57be3b6dd3 != 0xd4465e57be3b6dd3 at offset 0x22fa306f.
FAILURE: 0x4d68cd0432846463 != 0x4d694d0432846463 at offset 0x22fa309f.
FAILURE: 0xc1d3169cd43e80bf != 0xc1d3189cd43e80bf at offset 0x22fa30b7.
FAILURE: 0x1570bc1940cbc28f != 0x1570c01940cbc28f at offset 0x22fa31bf.
FAILURE: 0x8f93f083bff9089 != 0x8f93b083bff9089 at offset 0x22fa31d7.
FAILURE: 0xe8e3473cb6538f88 != 0xe8e2a73cb6538f88 at offset 0x22fa5db7.
FAILURE: 0x86489deb3e3a8ed6 != 0x86489feb3e3a8ed6 at offset 0x22fa5dcf.
FAILURE: 0xc2f3a0e0f3842b85 != 0xc2f420e0f3842b85 at offset 0x22fa5de7.
FAILURE: 0x7c6842247e43caaf != 0x7c6844247e43caaf at offset 0x22fa5e2f.
FAILURE: 0x81be9e16023c86bc != 0x81bea016023c86bc at offset 0x22fa5e47.
FAILURE: 0xe9eca906b6aba321 != 0xe9ecab06b6aba321 at offset 0x22fa5f07.
FAILURE: 0xe37092ab5ebfc50b != 0xe370b2ab5ebfc50b at offset 0x22fa5f37.
FAILURE: 0x35835e3c5ebe72dd != 0x35835a3c5ebe72dd at offset 0x22fa5f7f.
FAILURE: 0x47092e3abf3bda06 != 0x47094e3abf3bda06 at offset 0x22fa5ff7.
FAILURE: 0xd661208c7e63f1f1 != 0xd660a08c7e63f1f1 at offset 0x22fa609f.
FAILURE: 0x2a48a80fd6bc0f8f != 0x2a48a60fd6bc0f8f at offset 0x22fa60b7.
FAILURE: 0x4255821fc03fc852 != 0x4255881fc03fc852 at offset 0x22fa60cf.
FAILURE: 0xc840eba7a63c35dd != 0xc8416ba7a63c35dd at offset 0x22fa612f.
FAILURE: 0x488f0737c1bbac5f != 0x488f0337c1bbac5f at offset 0x22fa61a7.
FAILURE: 0xf0912d56feaab1dc != 0xf0912f56feaab1dc at offset 0x22fa62af.
  Compare SUB         : FAILURE: 0x770f6d7d59b31bc0 != 0x64680d7d59b31bc0 at offset 0x22fa2d27.
FAILURE: 0x17cc5d24cac64438 != 0x6cb85d24cac64438 at offset 0x22fa2de7.
FAILURE: 0x8bda06d04720f630 != 0xf69f06d04720f630 at offset 0x22fa2e47.
FAILURE: 0x191ec89353a0b578 != 0xe4b7189353a0b578 at offset 0x22fa2e5f.
FAILURE: 0xb1bc42b59ee189c0 != 0x5cd042b59ee189c0 at offset 0x22fa2f1f.
FAILURE: 0x86575304584f0af8 != 0x316b5304584f0af8 at offset 0x22fa306f.
FAILURE: 0x4d053942f1b9f178 != 0xf8193942f1b9f178 at offset 0x22fa309f.
FAILURE: 0x206d80c97f8947d8 != 0x9719d0c97f8947d8 at offset 0x22fa30b7.
FAILURE: 0x6cf332350ca77058 != 0x5a4bd2350ca77058 at offset 0x22fa31bf.
FAILURE: 0xcae418d26b8f9b68 != 0xdd8b78d26b8f9b68 at offset 0x22fa31d7.
FAILURE: 0xdc58887f8f9e1d40 != 0xc67f887f8f9e1d40 at offset 0x22fa5db7.
FAILURE: 0x66470a0f89743570 != 0xdcf35a0f89743570 at offset 0x22fa5dcf.
FAILURE: 0x678cf2f8066c7ac8 != 0x12a0f2f8066c7ac8 at offset 0x22fa5de7.
FAILURE: 0x7e705d8189837558 != 0xf51cad8189837558 at offset 0x22fa5e2f.
FAILURE: 0x62530ceb060c3560 != 0xd8ff5ceb060c3560 at offset 0x22fa5e47.
FAILURE: 0xe1f0dc5c71b99328 != 0x589d2c5c71b99328 at offset 0x22fa5f07.
FAILURE: 0xa1a2ad6a30317bb8 != 0xc67ad6a30317bb8 at offset 0x22fa5f37.
FAILURE: 0x1d86ded3b0c73088 != 0x302e3ed3b0c73088 at offset 0x22fa5f7f.
FAILURE: 0xf1b9a3e0fcfa14f0 != 0x5c7ea3e0fcfa14f0 at offset 0x22fa5ff7.
FAILURE: 0x806d7d4232efc3a8 != 0xd5597d4232efc3a8 at offset 0x22fa609f.
FAILURE: 0x98a6f4d473117858 != 0x21faa4d473117858 at offset 0x22fa60b7.
FAILURE: 0x617c86f84528d8d0 != 0xc58176f84528d8d0 at offset 0x22fa60cf.
FAILURE: 0xbca540978cefa888 != 0x67b940978cefa888 at offset 0x22fa612f.
FAILURE: 0xe6ebd2a5a59fd8d8 != 0xf99332a5a59fd8d8 at offset 0x22fa61a7.
FAILURE: 0x1a30be8e9a1fb260 != 0x90dd0e8e9a1fb260 at offset 0x22fa62af.
  Compare MUL         : ^C^C^C^C^C