Solving reliability and scalability problems with iSCSI, part 2

From lxadm | Linux administration tips, tutorials, HOWTOs and articles
Jump to: navigation, search

The latest stable IET release, 0.4.15, suffers yet another misfeature: it will likely break all initiators when the ietd process is restarted (ietd restart, machine restart etc.).

This is because on ietd shutdown, the user space daemon is just killed, but it doesn’t have the corresponding signal handler and the kernel space module doesn’t perform any cleanups for it.

From initiator’s perspective, several things may happen:

  • the change will happen so fast that the initiator won’t notice anything
  • initiator will break the connection, but after reconnection, you will see your iSCSI drives remounted read-only, weird hangs etc.
  • initiator won’t be able to connect again

Of course, no one wants to have the filesystem remounted read only, or to have the hard drive ripped off from a working system (this is how it looks from the perspective of the kernel if we loose a iSCSI connection).

There is a simple solution to that, although some may say it’s an ugly hack.

Using iptables to block the traffic to and from the IET target just before it shuts down does the job – the change is trivial – in /etc/init.d/iscsi-target, add iptables lines as follows:

echo -n "Starting iSCSI enterprise target service: "
modprobe -q crc32c
modprobe iscsi_trgt
start-stop-daemon --start --exec $DAEMON --quiet
sleep 3s
iptables -D OUTPUT -p tcp --dport 3260 -j DROP
iptables -D INPUT -p tcp --dport 3260 -j DROP

if [ $RETVAL == “0” ]; then
echo “succeeded.”
echo “failed.”

iptables -A OUTPUT -p tcp –dport 3260 -j DROP
iptables -A INPUT -p tcp –dport 3260 -j DROP
echo -n “Removing iSCSI enterprise target devices: ”
# ugly, but ietadm does not allways provides correct exit values
RETURN=`ietadm –op delete 2>&1`

With it, and with changes described in part 1, your IET iSCSI target installation should be really bullet-proof.