Java leap second bug – 30 June / 1 July 2012 – fix

From lxadm | Linux administration tips, tutorials, HOWTOs and articles
Jump to: navigation, search

Update: the same fix works for 30 June / 1 July 2015 issue.

If your java applications suddenly started to use 100% CPU, you’ve hit java leap second bug (actually it’s a bug in Linux kernel, just java programs seem to be severely affected).

First, you may check if you have the following in your dmesg:

[10703552.860274] Clock: inserting leap second 23:59:60 UTC

Fortunately the fix is straightforward:

/etc/init.d/ntp stop
date -s "$(date)"

(the other suggested way around the web would be date `date +"%m%d%H%M%C%y.%S"`, having the same effect).

You don’t have to restart your java applications (tomcat, solr, wowza, or whatever using java, but also non-java Ruby, Redmine, collectd etc. were affected); it should be enough to run the above commands.

You may want to enable ntp the next day (there were some reports that enabling it makes java misbehave on some installations).

Apparently this issue has knocked down lots of Linux servers running Debian, Ubuntu, CentOS and the like around the globe!