SecureRandom.generateSeed() getting hanged on Tomcat startup in Linux

Another mysterious production Issue !!!

In production environment, on one day our Tomcat web servers start getting hanged without any provocation. No activity is happening in the server , but still its hung and we are not even able to take the tomcat console. Analyzing the thread dump points to the below suspicious thread stack

  1. "main" #1 prio=5 os_prio=0 tid=0x00007fcbb8009800 nid=0x6 runnable [0x00007fcbc0b1b000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:539) at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:144) at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:139) at java.security.SecureRandom.generateSeed(SecureRandom.java:533)

    It seems like thread is waiting for something to return from the readBytes.

    Root Cause Of the Issue

    For generating secured random numbers,SecureRandom.generateSeed() method internally uses
    Linux random number generator which uses entropy pool with entropy count for its generation

    There are two ways 

  2.  True Random Number generator , which refers to /dev/random. It takes the random number straight out of the entropy pool. If its not avaialble, it will block util a new entropy is entered in the pool.
  3. /dev/urandom cryptographically safe pseudo random generator (CSPRG) which is better than /dev/random because it does not block.  If sufficient entropy is available, it will provide random numbers just as strong as /dev/random, if not, it uses the SHA cryptographic hash algorithm to generate very strong random numbers

In my case,the default value of the generator is set in the path <JRE_HOME>/lib/security/ java.security  as /dev/random , so it was blocking the complete tomcat process during its startup since the entropy level in the pool was very less. 

To check the available entropy in Linux , execute the following command

cat /proc/sys/kernel/random/entropy_avail

If it returns value less than 200, you have a problem. In my case, it was 1.


Solution

The solution is to use the /dev/urandom random generator by explictly giving the following java option in the server startup

-Djava.security.egd=file:/dev/./urandom

References:
http://www.2uo.de/myths-about-urandom/
http://stackoverflow.com/questions/137212/how-to-solve-performance-problem-with-java-securerandom
https://blog.virtualzone.de/2011/10/javas-securerandomgenerateseed-on-linux.html
https://bugs.openjdk.java.net/browse/JDK-6202721
https://lwn.net/Articles/261804/

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s