Another mysterious production Issue !!!
In production environment, on one day our Tomcat web servers start getting hanged without any provocation. No activity is happening in the server , but still its hung and we are not even able to take the tomcat console. Analyzing the thread dump points to the below suspicious thread stack
"main" #1 prio=5 os_prio=0 tid=0x00007fcbb8009800 nid=0x6 runnable [0x00007fcbc0b1b000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:539) at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:144) at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:139) at java.security.SecureRandom.generateSeed(SecureRandom.java:533)
It seems like thread is waiting for something to return from the readBytes.
Root Cause Of the Issue
For generating secured random numbers,SecureRandom.generateSeed() method internally uses
Linux random number generator which uses entropy pool with entropy count for its generation
There are two ways
True Random Number generator , which refers to /dev/random. It takes the random number straight out of the entropy pool. If its not avaialble, it will block util a new entropy is entered in the pool.
/dev/urandom cryptographically safe pseudo random generator (CSPRG) which is better than /dev/random because it does not block. If sufficient entropy is available, it will provide random numbers just as strong as /dev/random, if not, it uses the SHA cryptographic hash algorithm to generate very strong random numbers
In my case,the default value of the generator is set in the path <JRE_HOME>/lib/security/ java.security as /dev/random , so it was blocking the complete tomcat process during its startup since the entropy level in the pool was very less.
To check the available entropy in Linux , execute the following command
If it returns value less than 200, you have a problem. In my case, it was 1.
The solution is to use the /dev/urandom random generator by explictly giving the following java option in the server startup