Troubleshooting Hadoop Problems – FSNamesystem: FSNamesystem initialization failed.

When Namenode starts sometime you do not see the Namenode process running so you can take a look at the logs to understand what is going on…

This is the message you get when you start namenode:

[exec] Starting namenodes on [localhost]
[exec] localhost: starting namenode, logging to */hadoop-0.23.10/logs/hadoop-avkashchauhan-namenode-Avkashs-MacBook-Pro.local.out

Now you can see the log by opening the log using */hadoop-0.23.10/logs/hadoop-avkashchauhan-namenode-Avkashs-MacBook-Pro.local.log  <- make sure use the log extension

In the log you will see the error as below:

STARTUP_MSG: java = 1.7.0_45
************************************************************/
2014-01-31 11:36:16,789 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2014-01-31 11:36:16,848 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2014-01-31 11:36:16,848 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2014-01-31 11:36:17,038 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
java.io.IOException: Missing directory /tmp/hdfs23/namenode
at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.addDirsToCheck(NameNodeResourceChecker.java:88)
at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.<init>(NameNodeResourceChecker.java:71)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:348)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:332)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:303)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:346)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:472)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:464)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:765)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:814)
2014-01-31 11:36:17,039 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system…
2014-01-31 11:36:17,039 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2014-01-31 11:36:17,040 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2014-01-31 11:36:17,040 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: Missing directory /tmp/hdfs23/namenode
at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.addDirsToCheck(NameNodeResourceChecker.java:88)
at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.<init>(NameNodeResourceChecker.java:71)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:348)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:332)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:303)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:346)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:472)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:464)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:765)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:814)
2014-01-31 11:36:17,041 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2014-01-31 11:36:17,041 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at avkashs-macbook-pro.local/10.0.0.17
************************************************************/

Step-by-step guide

  • Jump to you hadoop_install/bin folder
  • Launch command ./hadoop namenode -format
  • Make sure you see the following message showing the format was done correctly:

14/01/31 11:41:48 INFO namenode.NNStorage: Storage directory /tmp/hdfs23/namenode has been successfully formatted.
14/01/31 11:41:48 INFO namenode.FSImage: Saving image file /tmp/hdfs23/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
14/01/31 11:41:48 INFO namenode.FSImage: Image file of size 128 saved in 0 seconds.
14/01/31 11:41:48 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/01/31 11:41:48 INFO util.ExitUtil: Exiting with status 0
14/01/31 11:41:48 INFO namenode.NameNode: SHUTDOWN_MSG:

  • Now restart the namenode again.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s