Troubleshooting Cloudera Manager installation and start issues

After Cloudera Manager is installed and running you can access it over the web UI through <Your_IP_Address>:7180 , if you could not access it, then here are few ways to troubleshoot the problem. These details are helpful with you are installing Cloudera Hadoop on a remotely located machine and you just have shell access to that machine over SSH.

Verify if Cloudera Manager is running: 

[root@cloudera-master ~]# service cloudera-scm-server status
cloudera-scm-server (pid 4652) is running…

Now lets check the Cloudera Manager logs for further verification:

[root@cloudera-master ~]# ps -ef | grep cloudera-scm
498        977     1  0 16:31 ?        00:00:01 /usr/bin/postgres -D /var/lib/cloudera-scm-server-db/data
root      3729     1  0 16:59 pts/0    00:00:00 su cloudera-scm -s /bin/bash -c nohup /usr/sbin/cmf-server
498       3731  3729 53 16:59 ?        00:00:47 /usr/java/jdk1.6.0_31/bin/java -cp .:lib/*:/usr/share/java/mysql-connector-java.jar -Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties -Dcmf.root.logger=INFO,LOGFILE -Dcmf.log.dir=/var/log/cloudera-scm-server -Dcmf.log.file=cloudera-scm-server.log -Dcmf.jetty.threshhold=WARN -Dcmf.schema.dir=/usr/share/cmf/schema -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dpython.home=/usr/share/cmf -Xmx2G -XX:MaxPermSize=256m com.cloudera.server.cmf.Main
root      3835  1180  0 17:00 pts/0    00:00:00 grep cloudera-scm

You can access Cloudera Manager Logs located as below (as shown in the above commandline):

[root@cloudera-master ~]# tail -5 /var/log/cloudera-scm-server/cloudera-scm-server.log 
2014-02-06 16:59:29,596  INFO [WebServerImpl:cmon.JobDetailGatekeeper@71] ActivityMonitor configured to allow job details for all jobs.
2014-02-06 16:59:29,597  INFO [WebServerImpl:cmon.JobDetailGatekeeper@71] ActivityMonitor configured to allow job details for all jobs.
2014-02-06 16:59:29,601  INFO [WebServerImpl:mortbay.log@67] jetty-6.1.26.cloudera.2
2014-02-06 16:59:29,605  INFO [WebServerImpl:mortbay.log@67] Started SelectChannelConnector@0.0.0.0:7180
2014-02-06 16:59:29,606  INFO [WebServerImpl:cmf.WebServerImpl@280] Started Jetty server.

Lets disable Linux Machine Firewall to see if it works:

First make sure you have to fiddle with your Firewall, below I am disabling the firewall in my Linux machine:

[root@cloudera-master ~]# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
[root@cloudera-master ~]# /etc/init.d/iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@cloudera-master ~]# /etc/init.d/ip6tables save
ip6tables: Saving firewall rules to /etc/sysconfig/ip6table[  OK  ]
[root@cloudera-master ~]# /etc/init.d/ip6tables stop
ip6tables: Setting chains to policy ACCEPT: filter         [  OK  ]
ip6tables: Flushing firewall rules:                        [  OK  ]
ip6tables: Unloading modules:                              [  OK  ]
[root@cloudera-master ~]# chkconfig 
ip6tables           0:off     1:off     2:off     3:off     4:off     5:off     6:off
iptables            0:off     1:off     2:off     3:off     4:off     5:off     6:off

Once Firewall is stopped completely restart Cloudera Manager: 

If your machines are behind a firewall you can go ahead and disable the firewall on a machine which has Cloudera Manager running. If not please setup/open specific ports for Cloudera Manager to get it working.

[root@cloudera-master ~]# service –status-all | grep cloudera
cloudera-scm-server (pid  1082) is running…
/usr/bin/postgres “-D” “/var/lib/cloudera-scm-server-db/data”

[root@cloudera-master ~]# service cloudera-scm-server 
Usage: cloudera-scm-server {start|stop|restart|status}

[root@cloudera-master ~]# service cloudera-scm-server restart
Stopping cloudera-scm-server:                              [  OK  ]
Starting cloudera-scm-server:                              [  OK  ]

Note: that cloudera-scm-server is listed through chkconfig however the status shows OFF.

[root@cloudera-master ~]# chkconfig 
cloudera-scm-server     0:off     1:off     2:off     3:off     4:off     5:off     6:off
cloudera-scm-server-db     0:off     1:off     2:off     3:on     4:on     5:on     6:off

Once working and accessible the Cloudera Manager Login Page looks as below:

 

 

Screen Shot 2014-02-06 at 5.20.35 PM

Advertisements

Unix/OSX trick to parse column based console text to get specific row and column text value

With either OSX or *nix OS, console outputs have text data in columns. Sometimes it may required to parse column based console output to get specific row or column value.  In this article we will use a combination of sort, head, tail, tr, cut commands to get particular row or column value from console output.

Lets start with ls command which returns the folder and files name as shown below:

$ ls -l

total 8
drwxr-xr-x 2 avkash staff 68 Sep 17 00:10 Any
drwxr-xr-x 2 avkash staff 68 Sep 17 00:07 Apps
drwxr-xr-x 2 avkash staff 68 Sep 17 00:07 Books
drwxr-xr-x 2 avkash staff 68 Sep 17 00:08 Code
drwxr-xr-x 2 avkash staff 68 Sep 17 00:07 Music
drwxr-xr-x 2 avkash staff 68 Sep 17 00:07 Songs
drwxr-xr-x 2 avkash staff 68 Sep 17 00:08 Zeppos
-rw-r–r– 1 avkash staff 46 Sep 17 00:23 data.txt

$ ls -l | head -n 2 | tr -s ‘ ‘ | cut -d’ ‘ -f9

Any

$ ls -l | tail -n 1 | tr -s ‘ ‘ | cut -d’ ‘ -f9
data.txt

You can also use this trick with sorting command output using sort command (-n, -M, -d etc….)

$ ls -l | sort -n
-rw-r–r– 1 avkash staff 46 Sep 17 00:23 data.txt
drwxr-xr-x 2 avkash staff 68 Sep 17 00:07 Apps
drwxr-xr-x 2 avkash staff 68 Sep 17 00:07 Books
drwxr-xr-x 2 avkash staff 68 Sep 17 00:07 Music
drwxr-xr-x 2 avkash staff 68 Sep 17 00:07 Songs
drwxr-xr-x 2 avkash staff 68 Sep 17 00:08 Code
drwxr-xr-x 2 avkash staff 68 Sep 17 00:08 Zeppos
drwxr-xr-x 2 avkash staff 68 Sep 17 00:10 Any
total 8

$ ls -l | sort -n | tail -n 2 | tr -s ‘ ‘ | cut -d’ ‘ -f9
Any

$ ls -l | sort -n | head -n 1 | tr -s ‘ ‘ | cut -d’ ‘ -f9
data.txt

This trick does works with any text in console i.e. you could use output of a file as well.

Example 2:

$ cat data.txt

Any
Apps
Books
Code
Gum
Music
Songs
Zeppos
X

$ cat data.txt  | tail -n 1 | tr -s ‘ ‘ | cut -d’ ‘ -f1

X

$ cat data.txt | head -n 1 | tr -s ‘ ‘ | cut -d’ ‘ -f1

Any

Thats all. Have fun!!

Thanks to my friend Aaron for showing this great trick.

Keywords: ls, sort, head, tail, tr, cut

 

mdfind in mac OSX : Command line search utility

mdfind is a great utility, similar to “find” in Unix/Linux however mdfind use Mac OSX spotlight index to returns to results. The command help looks like as below:

—————————————————————————————————————————————————-

Usage: mdfind [-live] [-count] [-onlyin directory] [-name fileName | -s smartFolderName | query]

list the files matching the query
query can be an expression or a sequence of words

-live Query should stay active
-count Query only reports matching items count
-onlyin <dir> Search only within given directory
-name <name> Search on file name only
-s <name> Show contents of smart folder <name>
-0 Use NUL (“”) as a path separator, for use with xargs -0.

example: mdfind image
example: mdfind -onlyin ~ image
example: mdfind -name stdlib.h
example: mdfind “kMDItemAuthor == ‘*MyFavoriteAuthor*'”
example: mdfind -live MyFavoriteAuthor

—————————————————————————————————————————————————-

An example of using mdfind to bettertouchtool as below:

$ mdfind bettertouchtool
/Users/hadoopworld/Library/Application Support/CrashReporter/BetterTouchTool_1DB9FB92-A7F5-5334-9193-9ABD33C6389D.plist
/Users/hadoopworld/Library/Application Support/BetterTouchTool
/Applications/BetterTouchTool.app
/Users/hadoopworld/Downloads/btt0.967.zip
/Users/hadoopworld/Library/Containers/com.evernote.Evernote/Data/Library/CoreData/com.evernote.Evernote/5FEB3D01-3FEE-4E34-BE5A-576E028BC763/ENNote/_records/0/p20.ennote
/Users/hadoopworld/Library/Containers/com.evernote.Evernote/Data/Library/Application Support/Evernote/accounts/Evernote/avkashnotes/content/p20/content.html