Uploading file(s) to Amazon S3 using python + boto & s3afe.py script

Tools Used: python, boto, s3afe.py

Step 1: Be sure to have python first and then make sure you can Install boto module in python as well

  • $sudo easy_install pip
  • $ sudo pip install boto

 

Step 2: Download s3afe.py from the link below:

 

Step 3: Setup execution mode (chmod) to s3afe.py and verify that it executes on your machine:

AvkashMBP2:~ userx$ /usr/local/bin/s3afe.py
Usage: s3afe.py -k [access key id] -s [secret access key] -b [bucketname] -n [keyname] -f [file]

S3afe stores a file on Amazon S3

Options:
–version show program’s version number and exit
-h, –help show this help message and exit
-k AWSKEY, –aws_access_key_id=AWSKEY
-s AWSSECRET, –aws_secret_access_key=AWSSECRET
-f FILENAME, –filename=FILENAME
-b BUCKETNAME, –bucketname=BUCKETNAME
-n KEYNAME, –keyname=KEYNAME
-a ACL, –acl=ACL

 

Step 4: To upload a file from your local machine to Amazon S3 you will use the following parameters:

  -k AWSKEY, --aws_access_key_id=AWSKEY
  -s AWSSECRET, --aws_secret_access_key=AWSSECRET
  -f FILENAME, --filename=FILENAME  <-- Source File Name
  -b BUCKETNAME, --bucketname=BUCKETNAME
  -n KEYNAME, --keyname=KEYNAME  <-- New File Name after the upload
  -a ACL, --acl=ACL <-- Use 'private' here

 

Real Usage Example (From my mac):

$/usr/local/bin/s3afe.py  -f ‘/Users/userx/Downloads/file1.tgz’ -n ‘file.tgz’ -b ‘AMAZON_BUCKET_NAME’ -a ‘private’ -k ‘AMAZON_S3_ACCESS_KEY’ -s ‘AMAZON_S3_SECRET_KEY’ 

Note: After file upload is completed, you may get an error as below which is benign because the file does upload and I am not sure why I get this error.

Error: Please check your Amazon credentials!

S3afe.py Documentation: https://code.google.com/p/s3afe/

 

 

Permission denied (publickey,gssapi-keyex,gssapi-with-mic) error with SSH connection to Amazon EC2 Instance

Problem: I tried accessing my EC2 instance and got the error as below:

Avkash-Machine:~ avkash$ ssh -i MyUtils/SSHAccessKey.pem ec2_user@xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Lets try to have verbose SSH connection to see what could be the issue:

Avkash-Machine:~ avkash$ ssh -v -i MyUtils/SSHAccessKey.pem ec2_user@xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com
OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 53: Applying options for *
debug1: Connecting to xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com [xx.xxx.xxx.xx] port 22.
debug1: Connection established.
debug1: identity file MyUtils/SSHAccessKey.pem type -1
debug1: identity file MyUtils/SSHAccessKey.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 0f:ce:27:19:18:ee:40:86:df:db:f0:95:79:29:49:05
debug1: Host ‘xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com’ is known and matches the RSA host key.
debug1: Found key in /Users/avkash/.ssh/known_hosts:15
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Trying private key: MyUtils/SSHAccessKey.pem
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Above you can see that the SSH key is valid and readable and the connection did try various auth methods however connection still failed.

As you can see below that SSH Key SSHAccessKey.pem have proper access mode configured:

Avkash-Machine:~ avkash$ ls -l MyUtils/SSHAccessKey.pem
-rwx——@ 1 avkash staff 1692 May 20 14:38 MyUtils/SSHAccessKey.pem
Now the connection is working here as shown in the following full verbose log:

Avkash-Machine:~ avkash$ ssh -v -i MyUtils/SSHAccessKey.pem ec2-user@xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com
OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 53: Applying options for *
debug1: Connecting to xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com [xx.xxx.xxx.xx] port 22.
debug1: Connection established.
debug1: identity file MyUtils/SSHAccessKey.pem type -1
debug1: identity file MyUtils/SSHAccessKey.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 0f:ce:27:19:18:ee:40:86:df:db:f0:95:79:29:49:05
debug1: Host ‘xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com’ is known and matches the RSA host key.
debug1: Found key in /Users/avkash/.ssh/known_hosts:15
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Trying private key: MyUtils/SSHAccessKey.pem
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com ([xx.xxx.xxx.xx]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LC_MONETARY = en_US.utf-8
debug1: Sending env LC_NUMERIC = en_US.utf-8
debug1: Sending env LC_MESSAGES = en_US.utf-8
debug1: Sending env LC_COLLATE = en_US.utf-8
debug1: Sending env LANG = ru_RU.UTF-8
debug1: Sending env LC_CTYPE = en_US.utf-8
debug1: Sending env LC_TIME = en_US.utf-8

Verification done:

  1.  I checked that my EC2 Instance have proper SSH key
  2. Verified that EC2 instance have ssh port open in firewall (Security Policy)

Solution: I was using incorrect user Id (i.e. ec2_user) instead of the correct one (i.e. ec2-user)

 

 

 

Handling Apache Hive Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

Recently I hit the following  Error when starting Apache hive in my server:

ubuntu@HIVE_SERVER:~$ hive
log4j:ERROR Could not find value for key log4j.appender.DEBUG
log4j:ERROR Could not instantiate appender named “DEBUG”.
log4j:ERROR Could not find value for key log4j.appender.WARN
log4j:ERROR Could not instantiate appender named “WARN”.
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/hive/hive.log (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
at java.io.FileOutputStream.<init>(FileOutputStream.java:136)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:415)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:52)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:629)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].
Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
Hive history file=/tmp/ubuntu/hive_job_log_ubuntu_201306111657_1917799460.txt
hive> show tables;
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive> quit;

Potential Root Cause:

This is mainly because the I have had previous hive installation in the machine and uninstalled it. Not all the files were removed completelty and the /var/log/hive/hive.log file was not there and most of the settings were messed up.

Even when I created an empty /var/log/hive/hive.log ($touch /var/log/hive/hive.log and chmod 777 /var/log/hive/hive.log) still the problem persisted.

Removing Hive:

ubuntu@HIVE_SERVER:~$ sudo apt-get remove hive
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
libopts25 libcap2 hbase ntp
Use ‘apt-get autoremove’ to remove them.
The following packages will be REMOVED:
hive
0 upgraded, 0 newly installed, 1 to remove and 26 not upgraded.
After this operation, 175 MB disk space will be freed.
Do you want to continue [Y/n]? Y
(Reading database … 57309 files and directories currently installed.)
Removing hive …
Processing triggers for man-db …

Checking what else is left after removal:

ubuntu@HIVE_SERVER:~$ sudo find / -name “hive”
/etc/hive
/usr/lib/hive
/var/lib/hive

Deleting all of above residual files in Hive:

ubuntu@HIVE_SERVER:~$ sudo rm -rf /etc/hive
ubuntu@HIVE_SERVER:~$ sudo rm -rf /usr/lib/hive
ubuntu@HIVE_SERVER:~$ sudo rm -rf /var/lib/hive

Verifying that none of hive is in my machine:

ubuntu@HIVE_SERVER:~$ sudo find / -name “hive”

Installing Hive (from cloudera distribution)

ubuntu@HIVE_SERVER:~$ sudo apt-get install hive
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
hive
0 upgraded, 1 newly installed, 0 to remove and 26 not upgraded.
Need to get 0 B/32.4 MB of archives.
After this operation, 175 MB of additional disk space will be used.
Selecting previously unselected package hive.
(Reading database … 50478 files and directories currently installed.)
Unpacking hive (from …/hive_0.10.0+78-1.cdh4.2.1.p0.8~precise-cdh4.2.1_all.deb) …
Processing triggers for man-db …
Setting up hive (0.10.0+78-1.cdh4.2.1.p0.8~precise-cdh4.2.1) …
update-alternatives: using /etc/hive/conf.dist to provide /etc/hive/conf (hive-conf) in auto mode.

Listing hive in my machine:

ubuntu@HIVE_SERVER:~$ sudo find / -name “hive”
/etc/hive
/run/hive
/usr/lib/hive
/usr/lib/hive/lib/php/packages/serde/org/apache/hadoop/hive
/usr/lib/hive/docs/api/org/apache/hadoop/hive
/usr/lib/hive/docs/api/org/apache/hive
/usr/lib/hive/bin/hive
/usr/share/doc/hive
/usr/share/doc/hive/examples/test-plugin/src/org/apache/hive
/usr/bin/hive
/var/lib/hive
/var/log/hive

Testing Hive:

ubuntu@HIVE_SERVER:~$ hive
Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-0.10.0-cdh4.2.1.jar!/hive-log4j.properties
Hive history file=/tmp/ubuntu/hive_job_log_ubuntu_201306111709_559685140.txt
hive> show tables;
OK
Time taken: 14.431 seconds
hive> quit;

Its working!!

Adventure with Postgresql on Ubuntu

Setting postgresql to not start during startup:

To enable or disable postgresql start at machine startup edit start.conf located @ /etc/postgresql/9.2/main/ folder and add proper configuration auto |  manual | disabled 

$sudo vi /etc/postgresql/9.2/main/start.conf

# Automatic startup configuration
# auto: automatically start/stop the cluster in the init script
# manual: do not start/stop in init scripts, but allow manual startup with
# pg_ctlcluster
# disabled: do not allow manual startup with pg_ctlcluster (this can be easily
# circumvented and is only meant to be a small protection for
# accidents).

auto

postgres database server Startup or Shutdown:

postgresql database server can start directly as service from services tool i.e. “sudo services postgresql start|stop|*” or using a wrapper named pg_ctl, as below: 

$sudo services postgresql start

$ /usr/lib/postgresql/9.2/bin/pg_ctl 

If postgresql is start through wrapper with specific catalog, then the status display different results as below:   

$ sudo service postgresql status
9.2/main (port 5432): down

$ /usr/lib/postgresql/9.2/bin/pg_ctl status -D /home/hadoopuser/hadoopuser_DATA/pgdb -l /home/hadoopuser/hadoopuser_DATA/logs/pg.log -w 

pg_ctl: server is running (PID: 3381)
/usr/lib/postgresql/9.2/bin/postgres “-D” “/home/hadoopuser/hadoopuser_DATA/pgdb”
hadoopuser@HADOOP_CLUSTER:~$

Handling portgresql error “could not create lock file “/var/run/postgresql/.s.PGSQL.5432.lock”: Permission denied”

Sometimes starting postgresql server returns the following error:

WARNING: could not create listen socket for “*”
FATAL: could not create any TCP/IP sockets
FATAL: could not create lock file “/var/run/postgresql/.s.PGSQL.5432.lock”: Permission denied

The problem is that in ubuntu postgresql is running and during installation hadoopuser is starting it and because of ownership issue hadoopuser can not use it and user see the lock issue as shown in above logs:

 

hadoopuser@hadoopserver:~$ ls -l /var/run/
drwxrwsr-x 2 postgres postgres 40 May 28 22:16 postgresql

hadoopuser@hadoopserver:~$ sudo chmod 777 /var/run/postgresql

hadoopuser@hadoopserver:/usr/local/hadoopuser/current$ ls -l /var/run/
drwxrwsrwx 2 postgres postgres 80 May 28 22:32 postgresql

 

Handling postgresql error “Is another postmaster already running on port 5432”

Sometimes starting postgresql server returns the following error:

HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
LOG: could not bind IPv6 socket: Address already in use
HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
WARNING: could not create listen socket for “*”
FATAL: could not create any TCP/IP sockets

To solve this problem please first check if postgresql is running as below:

hadoopuser@hadoopserver:~$ ps auxwww | grep postg
hadoopuser 1805 0.0 0.1 137520 10212 pts/0 S 22:25 0:00 /usr/lib/postgresql/9.2/bin/postgres -D /home/hadoopuser/hadoopuser_DATA/pgdb
hadoopuser 1807 0.0 0.0 137520 1492 ? Ss 22:25 0:00 postgres: checkpointer process
hadoopuser 1808 0.0 0.0 137520 1732 ? Ss 22:25 0:00 postgres: writer process
hadoopuser 1809 0.0 0.0 137520 1492 ? Ss 22:25 0:00 postgres: wal writer process
hadoopuser 1810 0.0 0.0 138300 2776 ? Ss 22:25 0:00 postgres: autovacuum launcher process
hadoopuser 1811 0.0 0.0 97208 1572 ? Ss 22:25 0:00 postgres: stats collector process
hadoopuser 2165 0.0 0.0 8104 924 pts/0 S+ 22:32 0:00 grep –color=auto postg

Stop the postgresql service as below:

hadoopuser@hadoopserver:~$ sudo service postgresql stop
* Stopping PostgreSQL 9.2 database server [ OK ]

Note: Above command does not guarantee to stop services if the database server was started in other user context and actively running with an open database. 

or kill the postgresql process as 

hadoopuser@hadoopserver:~$ killall postg

 

SimpleHTTPServer: A simple lightweight web server in python

If you have Python 2.x installed along with module SimpleHTTPServer you can run a lightweight HTTP server in your machine as below:

 

$ python -m SimpleHTTPServer 

By default the web server serves  HTTP request/response on port 8000 as defined in python source below:
/usr/lib/python2.6/BaseHTTPServer.py

 

If you decide to change the port setting with SimpleHTTPServer you can just pass the port number as below:

$ python -m SimpleHTTPServer 8008

Once web server is up and running you can open web browser with local ipaddress:port or localhost:port as url. The By default the console shows http logs by the web server as below:

$ python -m SimpleHTTPServer 8008
Serving HTTP on 0.0.0.0 port 8008 …
c-x-x-x-x.hsd1.ca.comcast.net – – [06/Jun/2013 01:48:28] “GET / HTTP/1.1” 200 –
c-x-x-x-x.hsd1.ca.comcast.net – – [06/Jun/2013 01:50:21] “GET / HTTP/1.1” 200 –

 

To make sure you have SimpleHTTPServer part of your python install you can verify as below:

$python –version
Python 2.7.3
$ python
Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.

>>> import SimpleHTTPServer

If this setup does not return any error means you have SimpleHTTPServer installed and working in your machine.
>>> SimpleHTTPServer.__version__
‘0.6’

>>>help(‘modules’)

This will list all the modules installed with python and SimpleHTTPServer will be part of the list.

>>> exit()

 

Keywords: Python, Web Server

Handling Permission denied error on HDFS

When you try creating a folder or a file on HDFS you may hit the following error:

ubuntu@HADOOP_CLUSTER:~$ hdfs dfs -mkdir /abc

mkdir: Permission denied: user=ubuntu, access=WRITE, inode=”/user”:hdfs:hadoop:drwxr-xr-x
This particuler problem could happen because current logged user may not be part of hadoop group.

So lets find out who is the current logged user:

ubuntu@HADOOP_CLUSTER:~$ whoami
ubuntu

Now we can find out the list of groups the user “ubuntu” is associated with:

ubuntu@HADOOP_CLUSTER:~$ groups ubuntu
ubuntu : ubuntu adm dialout cdrom floppy audio dip video plugdev netdev admin

As we can see above that user ubuntu is not part of hadoop group so we will go ahead and add this user to hadoop group:

ubuntu@HADOOP_CLUSTER:~$ sudo adduser ubuntu hadoop
Adding user `ubuntu’ to group `hadoop’ …
Adding user ubuntu to group hadoop
Done.

Now we can try the exact same command we tried before:

ubuntu@HADOOP_CLUSTER:~$ hdfs dfs -mkdir /abc

The command is successfull and if we list files and folder at HDFS we can find our folder as below:

ubuntu@HADOOP_CLUSTER:~$ hdfs dfs -ls /
Found 5 items
drwxr-xr-x – ubuntu hadoop 0 2013-06-05 05:58 /abc
-rw-r–r– 1 ubuntu hadoop 28865 2013-06-04 16:14 /history.log
drwxr-xr-x – mapred hadoop 0 2013-06-04 15:56 /home
drwxrwxrwt – mapred hadoop 0 2013-06-05 05:48 /tmp
drwxr-xr-x – hdfs hadoop 0 2013-06-05 05:53 /user

To make sure that user “ubuntu” is part of “hadoop” group we can verify as below:

ubuntu@HADOOP_CLUSTER:~$ groups ubuntu
ubuntu : ubuntu adm dialout cdrom floppy audio dip video plugdev netdev admin hadoop

Note: IF above command does not work then you can use the following:

$ sudo -u hdfs hdfs dfs -mkdir /user/newfolder

Keywords: Hadoop, HDFS, Error

Handling Cloudera Hadoop Cluster from command line

If you have installed Hadoop from Cloudera distribution without Cloudera Manager you would have to manage your cluster from console and the things art not easy. Here are some of the important information to manage working on Cloudera Hadoop from console:

 

Where hadoop binary are located:

ubuntu@HADOOP_CLUSTER:~$ which hadoop

    • /usr/bin/hadoop

Files located at /usr/lib/hadoop/

drwxr-xr-x 2 root root 4096 May 22 21:00 bin
drwxr-xr-x 2 root root 4096 May 23 00:25 client
drwxr-xr-x 2 root root 4096 May 23 00:25 client-0.20
drwxr-xr-x 2 root root 4096 May 22 21:00 cloudera
drwxr-xr-x 2 root root 4096 May 22 21:00 etc
-rw-r–r– 1 root root 16678 Apr 22 17:38 hadoop-annotations-2.0.0-cdh4.2.1.jar
lrwxrwxrwx 1 root root 37 Apr 22 17:38 hadoop-annotations.jar -> hadoop-annotations-2.0.0-cdh4.2.1.jar
-rw-r–r– 1 root root 46858 Apr 22 17:38 hadoop-auth-2.0.0-cdh4.2.1.jar
lrwxrwxrwx 1 root root 30 Apr 22 17:38 hadoop-auth.jar -> hadoop-auth-2.0.0-cdh4.2.1.jar
-rw-r–r– 1 root root 2267883 Apr 22 17:38 hadoop-common-2.0.0-cdh4.2.1.jar
-rw-r–r– 1 root root 1213897 Apr 22 17:38 hadoop-common-2.0.0-cdh4.2.1-tests.jar
lrwxrwxrwx 1 root root 32 Apr 22 17:38 hadoop-common.jar -> hadoop-common-2.0.0-cdh4.2.1.jar
drwxr-xr-x 3 root root 4096 May 22 21:00 lib
drwxr-xr-x 2 root root 4096 May 23 00:25 libexec
drwxr-xr-x 2 root root 4096 May 22 21:00 sbin

 

Hadoop cluster specific XML configuration  files are stored here:

lrwxrwxrwx 1 root root 16 Apr 22 17:38 hadoop -> /etc/hadoop/conf
ubuntu@HADOOP_CLUSTER:~$ ls -l /usr/lib/hadoop/etc/hadoop
lrwxrwxrwx 1 root root 16 Apr 22 17:38 /usr/lib/hadoop/etc/hadoop -> /etc/hadoop/conf
ubuntu@HADOOP_CLUSTER:~$ ls -l /etc/hadoop/conf
lrwxrwxrwx 1 root root 29 May 22 21:00 /etc/hadoop/conf -> /etc/alternatives/hadoop-conf
ubuntu@HADOOP_CLUSTER:~$ ls -l /etc/alternatives/hadoop-conf
lrwxrwxrwx 1 root root 23 May 22 22:02 /etc/alternatives/hadoop-conf -> /etc/hadoop/conf.avkash
ubuntu@HADOOP_CLUSTER:~$ ls -l /etc/hadoop/conf.avkash/

    • core-site.xml
    • hadoop-metrics.properties
    • hadoop-metrics2.properties
    • hdfs-site.xml
    • log4j.properties
    • mapred-site.xml
    • slaves
    • ssl-client.xml.example
    • ssl-server.xml.example
    • yarn-env.sh
    • yarn-site.xml

Note: Otherwise you can try to find Hadoop configuration files as

    • ubuntu@ec2-54-214-67-144:~$ sudo find / -name “hdfs*.xml”

 

Hadoop cluster specific scripts are located here:

  • Hadoop
    • /usr/lib/hadoop/libexec/hadoop-config.sh
    • /usr/lib/hadoop/libexec/hadoop-layout.sh
    • /usr/lib/hadoop/sbin/hadoop-daemon.sh
    • /usr/lib/hadoop/sbin/hadoop-daemons.sh
  • MapReduce
    • /usr/lib/hadoop-0.20-mapreduce/bin/hadoop-daemon.sh
    • /usr/lib/hadoop-0.20-mapreduce/bin/hadoop-config.sh
    • /usr/lib/hadoop-0.20-mapreduce/bin/hadoop-daemons.sh

To start/stop/restart Hadoop service, the scripts are located here: 

  • Hadoop Namenode and Job Tracker
    • /etc/init.d/hadoop-0.20-mapreduce-jobtracker
    • /etc/init.d/hadoop-hdfs-namenode
  • Hadoop Datanode and TaskTracker
    • /etc/init.d/hadoop-hdfs-datanode
    • /etc/init.d/hadoop-0.20-mapreduce-tasktracker

 

If you decided to start Hadoop Service manually you can do the following:

  • Stop Services:
    • sudo /etc/init.d/hadoop-hdfs-namenode stop
    • sudo /etc/init.d/hadoop-hdfs-datanode stop
    • sudo /etc/init.d/hadoop-0.20-mapreduce-jobtracker stop
    • sudo /etc/init.d/hadoop-0.20-mapreduce-tasktracker stop
  • Start Services
    • sudo /etc/init.d/hadoop-hdfs-namenode start
    • sudo /etc/init.d/hadoop-hdfs-datanode start
    • sudo /etc/init.d/hadoop-0.20-mapreduce-jobtracker start
    • sudo /etc/init.d/hadoop-0.20-mapreduce-tasktracker start

 

Running hdfs command in hdfs user context:

  • sudo -u hdfs hdfs dfs -mkdir /tmp
  • sudo -u hdfs hdfs dfs -chmod -R 1777 /tmp
  • sudo -u hdfs hdfs dfs -mkdir -p /var/lib/hadoop-hdfs/cache/
  • hdfs dfs -ls /

 

Running Hadoop example jobs from console:

  • ubuntu@HADOOP_CLUSTER:~$ hdfs dfs -copyFromLocal history.log /
  • ubuntu@HADOOP_CLUSTER:~$ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar wordcount /history.log /home/ubuntu/results
  • 13/06/04 16:14:34 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  • 13/06/04 16:14:35 INFO input.FileInputFormat: Total input paths to process : 1
  • 13/06/04 16:14:35 INFO mapred.JobClient: Running job: job_201306041556_0005
  • 13/06/04 16:14:36 INFO mapred.JobClient: map 0% reduce 0%

 

The following error means the HDFS is running but JobTracker is not running at Hadoop: 

  • 13/06/04 15:48:48 INFO ipc.Client: Retrying connect to server: HADOOP_CLUSTER.us-west-2.compute.amazonaws.com/10.254.42.72:8021. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
  • 13/06/04 15:48:49 INFO ipc.Client: Retrying connect to server: HADOOP_CLUSTER.us-west-2.compute.amazonaws.com/10.254.42.72:8021. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
  • 13/06/04 15:48:50 INFO ipc.Client: Retrying connect to server: HADOOP_CLUSTER.us-west-2.compute.amazonaws.com/10.254.42.72:8021. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
  • 13/06/04 15:48:51 INFO ipc.Client: Retrying connect to server: HADOOP_CLUSTER.us-west-2.compute.amazonaws.com/10.254.42.72:8021. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

Keywords: Hadoop, MapReduce, Cloudera, Services