Big Data General Errors and Fix

Issue A)- Incompatible clusterIDs in Hadoop :  Some time you get below error in your logs , in this case you will not find datanode as running instance.

Error Stack - Incompatible clusterIDs in /usr/local/hadoop_store/hdfs/datanode: namenode clusterID = CID-52c3b330-d913-461d-8a5a-03c1659390bd; datanode clusterID = CID-bfea0588-2680-43cb-aa69-d5464168ca88
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:646)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:320)

                                                           Fix : 
ClusterIDs of datanodes and namenodes should match, then only can datanodes communicate with namenode. 

When first time you run your hadoop instance (using start-all.sh or other way) you can see datanode is in running state, but after that if you format your name node then  new ClusterID will be assigned for namenodes  and ClusterIDs in datanodes won't match.

You can locate a VERSION files in your /usr/local/hadoop_store/hdfs/datanode/current/ (datanode directory ).

You can fix above issue by two way

1)-  If you dont want to delete data of datanode then go to  /usr/local/hadoop_store/hdfs/datanode/current/  open version file and update clusterID value with updated one (as mention in error trace).

2)- If you are not ok with above apporoach you can execute below commands to generate new clusterId.

   a)- Stop all HDFS services.
   b)- rm -rf /usr/local/hadoop_store/hdfs/datanode/* 
   c)- rm -rf /usr/local/hadoop_store/hdfs/namenode/*
   d)-format hdfs again (hadoop namenode -format )


Issue B)- java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected .


                                          Fix : 
  a)- Open profile file(or any file where you stored path info of hive) -> sudo nano /etc/profile
  b)- add line at bottom of file -> export HADOOP_USER_CLASSPATH_FIRST=true
  c)- execute command -> source /etc/profile
  d)- type hive and hit Enter.

Now it should working fine and you able to see hive pointer. like
hive>

Issue C)- Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D


                                        Fix :

1)- execute command -> sudo nano hive-site.xml
2)- update below highlighted 3 property value with your created directory

<property>

<name>hive.exec.scratchdir</name>

    <value>/usr/local/hive_data</value>

    <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created,$

</property>

<property>

    <name>hive.exec.local.scratchdir</name>

    <value>/usr/local/hive_data</value>

    <description>Local scratch space for Hive jobs</description>

</property>

<property>

    <name>hive.downloaded.resources.dir</name>

    <value>/usr/local/hive_data/${hive.session.id}_resources</value>

    <description>Temporary local directory for added resources in the remote file system.</description>

</property> 

this will solve your above exception, 

Issue D)- java.io.IOException: Unable to initialize any output collector.
 at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158).


To fix this issue you can do two things,

a)- Increase io.sort.mb value
<property> <name>io.sort.mb</name> <value>800</value> </property>

b)- Go to you userlogs and check syslog of all created folder, As in several cases(like in my case there are some issue with the program that cause above issue to me)


Thanks :)