Tag: Hadoop

I recently came accross some maven errors while building Apache Ranger 0.5 leading to a failure while building the HDFS Security Plugin and the Storm Security Plugin.

To get rid of the error just change the version of the maven-assembly-plugin in Ranger’s main pom.xml from

<version>2.2-beta-5</version>

to

<version>2.3</version>

…, add the following repository definitions to the main pom.xml

<repository>
<id>conjars</id>
<name>Concurrent Conjars repository</name>
<url>http://conjars.org/repo</url>
<layout>default</layout>
</repository>
<repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>

…, add the following dependency to the storm-agent/pom.xml:

<dependency>
<groupId>ring</groupId>
<artifactId>ring-jetty-adapter</artifactId>
<version>0.3.11</version>
</dependency>

Now you should be able to build ranger. If you are facing issues with a missing pom.xml for the dependency tomcat:common-el then add the following repository to the main pom.xml:

<repository>
<id>conjars</id>
<name>Concurrent Conjars repository</name>
<url>http://conjars.org/repo</url>
<layout>default</layout>
</repository>

The absence of the pom.xml on the default maven repository should only be a temporary issue but it took me some time to figure it out so that you can avoid it easily by addind a second repo that provides the missing file.

I faced the requirement to set the setgid and/or the setuid for directories in a customer’s HDFS so that I spent the last 10 minutes searching the internet for news on a possibility to do so. Long story short: results were bad. Last time I tried with Hadoop 2.2.0 without success, today I tried with 2.7.1 but there were no changes at all. So commands like the following are NOT possible:

hdfs dfs -chmod g+s /tmp
hdfs dfs -chmod u+s /tmp

As a matter of fact, files created in HDFS receive the group of the parent directory and hence act as if the setgid flag is set. The uid a newly created referes always to the user that created the file.

Update: When mounting an HDFS as NFS Gateway the behavior is different since the file system acts as if the setgid was NOT set. Which is strange since same file operations return different results depending on on which interface the operation is executed on. Furthermore, the NFS Gateway does not take secondary groups into account when evaluating permissions to write into the mounted HDFS. The only options to achive an allowance is a user mapping as described at the bottom of the page of the Hadoop Permissions Guide.

If you face the need to make use of the flags think about using a posix compliant alternative to HDFS like Spectrum Scale/GPFS.