{"word_count": 905, "character_count": 9312}
One of the text-mining sub-projects aims to develop Recommender System for the OpenEdition Platforms. We use graph structure to store OpenEdition documents. Each node represents document (article, book, review, …) and each edge represents a specific relation between two documents (citation, similarity, …).We choose to use Hadoop and Giraph frameworks for graph processing.In this document, we describe how to set up and configure both Hadoop 2.4.0 and Giraph 1.1.0 on Ubuntu or Debian 64-bitPrerequisitesBefore starting the installation of Hadoop and Giraph, insure that you have installed :Java TM. Hadoop requires Java 7 or late version of java 6. Install Java with the following command:$ sudo apt-get install openjdk-7-jdkTo check the installed version:$ java -versionSSH. If not installed, use the following command in terminal:$ sudo apt-get install sshDownload Hadoop 2.4.0 from : https://www.apache.org/dyn/closer.cgi/hadoop/common/Download Giraph 1.1.0 using (see section “Deploying Giraph” below):$ git clone http://git-wip-us.apache.org/repos/asf/giraph.gitMaven 3 or higher. Giraph uses the munge plugin, which requires Maven 3, to support multiple versions of Hadoop. Also, the web site plugin requires Maven 3.Steps for Hadoop Installation1. Add a dedicated hadoop system user$ sudo addgroup hadoop$ sudo adduser –ingroup hadoop hduser2. SSH configuration$ su – hduser$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa$ cat ~/.ssh/id_rsa.pub &#62;&#62; ~/.ssh/authorized_keys$ ssh localhostIn this step, we enable access to local machine with newly created key.3. Download Hadoop 2.4.0Download from Apache Hadoop site version 2.4.0 and unpack it to a directory of your choice.$ sudo tar xzf hadoop-2.4.0.tar.gzMove the unpacked Hadoop to /usr/local/hadoop-2.4.0 and make hduser:hadoop its owner using:$ sudo mv hadoop-2.4.0 /usr/local/$ sudo cd /usr/local$ sudo ln -s hadoop-2.4.0 hadoop$ sudo chown -R hduser:hadoop hadoop-2.4.0$ sudo chown -R hduser:hadoop hadoop4. Hadoop ConfigurationIn Hadoop 2.x, we found 2 components→ HDFS: it is a distributed file system→ YARN (Yet Another Resource Negotiator): it is a resource manager which manage allocating containers where jobs can run using the data stored in HDFS.Hadoop can be run on a single-node in pseudo-distributed mode where each Hadoop daemon runs in separate java process.To configure Hadoop, some files must be edited.4.1. Edit etc/hadoop/hadoop-env.sh with the following# The java implementation to use. Here, 64-bit OpenJDK 7 is usedexport JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64# Hadoop installation directoryexport HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}# Extra Java runtime options. Disable IPv6 and set the library directory.export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.library.path=$HADOOP_PREFIX/lib"# Hadoop native library directorexport HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/nativeThe following error will result with the use of 64-bit JDK, which has been reported not to be a big issue.WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableTo resolve this error, we need to recompile Hadoop fro 64-bit JDK. (not necessary)4.2. Edit etc/hadoop/core-site.xml with the following&#60;property&#62;    &#60;name&#62;fs.defaultFS&#60;/name&#62;    &#60;value&#62;hdfs://localhost:9000&#60;/value&#62;&#60;/property&#62;&#60;property&#62;    &#60;name&#62;hadoop.tmp.dir&#60;/name&#62;    &#60;value&#62;/app/hadoop/tmp&#60;/value&#62;    &#60;description&#62;A base for other temporary directories.&#60;/description&#62;&#60;/property&#62;The second property of the core-site.xml file describes the default base temporary directory for the local file system and HDFS which is /app/hadoop/tmp. So, we have to set required ownership and permission with following:$ sudo mkdir -p /app/hadoop/tmp$ sudo chown -R hduser:hadoop /app/hadoop/tmp# ... and if you want to tighten up security, chmod from 755 to 750...$ sudo chmod 750 /app/hadoop/tmp4.3. Edit etc/hadoop/hdfs-site.xml with following&#60;property&#62;    &#60;name&#62;dfs.replication&#60;/name&#62;    &#60;value&#62;1&#60;/value&#62;&#60;/property&#62;&#60;property&#62;    &#60;name&#62;dfs.namenode.name.dir&#60;/name&#62;    &#60;value&#62;file:/home/hduser/mydata/hdfs/namenode&#60;/value&#62;&#60;/property&#62;&#60;property&#62;    &#60;name&#62;dfs.datanode.data.dir&#60;/name&#62;    &#60;value&#62;file:/home/hduser/mydata/hdfs/datanode&#60;/value&#62;&#60;/property&#62;4.4. In YARN versions of hadoop, MapReduce isn't the only computational model. Edit etc/hadoop/mapred-site.xml with following&#60;property&#62;    &#60;name&#62;mapreduce.framework.name&#60;/name&#62;    &#60;value&#62;yarn&#60;/value&#62;&#60;/property&#62;4.5. Edit etc/hadoop/yarn-site.xml with following&#60;property&#62;    &#60;name&#62;yarn.nodemanager.aux-services&#60;/name&#62;    &#60;value&#62;mapreduce_shuffle&#60;/value&#62;&#60;/property&#62;&#60;property&#62;    &#60;name&#62;yarn.nodemanager.aux-services.mapreduce_shuffle.class&#60;/name&#62;    &#60;value&#62;org.apache.hadoop.mapred.ShuffleHandler&#60;/value&#62;&#60;/property&#62;&#60;property&#62;    &#60;name&#62;yarn.resourcemanager.resource-tracker.address&#60;/name&#62;    &#60;value&#62;localhost:8025&#60;/value&#62;&#60;/property&#62;&#60;property&#62;    &#60;name&#62;yarn.resourcemanager.scheduler.address&#60;/name&#62;    &#60;value&#62;localhost:8030&#60;/value&#62;&#60;/property&#62;&#60;property&#62;    &#60;name&#62;yarn.resourcemanager.address&#60;/name&#62;    &#60;value&#62;localhost:8050&#60;/value&#62;&#60;/property&#62;5. Testing Local MapReduce Job5.1. Format HDFS filesystem$ bin/hdfs namenode -format5.2. Start nameNode daemon and DataNode daemon$ sbin/start-dfs.sh$ sbin/start-all.sh5.3. Check that all services got started using 'jps' command whose output should be similar to$ jps50831 Jps65180 DataNode65035 NameNode65612 ResourceManager65386 SecondaryNameNode65731 NodeManager5.4. Male HDFS directories required to execute MapReduce jobs$ bin/hdfs dfs -mkdir /user$ bin/hdfs dfs -mkdir /user/hduser5.5. Copy input files into HDFS$ sbin/hdfs dfs -put etc/hadoop /user/hduser/input5.6. Run an example$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep /user/hduser/input/hadoop output 'dfs[a-z.]+'5.7. Check the output using$ bin/hdfs dfs -cat output/*The output should be similar to:6 dfs.audit.logger4 dfs.class3 dfs.server.namenode.2 dfs.audit.log.maxbackupindex2 dfs.period2 dfs.audit.log.maxfilesize1 dfsmetrics.log1 dfsadmin1 dfs.servers1 dfs.replication1 dfs.file1 dfs.datanode.data.dir1 dfs.namenode.name.dirDeploying Giraph1. Git and Maven 3 or higher must be installed, if not, exit from hduser using 'exit' command and do following$ sudo apt-get install git$ sudo apt-get install maven$ mvn -version$ cd /usr/local/$ sudo git clone https://github.com/apache/giraph.git$ sudo chown -R hduser:hadoop giraph$ su – hduser2. Edit $HOME/.bashrc for user account and add the following lineexport GIRAPH_HOME=/usr/local/giraph3. After that, save and close file. Then do following$ source $HOME/.bashrc$ cd $GIRAPH_HOME4. To build giraph 1.1.0 with hadoop 2.4.0, you have two possibilities→ if you use YARN:$ mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package→ if you use just hadoop2 : $ mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests packageRunning a Giraph job (from: https://giraph.apache.org/quick_start.html)1. Create an example graph under /tmp/tiny_graph.txt with the following[0,0,[[1,1],[3,3]]][1,0,[[0,1],[2,2],[3,1]]][2,0,[[1,2],[4,4]]][3,0,[[0,3],[1,1],[4,4]]][4,0,[[3,4],[2,4]]]2. Save and close the file. Then copy the input file to HDFS$ HADOOP_HOME/bin/hadoop dfs -copyFromLocal /tmp/tiny_graph.txt /user/hduser/input/tiny_graph.txt$ HADOOP_HOME/bin/hadoop dfs -ls /user/hduser/inputYou should have this output :DEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.14/09/30 13:43:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableFound 2 itemsdrwxr-xr-x - hduser supergroup 0 2014-09-26 01:14 /user/hduser/input/hadoop-rw-r--r-- 1 hduser supergroup 112 2014-09-28 01:03 /user/hduser/input/tiny_graph.txt 3. Run the example as following$ bin/hadoop jar/usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation-vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat-vip /user/hduser/input/tiny_graph.txt-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat-op /user/hduser/output/shortestpaths-w 1-ca giraph.SplitMasterWorker=false4. To check the output, use$ bin/hadoop dfs -cat /user/hduser/output/shortestpaths/p* You should have:DEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.14/09/28 01:59:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable0 1.02 2.01 0.03 1.04 5.0We can stop all services and check with 'jps':$ cd /usr/local/hadoop$ sbin/stop-all.sh