ubuntu14.04上hadoop2.6.0伪分布式集群部署

1. 环境要求：

VMware 10.0 虚拟机

ubuntu14.04.1 64位操作系统

hadoop2.6.0

jdk-8u65-linux-x64.tar.gz

2. 配置SSH无密码登陆

进入root权限，使用apt-get update更新系统

安装SSH，apt-get install openssh-server

启动服务 /etc/init.d/ssh start；查看 ps -e | grep ssh

生成私钥和公钥 ssh-keygen -t rsa -P ""

根据生成的.ssh的路径进入.ssh目录下，将id_rsa.pub保存为authorized_keys

命令是 cat ./id_rsa.pub >> ./authorized_keys

使用 ssh localhost验证其成功

3. 安装java

将jdk-8u65-linux-x64.tar.gz文件拷贝到/usr/lcoal/下并解压

tar zxvf jdk-8u65-linux-x64.tar.gz

gedit ~/.bashrc 并在文件最前或者最后添加jdk路径

export JAVA_HOME=/usr/local/jdk1.8.0_65
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:
export PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin:$PATH

在终端输入命令java -version和 javac -version

4. hadoop 安装

4.1 将hadoop2.6.0拷贝到/usr/local下并解压 tar zxvf hadoop-2.6.0.tar.gz

mv hadoop 2.6.0 hadoop

gedit ~/.bashrc 并加入以下内容

export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export JAVA_LIBRARY_PATH=/usr/local/hadoop/lib/native
export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath):$CLASSPATH

编辑/usr/local/hadoop/etc/hadoop/hadoop-env.sh，在这个文件添加java的路径

export JAVA_HOME=/usr/local/jdk1.8.0_65

使用hadoop version验证版本

4.2 Hadoop伪分布式内容配置

在/usr/local/hadoop/etc/hadoop中分别配置core-site.xml，hdfs-site.xml，mapred-site.xml，yarn-site.xml

4.2.1 在core-site.xml中添加

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/local/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

4.2.1 在hdfs-site.xml中添加

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>

4.2.3 cp mapred-site.xml.template mapred-site.xml，并在mapred-site.xml中添加

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://localhost:9001</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

4.2.4在yarn-site.xml中添加

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

4.3 启动hadoop

进入/usr/local/hadoop/bin格式化节点

hdfs namenode -format

进入/usr/local/hadoop/sbin

运行start-all.sh后在终端输入jps

ubuntu14.04上hadoop2.6.0伪分布式集群部署

猜你在找的Ubuntu相关文章