老王大数据教程(一) centos上安装hadoop集群

前端之家收集整理的这篇文章主要介绍了老王大数据教程(一) centos上安装hadoop集群前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。
  1. 准备机器

    主机名 ip 对应的角色 master 192.168.202.5 NameNode
    node1 192.168.202.6 Datanode1
    node2 192.168.202.7 Datanode2
  2. 修改hosts

    1. ###修改hostname
        192.168.202.5 主机:
        echo "master" > /etc/hostname
    
        192.168.202.6 主机:
        echo "node1" > /etc/hostname
    
        192.168.202.7 主机:
        echo "node2" > /etc/hostname
    
    2. ###master上修改hosts文件
         echo "192.168.202.5 master" >>/etc/hosts
         echo "192.168.202.6 node1" >>/etc/hosts
         echo "192.168.202.7 node2" >>/etc/hosts
    
    3. ###node修改同步到node1,node2上的主机
          scp /etc/hosts node1:/etc/
          scp /etc/hosts node2:/etc/
    
    4. ###各台主机上相互ping,测试是否可以通过host连通
         ping master 
         ping node1 
         ping node2
    
    5. ###master,node1,node2上关闭防火墙
      systemctl stop firewalld 
      systemctl disable firewalld
  3. 配置免密码登陆

    1. 本机无密钥登录

      master 机器上配置如下 
      ssh-keygen -t rsa
      cd ~/.ssh
      cp id_rsa.pub authorized_keys 
      service sshd restart
      ssh master
    2. master 与node1,node2 无密登录配置

      在node1和node2上面生成ssh 
      ssh-keygen -t rsa
      在maste 机器上
      scp ~/.ssh/id_rsa.pub root@192.168.30.251:~/.ssh
      scp ~/.ssh/id_rsa.pub root@192.168.30.252:~/.ssh
  4. hadoop 安装(master)

下载 hadoop-2.7.3.tar.gz 把文件解压到 /usr/local/hadoop 修改环境变量 vi /etc/profile 环境变量(在master,node1,node2上都修改)

export JAVA_HOME=/usr/java/jdk1.8.0_65  
    export HADOOP_HOME=/usr/local/hadoop   
    export PATH=$PATH:${HADOOP_HOME}/bin  
    关闭防火墙 service iptables stop
  1. 修改hadoop配置

    master 服务器配置

    1. vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh

      export JAVAHOME=/usr/java/jdk1.8.0121

    2. vim /usr/local/hadoop/etc/hadoop/core-site.xml

      <property>  
              <name>hadoop.tmp.dir</name>  
              <value>/hadoop/tmp</value>  
              <description>Abase for other temporary directories.</description>  
      </property>  
      <property>  
              <name>fs.defaultFS</name>  
              <value>hdfs://master:9000</value>  
      </property>  
      <property>  
              <name>io.file.buffer.size</name>  
              <value>4096</value>  
      </property>
    3. vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml

      <property>  
              <name>dfs.replication</name>  
              <value>2</value>  
              <description>nodes total count</description>  
      </property>
    4. cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml

    5. vi /usr/local/hadoop/etc/hadoop/mapred-site.xml

      <property>  
          <name>mapreduce.framework.name</name>  
          <value>yarn</value>  
          <final>true</final>  
      </property>  
      <property>  
          <name>mapreduce.jobtracker.http.address</name>  
          <value>master:50030</value>  
      </property>  
      <property>  
          <name>mapreduce.jobhistory.address</name>  
          <value>master:10020</value>  
      </property>  
      <property>  
          <name>mapreduce.jobhistory.webapp.address</name>  
          <value>master:19888</value>  
      </property>  
      <property>  
           <name>mapred.job.tracker</name>  
           <value>http://master:9001</value>  
      </property>
    6. vi /usr/local/hadoop/etc/hadoop/yarn-site.xml

      <property>  
           <name>yarn.resourcemanager.hostname</name>  
           <value>master</value>  
      </property>  
      <property>  
          <name>yarn.nodemanager.aux-services</name>  
          <value>mapreduce_shuffle</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.address</name>  
          <value>master:8032</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.scheduler.address</name>  
          <value>master:8030</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.resource-tracker.address</name>  
          <value>master:8031</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.admin.address</name>  
          <value>master:8033</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.webapp.address</name>  
          <value>master:8088</value>  
      </property>
    7. vi /usr/local/hadoop/etc/hadoop/slaves

      node1  
       node2
    8. node1/node2服务器配置和master一样

      scp -r /usr/local/hadoop node1:/usr/local
      scp -r /usr/local/hadoop node1:/usr/local
    9. 格式化master

      cd /usr/local/hadoop
      ./hadoop namenode -format
    10. 启动hadoop(只在master启动)

      cd /usr/local/hadoop/sbin
       ./start-all.sh 
      输入几个yes
    11. 查看进程 在master 上面 输入jps

      24400 SecondaryNameNode
      24565 ResourceManager
      24678 NodeManager
      24998 Jps
      24200 Datanode
      1391 Bootstrap
      24063 NameNode

      在node1和node2上输入jps

      11408 Jps
      11209 Datanode
      1482 Bootstrap
      11307 NodeManager

hadoop在web页面查看

http://192.168.202.5:8088/

参考文献

理论文档 
http://www.cnblogs.com/edisonchou/p/3485135.html

安装教程 

http://www.cnblogs.com/laov/p/3421479.html

https://my.oschina.net/lanzp/blog/309078

猜你在找的CentOS相关文章