1.Hadoop-Yarn
可参考我的文章《HadoopHA 搭建》
2.修改环境变量
cd /opt//hadoop-3.1.2/etc/hadoop
vi hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_111export HDFS_NAMENODE_USER=rootexport HDFS_DATANODE_USER=rootexport HDFS_JOURNALNODE_USER=rootexport HDFS_ZKFC_USER=rootexport YARN_RESOURCEMANAGER_USER=rootexport YARN_NODEMANAGER_USER=root
修改配置文件
vi mapred-site.xml
<!-- 指定mr框架为yarn方式 --><property><name>mapreduce.framework.name</name><value>yarn</value></property><!-- 指定mapreduce jobhistory地址 --><property><name>mapreduce.jobhistory.address</name><value>10.1.1.58:10020</value></property><!-- 任务历史服务器的web地址 --><property><name>mapreduce.jobhistory.webapp.address</name><value>10.1.1.58:19888</value></property><!-- 配置运行过的日志存放在hdfs上的存放路径 --><property><name>mapreduce.jobhistory.done-dir</name><value>/history/done</value></property><!-- 配置正在运行中的日志在hdfs上的存放路径 --><property><name>mapreudce.jobhistory.intermediate.done-dir</name><value>/history/done/done_intermediate</value></property><property><name>mapreduce.application.classpath</name><value>/opt/test/hadoop-3.1.2/etc/hadoop,/opt/test/hadoop-3.1.2/share/hadoop/common/*,/opt/test/hadoop-3.1.2/share/hadoop/common/lib/*,/opt/test/hadoop-3.1.2/share/hadoop/hdfs/*,/opt/test/hadoop-3.1.2/share/hadoop/hdfs/lib/*,/opt/test/hadoop-3.1.2/share/hadoop/mapreduce/*,/opt/test/hadoop-3.1.2/share/hadoop/mapreduce/lib/*,/opt/test/hadoop-3.1.2/share/hadoop/yarn/*,/opt/test/hadoop-3.1.2/share/hadoop/yarn/lib/*</value></property>
vi yarn-site.xml
<!-- 开启RM高可用 --><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property><!-- 指定RM的cluster id --><property><name>yarn.resourcemanager.cluster-id</name><value>yarn-test</value></property><!-- 指定RM的名字 --><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value></property><!-- 分别指定RM的地址 --><property><name>yarn.resourcemanager.hostname.rm1</name><value>10.1.1.58</value></property><property><name>yarn.resourcemanager.hostname.rm2</name><value>10.1.1.195</value></property><property><name>yarn.resourcemanager.webapp.address.rm1</name><value>10.1.1.58:8088</value></property><property><name>yarn.resourcemanager.webapp.address.rm2</name><value>10.1.1.195:8088</value></property><!-- 指定zk集群地址 --><property><name>yarn.resourcemanager.zk-address</name><value>10.1.1.201:2181,10.1.1.158:2181,10.1.1.185:2181</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><!-- 开启日志聚合 --><property><name>yarn.log-aggregation-enable</name><value>true</value></property><property><name>yarn.log-aggregation.retain-seconds</name><value>86400</value></property><!-- 启用自动恢复 --><property><name>yarn.resourcemanager.recovery.enabled</name><value>true</value></property><!-- 制定resourcemanager的状态信息存储在zookeeper集群上 --><property><name>yarn.resourcemanager.store.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value></property><!-- Whether virtual memory limits will be enforced for containers. --><property><name>yarn.nodemanager.vmem-check-enabled</name><value>false</value></property><property><name>yarn.nodemanager.vmem-pmem-ratio</name> <value>3</value></property>
分发
scp yarn-site.xml mapred-site.xml root@10.1.1.152:`pwd`scp yarn-site.xml mapred-site.xml root@10.1.1.195:`pwd`
启动测试
zkServer.sh start
start-dfs.sh
start-yarn.sh
启动命令(会产生两条警告信息):mr-jobhistory-daemon.sh start historyserver可以使用命令启动:mapred --daemon start historyserver关闭命令:mr-jobhistory-daemon.sh stop historyserver
测试
http://10.1.1.195:8088
成功
如果觉得《Hadoop HA《hadoop-Yarn》》对你有帮助,请点赞、收藏,并留下你的观点哦!