我正在使用uhopper / hadoop docker image来创建纱线簇.我有3个节点,每个节点有64GB RAM.我添加了配置.我给了32GB
到纱线.总簇内存为96GB.
- name: YARN_CONF_yarn_scheduler_minimum___allocation___mb
value: "2048"
- name: YARN_CONF_yarn_scheduler_maximum___allocation___mb
value: "16384"
- name: MAPRED_CONF_mapreduce_framework_name
value: "yarn"
- name: MAPRED_CONF_mapreduce_map_memory_mb
value: "8192"
- name: MAPRED_CONF_mapreduce_reduce_memory_mb
value: "8192"
- name: MAPRED_CONF_mapreduce_map_java_opts
value: "-Xmx8192m"
- name: MAPRED_CONF_mapreduce_reduce_java_opts
value: "-Xmx8192m"
- name: YARN_CONF_yarn_nodemanager_resource_memory___mb
value: "32768"
最大应用程序主资源为10240 MB.我运行了5个火花作业,每个3 GB驱动程序内存,2个作业从未进入RUNNING状态,因为10240MB.我无法充分利用我的硬件.
如何增加Max Application Master Resources内存？
最佳答案
我希望,我找到了答案,如果你改变yarn.scheduler.capacity.maximum-am-resource-percent,那么Max Application Master Resources将会改变.这是一个文档 – 来自docs.hortonworks.com的Setting Application Limits
让我知道它是否有效.
