blocks|key|3181749|text|当通过设置spark.cores.max和spark.executor.memory+resp‘请求的核心数量或内存数量(每个节点)超过可用时，您就会得到这个结果。因此，即使没有其他人使用集群，并且您指定要使用，假设每个节点使用100+get，但是节点只能支持90+get，那么您将收到此错误消息。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|3181750|公平地说，在这种情况下，这个信息是模糊的，如果它说你超过了最大值会更有帮助。|3181751|entityMap^0|5|F|L|L|0|0^^$0|@$1|2|3|4|5|6|7|J|8|@$9|K|A|L|B|C]|$9|M|A|N|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|O|8|@]|D|@]|E|$]]|$1|H|3|-4|5|6|7|P|8|@]|D|@]|E|$]]]|I|$]]

You get this when either the number of cores or amount of RAM (per node) you request via setting <code>spark.cores.max</code> and <code>spark.executor.memory</code> resp' exceeds what is available. Therefore even if no one else is using the cluster, and you specify you want to use, say 100GB RAM per node, but your nodes can only support 90GB, then you will get this error message.

To be fair the message is vague in this situation, it would be more helpful if it said your exceeding the maximum.

blocks|key|719089|text|看来火花公子主人不能为这个任务指派任何工人。要么工人没有开始工作，要么他们都很忙。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|719090|检查主节点上的Spark+(默认情况下由SPARK_MASTER_WEBUI_PORT在spark-env.sh中指定端口，8080+)。它应该是这样的：|offset|length|style|CODE|719091|📷|atomic|719092|​|719093|为使集群正常运作：|719094|一定有一些国家“活着”的工人|unordered-list-item|719095|必须有一些可用的核心(例如，如果所有核心都忙于冻结的任务，集群将不接受新任务)|719096|必须有足够的内存可用|719097|719098|719099|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://developer.qcloudimg.com/http-save/yehe-900000/039bfe1342900c883fe16e4fa3728e44.png|imageAlt^0|0|K|N|18|C|0|0|1|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|16|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|17|8|@$D|18|E|19|F|G]|$D|1A|E|1B|F|G]]|9|@]|A|$]]|$1|H|3|I|5|J|7|1C|8|@]|9|@$D|1D|E|1E|1|1F]]|A|$]]|$1|K|3|L|5|6|7|1G|8|@]|9|@]|A|$]]|$1|M|3|N|5|6|7|1H|8|@]|9|@]|A|$]]|$1|O|3|P|5|Q|7|1I|8|@]|9|@]|A|$]]|$1|R|3|S|5|Q|7|1J|8|@]|9|@]|A|$]]|$1|T|3|U|5|Q|7|1K|8|@]|9|@]|A|$]]|$1|V|3|-4|5|6|7|1L|8|@]|9|@]|A|$]]|$1|W|3|-4|5|6|7|1M|8|@]|9|@]|A|$]]|$1|X|3|-4|5|6|7|1N|8|@]|9|@]|A|$]]]|Y|$Z|$5|10|11|12|A|$13|14|15|-4]]]]

Looks like Spark master can't assign any workers for this task. Either the workers aren't started or they are all busy. 

Check Spark UI on master node (port specified by <code>SPARK_MASTER_WEBUI_PORT</code> in <code>spark-env.sh</code>, 8080 by default). It should look like this: <img src="https://i.stack.imgur.com/AY3Wh.png" alt="SparkUI Sample">

For cluster to function properly:

<ul>
<li>There must be some workers with state "Alive"</li>
<li>There must be some cores available (for example, if all cores are busy with the frozen task, the cluster won't accept new tasks)</li>
<li>There must be sufficient memory available</li>
</ul>

blocks|key|719160|text|另外，确保你的火花工作人员可以与司机进行双向交流。检查防火墙等。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|719161|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Also make sure your spark workers can communicate both ways with the driver. Check for firewalls, etc.

blocks|key|171685|text|我确实有过这个问题。我有一个简单的1节点星火集群，当我试图运行我的星火应用程序时，我得到了这个错误。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|171686|我浏览了上面的一些建议，当我试图针对集群运行Spark而无法在UI中看到这一点时，我开始怀疑我的集群没有正常工作。|171687|在我的主机文件中，我有一个条目，比如SparkNode，它引用了正确的IP地址。|offset|length|style|CODE|171688|我无意中将conf/spark-env.sh文件中的IP地址与SPARK_MASTER_IP变量放在一起。我将其更改为SparkNode，并将SPARK_LOCAL_IP更改为指向SparkNode。|171689|为了测试这一点，我在浏览器中使用SparkNode:7077打开了UI，我可以看到一个正在运行的Spark实例。|171690|然后，我使用Wildfires建议运行shell，如下所示：|171691|MASTER=spark://SparkNode:7077+bin/spark-shell|code-block|syntax|javascript|171692|回到UI，我现在可以看到shell应用程序正在运行，而我以前无法这样做。|171693|因此，我退出了shell，并使用Submit运行了我的应用程序，现在它正常工作了。|171694|绝对值得检查所有的IP和主机条目，这是我的问题的根本原因。|171695|entityMap^0|0|0|I|9|0|5|H|V|F|1N|9|1Z|E|2I|9|0|G|E|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|12|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|13|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|14|8|@$F|15|G|16|H|I]]|9|@]|A|$]]|$1|J|3|K|5|6|7|17|8|@$F|18|G|19|H|I]|$F|1A|G|1B|H|I]|$F|1C|G|1D|H|I]|$F|1E|G|1F|H|I]|$F|1G|G|1H|H|I]]|9|@]|A|$]]|$1|L|3|M|5|6|7|1I|8|@$F|1J|G|1K|H|I]]|9|@]|A|$]]|$1|N|3|O|5|6|7|1L|8|@]|9|@]|A|$]]|$1|P|3|Q|5|R|7|1M|8|@]|9|@]|A|$S|T]]|$1|U|3|V|5|6|7|1N|8|@]|9|@]|A|$]]|$1|W|3|X|5|6|7|1O|8|@]|9|@]|A|$]]|$1|Y|3|Z|5|6|7|1P|8|@]|9|@]|A|$]]|$1|10|3|-4|5|6|7|1Q|8|@]|9|@]|A|$]]]|11|$]]

I had this exact issue. I had a simple 1-node Spark cluster and was getting this error when trying to run my Spark app.

I ran through some of the suggestions above and it was when I tried to run the Spark shell against the cluster and not being able to see this in the UI that I became suspicious that my cluster was not working correctly.

In my hosts file I had an entry, let's say <code>SparkNode</code>, that referenced the correct IP Address.

I had inadvertently put the wrong IP Address in the <code>conf/spark-env.sh</code> file against the <code>SPARK_MASTER_IP</code> variable. I changed this to <code>SparkNode</code> and I also changed <code>SPARK_LOCAL_IP</code> to point to <code>SparkNode</code>.

To test this I opened up the UI using <code>SparkNode:7077</code> in the browser and I could see an instance of Spark running.

I then used Wildfires suggestion of running the Spark shell, as follows: 

<pre><code>MASTER=spark://SparkNode:7077 bin/spark-shell
</code></pre>

Going back to the UI I could now see the Spark shell application running, which I couldn't before.

So I exited the Spark shell and ran my app using Spark Submit and it now works correctly.

It is definitely worth checking out all of your IP and host entries, this was the root cause of my problem.

blocks|key|719181|text|您需要指定正确的SPARK_HOME和驱动程序的IP地址，以防火花可能无法找到您的Netty服务器。请注意，您的火花主人应该听正确的IP地址，你应该使用。这可以通过在文件spark-env.sh中设置SPARK_MASTER_IP=yourIP来实现。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|719182|+++val+conf+=+new+SparkConf()
++.setAppName("test")
++.setMaster("spark://yourSparkMaster:7077")
++.setSparkHome("YourSparkHomeDir")
++.set("spark.driver.host",+"YourIPAddr")|code-block|syntax|javascript|719183|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

You need to specify the right SPARK_HOME and your driver program's IP address in case Spark may not able to locate your Netty jar server. Be aware that your Spark master should listen to the correct IP address which you suppose to use. This can be done by setting SPARK_MASTER_IP=yourIP in file spark-env.sh.

<pre><code> val conf = new SparkConf()
 .setAppName("test")
 .setMaster("spark://yourSparkMaster:7077")
 .setSparkHome("YourSparkHomeDir")
 .set("spark.driver.host", "YourIPAddr")
</code></pre>

blocks|key|171635|text|检查与主机名、IP地址和回送有关的错误。确保设置SPARK_LOCAL_IP和SPARK_MASTER_IP。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|171636|entityMap^0|O|E|13|F|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]|$9|K|A|L|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|M|8|@]|D|@]|E|$]]]|G|$]]

Check for errors regard to hostname, IP address and loopback. Make sure to set <code>SPARK_LOCAL_IP</code> and <code>SPARK_MASTER_IP</code>.

blocks|key|3181939|text|我有类似的问题，最初的工作没有接受任何资源，通过指定火花正确下载url上的火花-env.sh或安装火花在所有奴隶。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3181940|export+SPARK_EXECUTOR_URI=http://mirror.fibergrid.in/apache/spark/spark-1.6.1/spark-1.6.1-bin-hadoop2.6.tgz|code-block|syntax|javascript|3181941|初始作业未接受任何资源；请检查群集用户界面，以确保工作人员已注册并拥有足够的内存。|3181942|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

I had similar issue Initial job has not accepted any resource, fixed it by specify the spark correct download url on spark-env.sh or installing spark on all slaves.
<pre><code>export SPARK_EXECUTOR_URI=http://mirror.fibergrid.in/apache/spark/spark-1.6.1/spark-1.6.1-bin-hadoop2.6.tgz
</code></pre>
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

I have a Apache Spark 0.9.0 Cluster installed where I am trying to deploy a code which reads a file from HDFS. This piece of code throws a warning and eventually the job fails. Here is the code 

<pre><code>/**
 * running the code would fail 
 * with a warning 
 * Initial job has not accepted any resources; check your cluster UI to ensure that 
 * workers are registered and have sufficient memory
 */

object Main extends App {
 val sconf = new SparkConf()
 .setMaster("spark://labscs1:7077")
 .setAppName("spark scala")
 val sctx = new SparkContext(sconf)
 sctx.parallelize(1 to 100).count
}
</code></pre>

The below is the WARNING message

<blockquote>
 Initial job has not accepted any resources; check your cluster UI to
 ensure that workers are registered and have sufficient memory
</blockquote>

how to get rid of this or am I missing some configurations.

Running a Job on Spark 0.9.0 throws error

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我已经安装了一个ApacheSpark0.9.0集群，在这里我试图部署一个从HDFS读取文件的代码。这段代码抛出一个警告，最终作业失败。这是代码/** * running the code would fail  * with a warning  * Initial job has not accepted any resources; check your cluster UI to ensu

问在Spark0.9.0抛出错误上运行作业
EN

回答 7

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在Spark0.9.0抛出错误上运行作业EN

回答 7

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在Spark0.9.0抛出错误上运行作业
EN