blocks|key|32238|text|您需要下载winutils可执行文件，而不是源代码。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|32239|你可以下载它，或者如果你真的想要整个here发行版，你可以找到2.6.0二进制文件here。然后，您需要将HADOOP_HOME设置为包含winutils.exe的目录。|offset|length|style|CODE|32240|另外，要确保你放置的目录不包含空格Spark，这一点非常重要，否则它将无法工作。|BOLD|32241|设置好后，不是启动spark-shell.sh，而是启动spark-shell.cmd|32242|C:\Spark\bin>spark-shell
log4j:WARN+No+appenders+could+be+found+for+logger+(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN+Please+initialize+the+log4j+system+properly.
log4j:WARN+See+http://logging.apache.org/log4j/1.2/faq.html#noconfig+for+more+info.
Using+Spark's+repl+log4j+profile:+org/apache/spark/log4j-defaults-repl.properties
To+adjust+logging+level+use+sc.setLogLevel("INFO")
Welcome+to
++++++____++++++++++++++__
+++++/+__/__++___+_____/+/__
++++_\+\/+_+\/+_+`/+__/++'_/
+++/___/+.__/\_,_/_/+/_/\_\+++version+1.6.1
++++++/_/

Using+Scala+version+2.10.5+(Java+HotSpot(TM)+64-Bit+Server+VM,+Java+1.8.0_91)
Type+in+expressions+to+have+them+evaluated.
Type+:help+for+more+information.
Spark+context+available+as+sc.
16/05/18+19:31:56+WARN+General:+Plugin+(Bundle)+"org.datanucleus"+is+already+registered.+Ensure+you+dont+have+multiple+JAR+versions+of+the+same+plugin+in+the+classpath.+The+URL+"file:/C:/Spark/lib/datanucleus-core-3.2.10.jar"+is+already+registered,+and+you+are+trying+to+register+an+identical+plugin+located+at+URL+"file:/C:/Spark/bin/../lib/datanucleus-core-3.2.10.jar."
16/05/18+19:31:56+WARN+General:+Plugin+(Bundle)+"org.datanucleus.api.jdo"+is+already+registered.+Ensure+you+dont+have+multiple+JAR+versions+of+the+same+plugin+in+the+classpath.+The+URL+"file:/C:/Spark/lib/datanucleus-api-jdo-3.2.6.jar"+is+already+registered,+and+you+are+trying+to+register+an+identical+plugin+located+at+URL+"file:/C:/Spark/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
16/05/18+19:31:56+WARN+General:+Plugin+(Bundle)+"org.datanucleus.store.rdbms"+is+already+registered.+Ensure+you+dont+have+multiple+JAR+versions+of+the+same+plugin+in+the+classpath.+The+URL+"file:/C:/Spark/lib/datanucleus-rdbms-3.2.9.jar"+is+already+registered,+and+you+are+trying+to+register+an+identical+plugin+located+at+URL+"file:/C:/Spark/bin/../lib/datanucleus-rdbms-3.2.9.jar."
16/05/18+19:31:56+WARN+Connection:+BoneCP+specified+but+not+present+in+CLASSPATH+(or+one+of+dependencies)
16/05/18+19:31:56+WARN+Connection:+BoneCP+specified+but+not+present+in+CLASSPATH+(or+one+of+dependencies)
16/05/18+19:32:01+WARN+ObjectStore:+Version+information+not+found+in+metastore.+hive.metastore.schema.verification+is+not+enabled+so+recording+the+schema+version+1.2.0
16/05/18+19:32:01+WARN+ObjectStore:+Failed+to+get+database+default,+returning+NoSuchObjectException
16/05/18+19:32:07+WARN+General:+Plugin+(Bundle)+"org.datanucleus"+is+already+registered.+Ensure+you+dont+have+multiple+JAR+versions+of+the+same+plugin+in+the+classpath.+The+URL+"file:/C:/Spark/lib/datanucleus-core-3.2.10.jar"+is+already+registered,+and+you+are+trying+to+register+an+identical+plugin+located+at+URL+"file:/C:/Spark/bin/../lib/datanucleus-core-3.2.10.jar."
16/05/18+19:32:07+WARN+General:+Plugin+(Bundle)+"org.datanucleus.api.jdo"+is+already+registered.+Ensure+you+dont+have+multiple+JAR+versions+of+the+same+plugin+in+the+classpath.+The+URL+"file:/C:/Spark/lib/datanucleus-api-jdo-3.2.6.jar"+is+already+registered,+and+you+are+trying+to+register+an+identical+plugin+located+at+URL+"file:/C:/Spark/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
16/05/18+19:32:07+WARN+General:+Plugin+(Bundle)+"org.datanucleus.store.rdbms"+is+already+registered.+Ensure+you+dont+have+multiple+JAR+versions+of+the+same+plugin+in+the+classpath.+The+URL+"file:/C:/Spark/lib/datanucleus-rdbms-3.2.9.jar"+is+already+registered,+and+you+are+trying+to+register+an+identical+plugin+located+at+URL+"file:/C:/Spark/bin/../lib/datanucleus-rdbms-3.2.9.jar."
16/05/18+19:32:07+WARN+Connection:+BoneCP+specified+but+not+present+in+CLASSPATH+(or+one+of+dependencies)
16/05/18+19:32:08+WARN+Connection:+BoneCP+specified+but+not+present+in+CLASSPATH+(or+one+of+dependencies)
16/05/18+19:32:12+WARN+ObjectStore:+Version+information+not+found+in+metastore.+hive.metastore.schema.verification+is+not+enabled+so+recording+the+schema+version+1.2.0
16/05/18+19:32:12+WARN+ObjectStore:+Failed+to+get+database+default,+returning+NoSuchObjectException
SQL+context+available+as+sqlContext.

scala>|code-block|syntax|javascript|32243|entityMap|0|LINK|mutability|MUTABLE|url|http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe|1|http://www.barik.net/archive/2015/01/19/172716/^0|0|1H|B|I|4|0|15|4|1|0|3|6|0|9|E|S|F|0|0^^$0|@$1|2|3|4|5|6|7|11|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|12|8|@$D|13|E|14|F|G]]|9|@$D|15|E|16|1|17]|$D|18|E|19|1|1A]]|A|$]]|$1|H|3|I|5|6|7|1B|8|@$D|1C|E|1D|F|J]]|9|@]|A|$]]|$1|K|3|L|5|6|7|1E|8|@$D|1F|E|1G|F|G]|$D|1H|E|1I|F|G]]|9|@]|A|$]]|$1|M|3|N|5|O|7|1J|8|@]|9|@]|A|$P|Q]]|$1|R|3|-4|5|6|7|1K|8|@]|9|@]|A|$]]]|S|$T|$5|U|V|W|A|$X|Y]]|Z|$5|U|V|W|A|$X|10]]]]

You need to download the winutils executable, not source code.

You can download it <a href="http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe" rel="noreferrer">here</a>, or if you really want the entire Hadoop distribution you can find the 2.6.0 binaries <a href="http://www.barik.net/archive/2015/01/19/172716/" rel="noreferrer">here</a>. Then, you need to set <code>HADOOP_HOME</code> to the directory containing winutils.exe. 

Also, make sure the directory you place Spark in is a directory that doesn't contain whitespaces, this is extremely important otherwise it won't work.

Once you've set it up, you don't start <code>spark-shell.sh</code>, you start <code>spark-shell.cmd</code>:

<pre><code>C:\Spark\bin&gt;spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
 ____ __
 / __/__ ___ _____/ /__
 _\ \/ _ \/ _ `/ __/ '_/
 /___/ .__/\_,_/_/ /_/\_\ version 1.6.1
 /_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_91)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
16/05/18 19:31:56 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-core-3.2.10.jar."
16/05/18 19:31:56 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
16/05/18 19:31:56 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-rdbms-3.2.9.jar."
16/05/18 19:31:56 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:31:56 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:32:01 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/05/18 19:32:01 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/05/18 19:32:07 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-core-3.2.10.jar."
16/05/18 19:32:07 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
16/05/18 19:32:07 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-rdbms-3.2.9.jar."
16/05/18 19:32:07 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:32:08 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:32:12 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/05/18 19:32:12 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
SQL context available as sqlContext.

scala&gt;
</code></pre>

blocks|key|2705599|text|在windows上，您需要显式指定hadoop二进制文件的位置。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2705600|下面是设置spark-scala独立应用程序的步骤。|2705601|2705602|下载winutil.exe并将其放到bin文件夹下的某个文件夹/目录中，例如c:\hadoop\bin|ordered-list-item|2705603|2705604|完整的路径看起来像c:\hadoop\bin\winutil.exe|2705605|2705606|现在，在创建sparkSession时，我们需要指定此路径。请参考下面的代码片段：|2705607|2705608|包com.test.config|2705609|导入org.apache.spark.sql.SparkSession对象Spark2Config扩展了可序列化的{+System.setProperty("hadoop.home.dir"，"C:\hadoop")+val+spark+=}|2705610|entityMap^0|0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|V|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|W|8|@]|9|@]|A|$]]|$1|E|3|F|5|G|7|X|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|Y|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|Z|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|10|8|@]|9|@]|A|$]]|$1|L|3|M|5|G|7|11|8|@]|9|@]|A|$]]|$1|N|3|-4|5|6|7|12|8|@]|9|@]|A|$]]|$1|O|3|P|5|6|7|13|8|@]|9|@]|A|$]]|$1|Q|3|R|5|6|7|14|8|@]|9|@]|A|$]]|$1|S|3|-4|5|6|7|15|8|@]|9|@]|A|$]]]|T|$]]

on windows you need to explicitly specify as where to locate the hadoop binaries.
below are steps to set up spark-scala standalone application.
<ol>
<li>download winutil.exe and place it to some folder/directory under bin folder say c:\hadoop\bin</li>
</ol>
complete path look like c:\hadoop\bin\winutil.exe
<ol start="2">
<li>now while creating the sparkSession we need to specify this path. Refer the code snippet below:
package com.test.config
<pre><code> import org.apache.spark.sql.SparkSession

 object Spark2Config extends Serializable{
 System.setProperty(&quot;hadoop.home.dir&quot;, &quot;C:\\hadoop&quot;)
 val spark = SparkSession.builder().appName(&quot;app_name&quot;).master(&quot;local&quot;).getOrCreate() 
 }
</code></pre>
</li>
</ol>

I am trying to install Spark 1.6.1 on windows 10 and so far I have done the following...

<ol>
<li>Downloaded spark 1.6.1, unpacked to some directory and then set SPARK_HOME</li>
<li>Downloaded scala 2.11.8, unpacked to some directory and then set SCALA_HOME</li>
<li>Set the _JAVA_OPTION env variable</li>
<li>Downloaded the winutils from <a href="https://github.com/steveloughran/winutils.git" rel="nofollow">https://github.com/steveloughran/winutils.git</a> by just downloading the zip directory and then set HADOOP_HOME env variable. (Not sure if this was incorrect, I could not clone the directory because of permission denied).</li>
</ol>

When I go to spark home and run bin\spark-shell I get

<pre><code>'C:\Program' is not recognized as an internal or external command, operable program or batch file.
</code></pre>

I must be missing something, I don't see how I could be running the bash scripts anyway from windows environment. But hopefully I don't need to understand just to get this working. I have been following this guy's tutorial - <a href="https://hernandezpaul.wordpress.com/2016/01/24/apache-spark-installation-on-windows-10/" rel="nofollow">https://hernandezpaul.wordpress.com/2016/01/24/apache-spark-installation-on-windows-10/</a> . Any help would be appreciated.

winutils spark windows installation env_variable

Spark 

Java

我正在尝试在Windows10上安装Spark 1.6.1，到目前为止我已经完成了以下工作……spark 下载了spark 1.6.1，解压到某个目录，然后设置SPARK_HOMEDownloaded scala 2.11.8，解压到某个目录，然后设置SCALA_HOME，只需下载压缩目录，然后设置_JAVA_OPTI...

问winutils spark windows安装env_variable
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问winutils spark windows安装env_variableEN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问winutils spark windows安装env_variable
EN