As an advanced open-source project of Apache, Spark is a swift, universal engine for large-scale data processing. It shares similarities with Hadoop's MapReduce computational framework. However, compared to MapReduce, Spark, with its scalable, memory-based computing features and the ability to directly read and write data in any format on Hadoop, is more efficient in batch processing and has lower latency. In fact, Spark has become a unified platform for lightweight, rapid big data processing. Various applications, such as real-time stream processing, machine learning, interactive queries, and more, can be established on different storage and operating systems via Spark.
Spark is a parallel computing framework for big data based on in-memory computing. By leveraging in-memory computation, Spark enhances the real-time data processing capabilities in a big data environment, while ensuring high fault tolerance and scalability. It allows users to deploy Spark on a multitude of inexpensive hardware, forming a cluster.
This tutorial demonstrates the submission of a word count task, which necessitates the prior upload of the file to be counted within the cluster.
1. Development Preparation
As the task requires access to Tencent Cloud Object Storage (COS), it is necessary to first create a storage bucket (Bucket) in COS.
Ensure that you have activated Tencent Cloud and created an EMR cluster. When creating the EMR cluster, you need to select the Spark component in the software configuration interface, and enable the authorization of object storage in Instance Information > Basic Configuration.
2. Creating a Project Using Maven
In this demonstration, we will not use the system's built-in demo program. Instead, we will create our own project, compile and package it, and then upload it to the EMR cluster for execution. It is recommended to use Maven to manage your project. Maven is a project management tool that can help you conveniently manage project dependencies. That is, it can obtain jar packages through the configuration of the pom.xml file, eliminating the need for manual addition.
First, download and install Maven, and configure Maven's environment variables. If you are using an IDE, please set the Maven related configuration in the IDE.
Create a new Maven project
Navigate to the directory where you want to create a new project in your local shell, for example, in
D://mavenWorkplace, and enter the following command to create a new Maven project:mvn archetype:generate -DgroupId=$yourgroupID -DartifactId=$yourartifactID -DarchetypeArtifactId=maven-archetype-quickstart
Where $yourgroupID is your package name. $yourartifactID is your project name, and maven-archetype-quickstart indicates the creation of a Maven Java project. Some files need to be downloaded during the project creation process, so please ensure a stable internet connection.
Upon successful creation, a project folder named $yourartifactID will be generated in the
D://mavenWorkplace directory. The file structure is as follows:simple---pom.xml Core configuration, located at the root of the project---src---main---java Java source code directory---resources Directory for Java configuration files---test---java Test source code directory---resources Test configuration directory
Our primary focus lies on the pom.xml file and the Java folder under main. The pom.xml file is primarily used for dependency and packaging configurations, while the Java folder houses your source code.
Firstly, add the Maven dependency in the pom.xml:
<dependencies><dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_2.11</artifactId><version>2.0.2</version></dependency></dependencies>
Proceed to add packaging and compilation plugins in the pom.xml:
<build><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><configuration><source>1.8</source><target>1.8</target><encoding>utf-8</encoding></configuration></plugin><plugin><artifactId>maven-assembly-plugin</artifactId><configuration><descriptorRefs><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs></configuration><executions><execution><id>make-assembly</id><phase>package</phase><goals><goal>single</goal></goals></execution></executions></plugin></plugins></build>
Right-click to create a new Java Class under src>main>Java, input your Class name, here we use WordCountOnCos, and add the sample code to the Class:
import java.util.Arrays;import org.apache.spark.SparkConf;import org.apache.spark.api.java.JavaRDD;import org.apache.spark.api.java.JavaSparkContext;import scala.Tuple2;/*** Created by tencent on 2018/6/28.*/public class WordCountOnCos {public static void main(String[] args){SparkConf sc = new SparkConf().setAppName("spark on cos");JavaSparkContext context = new JavaSparkContext(sc);JavaRDD<String> lines = context.textFile(args[0]);lines.flatMap(x -> Arrays.asList(x.split(" ")).iterator()).mapToPair(x -> new Tuple2<String, Integer>(x, 1)).reduceByKey((x, y) -> x+y).saveAsTextFile(args[1]);}}
If your Maven configuration is correct and the dependency packages have been successfully imported, then the entire project should be error-free and ready for compilation. Enter the project directory in the local command line mode and execute the following command to package the entire project:
mvn package
During the execution, additional files may need to be downloaded until a 'build success' message appears, indicating successful packaging. Subsequently, you can locate the packaged jar file in the target folder under the project directory.
Data Preparation
Initially, the compressed jar package needs to be uploaded to the EMR cluster using scp or sftp tools. Run the following command in the local command line mode:
scp $localfile root@PublicIPAddress:$remotefolder
Herein, $localfile represents the path and name of your local file; 'root' is the username of the CVM server; the public IP can be viewed in the node information of the EMR console or in the cloud server console; $remotefolder is the path on the CVM server where you wish to store the file. Upon completion of the upload, you can check in the EMR command line whether the corresponding folder contains the respective file.
The files to be processed need to be uploaded to COS in advance. If the files are local, they can be directly uploaded via the COS Console. If the files are on the EMR cluster, they can be uploaded using Hadoop commands. The directive is as follows:
[hadoop@10 hadoop]$ hadoop fs -put $testfile cosn://$bucketname/
Here, $testfile represents the full path and name of the file to be analyzed, and $bucketname is the name of your storage bucket. After the upload is complete, you can check in the COS console to see if the file is already in COS.
Execute Sample
Initially, it is necessary to log into any machine in the EMR cluster, preferably the Master node. For methods on how to log into EMR, please refer to Logging into Linux Instances. Here, we can opt to use WebShell for login. Click on the login button on the right side of the corresponding cloud server to enter the login interface. The default username is 'root', and the password is the one entered by the user when creating EMR. After entering correctly, you can access the command line interface.
In the EMR command line, use the following command to switch to the Hadoop user:
[root@172 ~]# su hadoop
Then, navigate to the folder where your jar package is stored and execute the following command:
[hadoop@10spark]$ spark-submit --class $WordCountOnCOS --masteryarn-cluster $packagename.jar cosn://$bucketname/$testfile cosn://$bucketname/output
Here, $WordCountOnCOS is the name of your Java Class, $packagename is the name of the jar package generated in your new Maven project, $bucketname is the name and path of your storage bucket, and $testfile is the name of the file you want to analyze. The final output file is located in the 'output' folder, this folder must not be created in advance, otherwise the operation will fail.
Upon successful execution, the results of the wordcount can be viewed in the designated storage bucket and folder.
[hadoop@172 /]$ hadoop fs -ls cosn://$bucketname/outputFound 3 items-rw-rw-rw- 1 hadoop Hadoop 0 2018-06-28 19:20 cosn://$bucketname/output/_SUCCESS-rw-rw-rw- 1 hadoop Hadoop 681 2018-06-28 19:20 cosn://$bucketname/output/part-00000-rw-rw-rw- 1 hadoop Hadoop 893 2018-06-28 19:20 cosn://$bucketname/output/part-00001[hadoop@172 demo]$ hadoop fs -cat cosn://$bucketname/output/part-0000018/07/05 17:35:01 INFO cosnative.NativeCosFileSystem: Opening 'cosn://$bucketname/output/part-00000' for reading(under,1)(this,3)(distribution,2)(Technology,1)(country,1)(is,1)(Jetty,1)(currently,1)(permitted.,1)(Security,1)(have,1)(check,1)