The content of this page has been automatically translated by AI. If you encounter any problems while reading, you can view the corresponding content in Chinese.

EMR Log access

Last updated: 2024-10-25 09:09:53

Prerequisites

You have a Tencent Cloud account. For more information on how to create an account, please see Signing up for a Tencent Cloud Account.
If using a sub-account to log in, please ensure the account has read and write permissions for ES.

Operation Steps

Logging in to the Console

1. Log in to the ES Console.
2. In the left sidebar under Serverless mode, select Log Analysis.

Create Project Space

1. click Create a new space.
2. Enter the project space name , supporting 1 - 20 Chinese characters, English letters, numbers, underscores, or the separator "-".
3. click Confirm. If the verification is correct, the project space will be successfully created.
Search for the required CAM policy as needed, and click to complete policy association.

Note:
In ES Serverless log analysis, you can create an index, and then write data through the API or access data sources such as CVM or TKE on the corresponding index's "Data Access" Tab page. You can also access data while creating the index to accomplish one-stop CVM log access, TKE log access, and EMR (EMR) log access. Below is an introduction to the one-stop EMR log access operation.

EMR log access

1. On the ES Serverless log analysis homepage, select EMR to go to the EMR log access page.
Search for the required CAM policy as needed, and click to complete policy association.

2. Go to the data source settings page, configure the data source, and after completing the configuration, click Next.
Region: Select the region of the EMR cluster. If you enter this page from a project space details page, the default is the same as the project space region.
VPC VPC: The VPC where the EMR cluster is located.
EMR Cluster : The EMR cluster from which logs need to be collected.
Log Type : You can specify the component's running logs to be collected. For instance, if there is a YARN component in the cluster, task logs can be collected.
Collection Strategy : Supports both full and incremental collection. If incremental collection is selected, only logs generated after data access will be collected.
Search for the required CAM policy as needed, and click to complete policy association.

3. Enter the index settings page to set up the index.
Region Location : The region where the project space is located, default is consistent with the EMR cluster.
Project Space : You can assign all indexes of the same business to a project space for easier management.
Index Name : Length is 1 - 100, supports lowercase letters, numbers, -, _, ;, @, &, =, !, ', %, $, ., +, (, ).
Field Mapping
Dynamic Generation: Enabled by default. Once enabled, it will automatically parse the written data and set the index fields.
After disabling Auto Generate for input sample configuration, you can use Input Sample Auto Configuration to generate field mappings for the index. Enter a JSON formatted data sample in the input box. After confirmation, the platform will automatically validate it. Once validated correctly, the relevant fields will be mapped to the field mapping table.
Field Mapping splits the original data into multiple tokens by field (i.e., key:value) for index construction and performs retrieval based on this mapping. Details are as follows:
Parameter
Feature Description
Field Name
Field names in the written data
Field Type
Field data types supported in the interface: "text,date,boolean,keyword,long,double,integer,ip,geo_point"—a total of 9 types. More field types are supported in JSON Edit Mode. For details, refer to the Official Documentation
Allow Chinese Characters
This feature can be enabled when the field contains Chinese and you need to retrieve the Chinese content. After enabling, the ik_max_word tokenizer will be used by default for the text field
Enabling index
After enabling, an index will be built for the field to be used for retrieval
Enable Statistics
After enabling, statistical analysis can be performed on the field values, which will increase index storage
Time Field The time field refers to the field of the type date in the actual data. Once the index is successfully created, this field cannot be changed.
Note:
The time field is enabled by default with Indexing Enabled and Statistics Enabled, and cannot be turned off.
Data Storage Duration:
You can set the data storage duration. By default, it is set to 30 days, and permanent storage is also supported.
Search for the required CAM policy as needed, and click to complete policy association.