Feature Overview
COS introduces the SCF-based Log Cleansing feature for users to process log files. When you upload a log file or enable Logging, the cloud function preset by COS will be triggered, which automatically cleanses log messages using the SQL expression you set in the function, and delivers the result to the specified bucket and path.
Supports and Limits
Log Cleansing uses the data extraction APIs of COS. For more information about the restrictions, please see SELECT Overview.
If you have previously added a log cleansing rule for a bucket in the COS console, you can view the created log cleansing function in the SCF console. Please do not delete or modify this log cleansing function, as it may cause your rule to become ineffective.
Currently, Log Cleansing is available only in the Guangzhou, Shanghai, Beijing, and Chengdu regions.
The Log Cleansing feature depends on the SCF service, which provides users with a free tier. You will be billed for the part exceeding the free tier according to SCF Product Pricing.
Instructions
1. Log in to the COS Console.
2. In the left navigation pane, click Application Integration > Extended Features and locate Log Cleansing.
3. Click Configure Log Cleansing Rule to enter the rule configuration page.
4. Click Add Function.
Note
If you have not activated the SCF service, please go to the SCF console to activate it and follow the instructions to complete the service authorization.
5. In the pop-up window, enter the following information:
Function Name: Uniquely identifies a function and cannot be modified after being set. You can view the function in the SCF console.
Associated Bucket: a COS bucket that stores the log files.
Event Type: The event refers to the operation that triggers the cloud function. For example, in an upload operation, the method could be either
PUT Object or POST Object. When the event type is set to Put Method Creation, only log files uploaded via the PUT Object API will trigger log cleansing.Note
If your files are uploaded to the bucket through various channels such as simple upload, multipart upload, or cross-region replication, we recommend selecting the Create All event.
Trigger Condition: Specifies the path where the log file upload will trigger the cloud function. If a specific range is selected, the cloud function will only be triggered when the log file is uploaded to the specified path. If the entire bucket is selected, the cloud function will be triggered when the log file is uploaded to any location within the bucket.
SCF Authorization: Required. To cleanse a log file, SCF should be authorized to read the log file from your bucket and upload the cleansed log file to the specified location.
6. Click Next to configure log cleansing. The configuration options are as follows:
Source File Configuration: Choose the file type, row/column delimiter, list header, and compression format based on your requirements.
SQL Expression: Enter the desired query statement, or select one of the provided SQL templates. For more information on SQL commands, please refer to SQL Functions.
Output Configuration: includes the export format, column/row delimiter, and escape configuration.
7. Click Next to proceed with the delivery configuration. The configuration options are as follows:
Destination Bucket: A bucket that stores the generated file after the log cleansing is completed.
Destination Path: A path to deliver the log cleansing result. To avoid triggering a loop, you are advised to set it to a path whose prefix is different from that (if any) set in Trigger Condition.
Note
If the delivery prefix and trigger condition prefix have an inclusive relationship, it may cause a loop trigger. Please avoid this situation. For example, if the delivery prefix is prefix/target/ and the trigger condition prefix is prefix/, uploading a log file named prefix/test.csv will automatically generate and deliver the log cleansing result prefix/target/test.cos_select_result.csv to the bucket, which will trigger a loop cleansing.
8. After adding the configuration, click Confirm to see that the function has been added successfully. You can perform the following operations on the newly created function:
Click Log to view the historical running status of log cleansing. If an error is reported, you can click Log to quickly redirect to the SCF console to view its details.
Click More > Edit to modify the log cleansing rule.
Click More > Delete to remove unused log cleansing rules.