Background
With bot traffic analysis, you can collect data from bot traffic management and quickly understand how a selected domain name with bot traffic analysis enabled is affected by bots. You can also view the bot classification trend, action trend, bot score distribution, top statistics by request count, and list of URLs vulnerable to attacks. Additionally, you can click View Details to see details about the corresponding source of access, discover access characteristics, and understand bot anomalies.
In the Bot Traffic Details section, you can quickly identify the top 10 sources of access for enabled bot traffic analysis features. If session settings are configured, you can also view session access information for the top 10 access sources, quickly navigate to the details and logs of the selected source of access. You can also perform targeted searches for information on specific sources of access/IPs.
Prerequisites
Directions
1. Log in to the WAF console and select Bot Traffic Analysis > Bot Traffic Details from the left sidebar.
2. On the Bot Traffic Details page, click the "All Domains" dropdown in the upper left corner and select the domain name you want to view.
3. On the Bot Traffic Details page, you can search for the top 10 traffic information for all domains or a specific domain by time or filter and quickly navigate to the relevant logs.
4. On the Bot Traffic Details page, select the desired source of access and click View Details to access the details page for that source.
View Overview
In the Overview module of the Details page, it displays the source of access risk value and hit policy information. You can also quickly target the source by viewing access logs, adding to the allowlist, blocklist, or defining custom rules.
Field Descriptions:
IP address and Tag: Basic information of the source IP and the hit Tag information.
Last request score: The score identified by the Bot during the last access of this source and the current access risk situation.
Number of sessions: Displays the total number of visits to the website by this source in the most recent continuous session. The total count may differ from the Bot action statistics on the right. This shows the session count of the current access source during the latest visit session.
Access destination: Displays the domain name accessed by the source.
Abnormal features: Displays the modules where abnormal features of the access source appeared.
Hit module: The module that triggered the action during the last disposal action.
Policy ID: Displays the policy ID hit by this source of access.
View access logs: Redirect to the access logs page to view the detailed access information of this source.
Add to allowlist: Add this source of access to the allowlist.
Add to blocklist: Add this source of access to the blocklist.
Add custom rules: Add custom rules for this access source.
View Bot score and proportion
In the Bot score and proportion module on the details page, it displays the Bot action proportion and Bot score distribution during the selected access time period. Through the Bot score distribution, you can quickly find the overall threat level of the current access source.
View Bot Information
In the detailed page of the Bot Information module, the basic characteristics, request features, threat intelligence, AI evaluation, smart statistics, and session features of the source of access bots are displayed. Through these features, you can quickly identify anomalies in current access requests, discover differences among various access sources, and swiftly handle such bots.
Basic Session Information
In the Basic Session Information tab, the basic IP information and session information of the current source of access are displayed. When the dimension displayed is the session ID, the basic IP information is not shown.
Field Descriptions:
IP Basic Information
Source IP: The IP address of the current source of access.
City: The city where the current source of access belongs.
IP Region: The country where the current source of access belongs.
IP Type: The type of IP of the current source of access.
IP Owner: The owner of the IP of the current source of access.
Session Basic Information
Average Session Speed: The average session speed of the current source of access in the latest session, calculated as the total number of session requests divided by the session duration, in requests per minute.
Total Number of Sessions: The total number of sessions in the latest session of the current source of access.
Robots.txt Access: Whether the current source of access has accessed the Robots.txt file in the latest session. Sessions that access this file are typically those of crawler bots.
Session Duration: The duration of the latest session of the current source of access.
Request Feature Information
On the request feature information tab, display the request feature information, Cookie information, User-Agent information, Referer information, and Query information of the current session request.
Field Descriptions:
Information Type | Information Name | Information Details |
Request Feature Information | URL Repetition Ratio | The URL repetition ratio in session requests, ranging from 0 to 1. Parameter configuration is based on actual business conditions. A ratio that is too high or too low is considered suspicious (judgment based on actual conditions). |
| Total URL Types | The number of deduped URLs in session requests. The number of deduped URLs in session requests. |
| URL Minimum Depth | The minimum level of URL in session requests. The minimum category level of URL in session requests. |
| URL Maximum Depth | The maximum number of URL levels in session requests. The maximum category level of URL in session requests. |
| URL Average Depth | The average number of URL levels in session requests. The average category level of URL in session requests. |
| URL Quantity | The total number of accessed URLs in session requests (not deduplicated). |
Cookie Information | Whether cookies are abused | Different UAs use the same cookie. |
| Cookie Existence | Whether cookies exist in session requests. |
| Cookie Repetition | The repetition proportion of cookies in session requests, ranging from 0 to 1. |
| Cookie Efficiency | The proportion of cookies that can be parsed correctly in session requests. |
| Most Frequent Cookie | The most frequent cookie in session requests. |
| Proportion of Most Frequent Cookie | The proportion of the most frequent cookie in session requests. |
User-Agent information | User-Agent Type | The type of User-Agent for accessing users in session requests. |
| User-Agent Existence | In session requests, check if the User-Agent field exists in the HTTP header. |
| User-Agent Randomness Index | The random distribution of UA in session requests, ranging from 0 to 1. The higher the index, the more abnormal it is. Reference threshold: suspected abnormal if exceeding 0.6, confirmed abnormal if exceeding 0.92. |
| User-Agent Category | The deduplicated number of UAs in session requests. Excessive number may indicate suspected abnormal (to be judged based on actual conditions), applicable to non-agent IPs. |
| User-Agent Effective Rate | The existence ratio of UA in session requests, ranging from 0 to 1. Too low indicates suspected abnormal (to be judged based on actual conditions). |
| Most Frequent User-Agent | In session requests, the most frequent value of the HTTP User-Agent field. |
| The Proportion of the Most Frequent User-Agent | In session requests, the proportion of the most frequent value of the HTTP User-Agent field in the overall traffic. |
| User-Agent Similarity Ratio | The similarity ratio between the most frequent value of the HTTP User-Agent field and other access requests in session requests. |
Referer information | Referer Repetition Ratio | The repetition ratio of Referer in session requests, ranging from 0 to 1, applicable to browser visits. Excessively high indicates suspected abnormal (to be judged based on actual conditions). |
| Referer Existence Ratio | The existence ratio of Referer in session requests, ranging from 0 to 1, applicable to browser visits. Too low indicates suspected abnormal (to be judged based on actual conditions). |
| Referer Effective Rate | The effective ratio of Referer in session requests, ranging from 0 to 1, applicable to browser visits. Too low indicates suspected abnormal (to be judged based on actual conditions). |
| Whether Referer is Abused | Various different UAs using the same Referer in the same session request. |
| Most Frequent Referer | In session requests, the most frequent value of the HTTP Referer field. |
| Proportion of Most Frequent Referer | In session requests, the proportion of the most frequent value of the HTTP Referer field in the requests. |
Query Information | Request Parameter Ratio | The repetition ratio of GET request parameters (Query Content) or POST request parameters (Body Content) in session requests, ranging from 0 to 1. Parameters should be configured based on actual business conditions. Excessively high or too low indicates suspected abnormal (to be judged based on actual conditions). |
| Request Parameter Category | The most frequent session request parameters, including GET Request Parameters (Query Content) or POST Request Parameters (Body Content). |
Threat Intelligence Module
In the Threat Intelligence tab, if the current Access Source IP/Session ID matches relevant threat intelligence information, it will display IDC Details of the access source and the matched threat intelligence information.
IDC Details: If the access source information includes IDC, it will display the current IDC information.
Threat Intelligence: If the Access Source IP information includes threat intelligence information, it will display the Tags of the matched threat intelligence and their corresponding explanations.
AI Evaluation Module
In the AI Evaluation tab, it will display the anomaly degree and corresponding Metric Feature Value of each dimension's anomaly feature information in the current session request. The displayed content includes Request Feature Anomaly Information, Cookie Anomaly Information, User-Agent Anomaly Information, Referer Anomaly Information, and Query Anomaly Information. If the AI derived anomaly degree is greater than 0, it indicates that the AI evaluation has found anomalous feature parameters.
Bot Flow Statistics Module
In the Bot Flow Statistics tab, it will display the latest anomaly feature information of each dimension and the current eigenvalue of the corresponding features in the current session request, as well as the normal client's Feature Threshold.
Field Descriptions:
Information Name | Information Details |
Session Average Speed Anomaly | The current session's Average Rate Feature Value is anomalous. If there is a session average speed anomaly, it will display the number of anomalies in the current session and the Critical Value of this feature in the current domain for abnormal access sessions. |
User-Agent Category Anomaly | An abnormal Value has been detected for the User-Agent Type Feature of the current session. If the session's User-Agent type is anomalous, the number of abnormal User-Agent type values in the current session will be displayed, along with the threshold value for this feature in the overall dashboard of abnormal access sessions. |
URL Type Anomaly | An abnormal value has been detected for the URL Type Feature of the current session. If the session's URL type is anomalous, the number of abnormal URL type values in the current session will be displayed, along with the threshold value for this feature in the overall dashboard of abnormal access sessions. |
Session Duration Anomaly | An abnormal value has been detected for the Duration Feature of the current session. If the session duration is anomalous, the duration of abnormal values in the current session will be displayed, along with the threshold value for this feature in the overall dashboard of abnormal access sessions. |
Session Total Count Anomaly | An abnormal value has been detected for the Total Count Feature of the current session. If the session total count is anomalous, the number of abnormal access counts in the current session will be displayed, along with the threshold value for this feature in the overall dashboard of abnormal access sessions. |
Session Management
On the Session Management tab, the IP addresses accessed by the current session and the number of accesses per IP address will be displayed. You can also view the access logs of historical access IPs for the current Session ID using the current access information.
Note:
When session settings are configured and the traffic contains the content of the session settings, the session management options will be displayed.
