Overview
Data Source Management allows you to register a specified directory from a storage instance (Cloud File Storage (CFS)/GooseFSx) as a data source, enabling quick mounting. Additionally, it supports implementing permission control for storage directories through Cloud Access Management (CAM) and tags.
Data Source
A data source refers to a specific directory of a storage instance. Specifically:
Storage instance: refers to a storage instance of a cloud product, such as CFS and GooseFSx on Tencent Cloud.
Directory: represents the specific path of data storage. Either the root directory or any subdirectory can be registered as a data source.
Data Source Management
It provides the capability to create and manage data sources, with details as follows:
Create data source: Supports registering storage instances purchased on Tencent Cloud to the Tencent Cloud TI-ONE Platform and creating specified directories as data sources.
Quickly mount data source: In Task-based Modeling, Dev Machines, Online Services, and other functional modules, supports one-click selection of specified directories during mounting, eliminating the need for repeated input.
Directory permission control: Supports using CAM and tags to implement permission control for storage directories. When CFS/GooseFSx is mounted within TI-ONE, you can only access data sources with specified tags.
Prerequisites
Before you use the Data Source Management feature, go to the CAM console to authorize the following CAM policy to the specified user/user group. The preset policies are as follows:
Preset policy for read/write permissions: QcloudTIONEResouceGroupFullAccessContainMultiservice
Preset policy for read-only permissions: QcloudTIONEReadOnlyAccessContainMultiservice
Creating a Data Source
1. Choose Platform Management > Data Source Management in the left sidebar to go to the Data Source Management page, and click New Data Source.
2. In the New Data Source pop-up window, fill in the relevant information, and click Confirm.
Configuration parameters and their descriptions are listed in the table below.
Field Name | Description |
Data Source Name | 1. Required. The length must not exceed 60 characters. Only Chinese, English, digits, underscores (_), and hyphens (-) are allowed. It must start with a Chinese character, English letter, or digit. 2. Duplicate name verification: The Data Source Name cannot be duplicated (across all storage instances). Additionally, the combination of storage directory + read/write permissions serves as a unique key, meaning that the storage directory and the read/write permissions cannot be repeated at the same time. |
Storage Type | Required. Options include GooseFSx and CFS. CFS is selected by default. |
Region | Displays the current region of the TI-ONE console, which cannot be modified. |
Storage Instance | 1. Required. The options are sourced from the information of purchased storage instances under the corresponding storage type. (Remarks: The text outside the parentheses indicates the name of the storage instance, and the text inside indicates the storage instance ID.) 2. Note: The instance here refers to the storage instance already purchased on Tencent Cloud. TI-ONE only registers and binds it, rather than creating a separate instance. |
Storage Directory | Required. Supports custom file paths. The path must start with a forward slash (/) and can only contain Chinese, English, digits, underscores (_), hyphens (-), and periods (.). It cannot contain "..". (Remarks: Only one directory can be configured per data source.) |
Read/Write Permissions | Required. Options include: Read-only, Read-write. Indicates the read/write permissions when modules mount the data source, including read-only mounting and read/write mounting. |
Tag | 1. Supports adding tags to data sources. Tag options are sourced from the tag console. Multiple tags can be added to a data source. 2. After tags are applied, you can control the visibility permissions of data sources by using tags in conjunction with CAM policies to achieve data isolation. |
Managing Data Sources
Adding a Data Source
The configuration parameters are consistent with those for creating a data source. You can add a data source for a specified storage instance. The type and name/ID of the selected storage instance cannot be modified.
Mount Control
Enabling mount control restricts users to accessing specified instances and storage paths only by mounting data sources; disabling it allows users to access any directory via the original method of specifying storage instance + source path.
The effective scope of mount control is at the level of the storage instance, which means that all data sources configured under that instance will be restricted.
Note: This feature retains the original mounting method while providing some permission management capabilities. It is recommended to enable it in scenarios with high requirements for permission control.
Tags & Permissions
Note:
For a tutorial on best practices for permission configuration, see Implementing Directory-Level Permission Control for Storage Using the Data Source Feature.
You can configure permissions for data sources based on CAM and tags. By assigning specific tags to data sources and configuring corresponding CAM policies, you can achieve data isolation between different Tencent Cloud sub-accounts, thereby implementing permission control for storage directories.
Editing
The configuration parameters are consistent with those for adding a new data source. During editing, existing information will be pre-populated. Modifications are allowed for the Data Source Name, Storage Directory, and Read/Write Permissions fields. The type and name/ID of the storage instance cannot be modified.
Deletion
Deleting a storage instance: This means to unbind the storage instance from TI-ONE Data Source Management, not to actually delete the storage instance or its data. This action will clear all data sources configured under the storage instance.
Deleting a data source: This means to delete a specific data source directory under the storage instance.
Using Data Sources
After data source configuration is completed, when storage paths are mounted in modules such as Data Set, Task-based Modeling, Dev Machines, Online Services, and Model Evaluation, the usage method is as follows:
1. When CFS/GooseFSx instances are mounted in the above modules using the original method, it is necessary to restrict whether users can select the corresponding instances based on whether the mount control feature is enabled for the storage instance.
If a restriction has been applied to an instance, the instance option will be disabled in the dropdown list, and the following message will be displayed. If no restriction is applied, the instance can be selected normally.
2. When the mount type is set to Data Source, one-click selection of specified directories is supported, eliminating the need for repeated input.
When you hover over a dropdown option, the storage type, storage instance name/ID, storage directory, and read/write permissions of the corresponding data will be displayed.
After an option is selected, the storage directory and read/write permissions of the selected data will be displayed below the text box.
3. Similarly, when data sources are mounted in business modules such as Task-based Modeling and Dev Machines, the dropdown list will only display data sources accessible to the corresponding team.
Others
1. After the launch of the Data Source Management feature, the original solution for achieving directory isolation between sub-accounts, which was solely based on the tag system, will no longer be maintained or effective.
2. Since CAM allows for complex custom policies, conflicts at the directory level might occur in extreme scenarios, as shown in the following example:
Extreme scenario: The parent directory is allowed, but the child directory is denied. Assume the directory hierarchy of the storage instance is "/", "/a", "/b", with the storage directory for Data Source A configured as "/" and Data Source B as "/a", both authorized to user Tom. Suppose a separate CAM policy is configured outside the platform to restrict Tom from accessing the storage directory "/b" of Data Source C.
Platform processing logic: Both parent and child directories are allowed. In this case, Tom can mount the directory "/b" within the platform via either the data source or the method of storage instance + source path, and can access it normally without triggering the denial of the child directory. (This is because Tom has access permission to the parent directory "/".)