WeData Data Development Platform Data Science Overview-Operation Guide-Help & Documentation-Tencent Cloud

Product Design Philosophy
Based on the MLOps product design philosophy, the data science module was built into the WeData Data Development and Governance Platform (hereinafter referred to as WeData).
MLOps Philosophy and Value
MLOps (Machine Learning Operations) is a set of engineering methods that connects the model development team with business and Ops teams. It establishes a standardized, automated, and continuously improved process management system for the full lifecycle of machine learning models. This enables organizations to reliably, scalably, and efficiently produce machine learning models on an ongoing basis, empowering business operations. The core objective is to achieve cost reduction and efficiency enhancement in large-scale AI development by addressing the following issues:
The full lifecycle of models lacks unified management.
Assets such as code, data, algorithms, and models lack unified version control and traceability capabilities.
Enterprises lack corresponding norms during the ML production-to-application process.
The iteration cycle for model development and deployment is long.
According to Algorithmia 2020: 64% of enterprises spend over a month deploying a new model, and 18% of companies require more than 90 days to go live.
Model services are not sustainable.
The speed of model iteration and deployment cannot keep up with the rapid changes in business requirements.
From the moment of deployment, models are at risk of degradation (such as data drift and performance drift).
The level of automation is low.
Numerous manual processes exist, resulting in low efficiency and high labor costs.
A lack of comprehensive monitoring and alarm mechanisms prevents errors from being captured and corrected in a timely manner before damage occurs.
Cross-team collaboration is difficult.
The tools and workflows of different teams vary significantly.
The silo effects and communication gaps between business, Ops, and AI teams are insurmountable.
Potential risks are high.
Technical risks: unstable model performance and fragile infrastructure.
Compliance risks: violation of government regulations and company policies.
Our Insights and Advantages
Building data science capabilities on a data platform like WeData inherently provides robust Data Integration, data development, and data governance capabilities, naturally addressing the fragmentation issue between traditional data platforms and AI development platforms:
Data development and AI development are separated.
Big data and AI are two independent systems, making it difficult to implement an end-to-end process for sample cleaning, storage, analysis, training, and inference.
Storage and computing costs are high.
Data needs to flow back and forth between two systems.
The CPU and GPU computing power of big data and AI cannot be reused.
Our Core Philosophy
1. Always drive AI project development with business objectives as the guiding principle.
2. Adopt a data-centric approach to drive AI development.
3. Drive the entire lifecycle through a modular platform approach: for example, data exploration, feature engineering, model training, and online services.
4. Use automated processes to achieve continuous training, continuous integration, and continuous delivery.
Feature Overview
The WeData Data Science module establishes four core feature modules: experiment management, feature management, model management, and model service. It closely collaborates with surrounding associated products such as Studio, workflow, data quality, and engines, thereby building MLOps capabilities and achieving end-to-end lifecycle capabilities spanning "data-model-inference".
Core Modules
Module
Core Feature
Model Experiment
Enabling the MLflow service in Studio allows you to record each experiment's parameters, metrics, and results by calling MLflow functions during experiments, and view them in experiment management, thereby achieving experiment tracking and reproducibility.
It also provides AutoML capabilities, supporting no-code development.
Feature Management
Using the feature processing APIs provided by WeData in Studio, you can create, write, read, search, synchronize, and consume feature tables, as well as view and manage features in feature management, achieving unified feature management and consumption.
Model Management
Enabling the MLflow service in Studio allows you to register models by calling MLflow functions during experiments or perform visual model registration in experiment management. It also supports viewing key model information and its associations with experiments/runs and services.
Model Service
Supports creating API services from models in Model Management, performing service monitoring and other features, and viewing the association with models for easy information tracing.
Auxiliary Modules
Module
Core Feature
Studio
The primary workspace for AI development, where users can edit, debug, and run code in Studio, and invoke MLflow and feature engineering APIs to perform operations such as CRUD on feature tables, model training, and model registration.
Workflow
The primary workspace for automated processes, where users can debug code in Studio and then submit it to a workflow to set up periodic scheduling, enabling automated and periodic model production.
Data Quality
Users can initiate data quality tasks for model service inference tables, feature tables, and training data tables to view corresponding quality information such as field analysis, drift analysis, and model metrics.
Engine
Data Science integrates with DLC and EMR engines, serving as the data source, offline feature store, and training resource for AI development.
Prerequisites and Notes
Type
Description
Studio
You must activate Studio to use data science-related features.
Note
Currently, the WeData Studio feature is an allowlist feature. To use it, contact the WeData team to enable it.
The primary workspace for AI development, where users can edit, debug, and run code in Studio, and invoke MLflow and feature engineering APIs to perform operations such as CRUD on feature tables, model training, and model registration.
Engine Support
DLC Standard Engine
It can be used for operations such as model training, model experiment reporting, feature management, and model registration.
Note that a resource group can be used for AutoML experiments only if the "wedata-data-science" image is selected when the resource group is created on the DLC engine.
EMR on CVM and EMR on TKE (Note: The EG component must be included; otherwise, it cannot be used in WeData).
It can be used for operations such as model training, model experiment reporting, feature management, and model registration.
Note that it cannot be used for AutoML experiments.
EMR: Ray on TKE (Note: The EG component must be included; otherwise, it cannot be used in WeData).
It can be used for operations such as model training, model experiment reporting, and model registration.
Note: It cannot be used for AutoML experiments or feature processing.
MLflow Version
WeData's experiment management is compatible with MLflow version 2.17.2 by default. The relevant images are pre-installed. After connecting to the Studio runtime environment, you can run commands to check:
%pip list | grep mlflow
Offline Feature Store
The offline feature tables managed by WeData currently support only DLC Iceberg tables and EMR Hive tables. When registering a feature table, you must specify its primary key and timestamp key. Subsequent operations will use the specified primary key and timestamp key for feature indexing.
Table Operation Permissions
DLC:
If Catalog is enabled:
Grant the corresponding library table permissions to users in DLC at "DLC--System Management--User and Permission Management".
and grant permissions to users in WeData at "Data Assets--Catalog Directory--Corresponding Catalog--Permissions".
If Catalog is not enabled:
Grant the corresponding library table permissions to users in DLC at "DLC--System Management--User and Permission Management".
EMR:
Set the engine access account and account mapping in WeData at "Project Management > Compute and Storage Engine Settings" to determine the permissions for accessing library tables.
Model Operation Permissions
DLC:
If Catalog is enabled:
Grant permissions to users in WeData at "Data Assets > Catalog Directory > Corresponding Model Catalog > Permissions".
If Catalog is not enabled:
No authorization is required.
EMR:
No authorization is required.
Online Feature Store
WeData supports Redis as the online feature store. Prepare as follows:
Step 1: Go to Distributed Cache to create a Redis instance.
The region and network must be consistent with the scheduling resource group and the engine used (DLC or EMR).
Step 2: In WeData's "Project Management -- Data Source Management", add a Redis data source and test its connectivity to the scheduling resource group.
Step 3: After connecting to the remote kernel in WeData's "Offline Development -- Studio", verify whether Redis is connected, as follows:
﻿
import redis
﻿
realtime_feature_redis_pwd = "mypwd" # If there is no key, you can comment out this line.
REDIS_HOST = "myhost" # Example: "10.0.0.51"
REDIS_PORT = "myport" # Example: "6379"
﻿
try:
    r = redis.Redis(
        host=REDIS_HOST,
        port=REDIS_PORT,
        socket_connect_timeout=3,
        password=realtime_feature_redis_pwd # If there is no key, you can comment out this line.
    )
﻿
    result = r.ping()
﻿
    if result:
        print("✅ Redis is reachable and the PING command succeeded.")
    else:
        print("❌ Redis is not responding.")
﻿
except Exception as e:
    print("❌ Redis connection failed.")
    print("Error message:", e)
Feature Engineering Package
After connecting to the engine, run the following command to install the latest feature engineering package:
%pip install tencent-wedata-feature-engineering
If you use the DLC engine and select the "wedata-data-science" image when a resource group is built, the feature engineering package is pre-installed. You can run commands to check the version:
%pip list | grep tencent-wedata-feature-engineering
The latest version information can be viewed at: tencent-wedata-feature-engineering
Secret Management Description
Invoking the feature engineering package requires a TencentCloud API ID and key. You can go to Tencent Cloud CAM to create access keys.
If your organization requires that key information cannot be transmitted in plaintext, you can use Tencent Cloud SSM or Tencent Cloud KMS to manage keys.
Environment Variable Configuration
DLC:
If Catalog is enabled, no additional configuration is required.
If Catalog is not enabled, you must set environment variables; otherwise, experiments and model reporting may fail:
import os
os.environ["MLFLOW_REGISTRY_URI"]   = ""
EMR:
Set environment variables; otherwise, experiments and model reporting may fail:
import os
os.environ["MLFLOW_REGISTRY_URI"]   = ""

Module	Core Feature
Model Experiment	Enabling the MLflow service in Studio allows you to record each experiment's parameters, metrics, and results by calling MLflow functions during experiments, and view them in experiment management, thereby achieving experiment tracking and reproducibility. It also provides AutoML capabilities, supporting no-code development.
Feature Management	Using the feature processing APIs provided by WeData in Studio, you can create, write, read, search, synchronize, and consume feature tables, as well as view and manage features in feature management, achieving unified feature management and consumption.
Model Management	Enabling the MLflow service in Studio allows you to register models by calling MLflow functions during experiments or perform visual model registration in experiment management. It also supports viewing key model information and its associations with experiments/runs and services.
Model Service	Supports creating API services from models in Model Management, performing service monitoring and other features, and viewing the association with models for easy information tracing.

Module	Core Feature
Studio	The primary workspace for AI development, where users can edit, debug, and run code in Studio, and invoke MLflow and feature engineering APIs to perform operations such as CRUD on feature tables, model training, and model registration.
Workflow	The primary workspace for automated processes, where users can debug code in Studio and then submit it to a workflow to set up periodic scheduling, enabling automated and periodic model production.
Data Quality	Users can initiate data quality tasks for model service inference tables, feature tables, and training data tables to view corresponding quality information such as field analysis, drift analysis, and model metrics.
Engine	Data Science integrates with DLC and EMR engines, serving as the data source, offline feature store, and training resource for AI development.

Type	Description
Studio	You must activate Studio to use data science-related features. Note Currently, the WeData Studio feature is an allowlist feature. To use it, contact the WeData team to enable it. The primary workspace for AI development, where users can edit, debug, and run code in Studio, and invoke MLflow and feature engineering APIs to perform operations such as CRUD on feature tables, model training, and model registration.
Engine Support	DLC Standard Engine It can be used for operations such as model training, model experiment reporting, feature management, and model registration. Note that a resource group can be used for AutoML experiments only if the "wedata-data-science" image is selected when the resource group is created on the DLC engine. EMR on CVM and EMR on TKE (Note: The EG component must be included; otherwise, it cannot be used in WeData). It can be used for operations such as model training, model experiment reporting, feature management, and model registration. Note that it cannot be used for AutoML experiments. EMR: Ray on TKE (Note: The EG component must be included; otherwise, it cannot be used in WeData). It can be used for operations such as model training, model experiment reporting, and model registration. Note: It cannot be used for AutoML experiments or feature processing.
MLflow Version	WeData's experiment management is compatible with MLflow version 2.17.2 by default. The relevant images are pre-installed. After connecting to the Studio runtime environment, you can run commands to check: %pip list \| grep mlflow
Offline Feature Store	The offline feature tables managed by WeData currently support only DLC Iceberg tables and EMR Hive tables. When registering a feature table, you must specify its primary key and timestamp key. Subsequent operations will use the specified primary key and timestamp key for feature indexing.
Table Operation Permissions	DLC: If Catalog is enabled: Grant the corresponding library table permissions to users in DLC at "DLC--System Management--User and Permission Management". and grant permissions to users in WeData at "Data Assets--Catalog Directory--Corresponding Catalog--Permissions". If Catalog is not enabled: Grant the corresponding library table permissions to users in DLC at "DLC--System Management--User and Permission Management". EMR: Set the engine access account and account mapping in WeData at "Project Management > Compute and Storage Engine Settings" to determine the permissions for accessing library tables.
Model Operation Permissions	DLC: If Catalog is enabled: Grant permissions to users in WeData at "Data Assets > Catalog Directory > Corresponding Model Catalog > Permissions". If Catalog is not enabled: No authorization is required. EMR: No authorization is required.
Online Feature Store	WeData supports Redis as the online feature store. Prepare as follows: Step 1: Go to Distributed Cache to create a Redis instance. The region and network must be consistent with the scheduling resource group and the engine used (DLC or EMR). Step 2: In WeData's "Project Management -- Data Source Management", add a Redis data source and test its connectivity to the scheduling resource group. Step 3: After connecting to the remote kernel in WeData's "Offline Development -- Studio", verify whether Redis is connected, as follows: import redis realtime_feature_redis_pwd = "mypwd" # If there is no key, you can comment out this line. REDIS_HOST = "myhost" # Example: "10.0.0.51" REDIS_PORT = "myport" # Example: "6379" try: r = redis.Redis( host=REDIS_HOST, port=REDIS_PORT, socket_connect_timeout=3, password=realtime_feature_redis_pwd # If there is no key, you can comment out this line. ) result = r.ping() if result: print("✅ Redis is reachable and the PING command succeeded.") else: print("❌ Redis is not responding.") except Exception as e: print("❌ Redis connection failed.") print("Error message:", e)
Feature Engineering Package	After connecting to the engine, run the following command to install the latest feature engineering package: %pip install tencent-wedata-feature-engineering If you use the DLC engine and select the "wedata-data-science" image when a resource group is built, the feature engineering package is pre-installed. You can run commands to check the version: %pip list \| grep tencent-wedata-feature-engineering The latest version information can be viewed at: tencent-wedata-feature-engineering
Secret Management Description	Invoking the feature engineering package requires a TencentCloud API ID and key. You can go to Tencent Cloud CAM to create access keys. If your organization requires that key information cannot be transmitted in plaintext, you can use Tencent Cloud SSM or Tencent Cloud KMS to manage keys.
Environment Variable Configuration	DLC: If Catalog is enabled, no additional configuration is required. If Catalog is not enabled, you must set environment variables; otherwise, experiments and model reporting may fail: import os os.environ["MLFLOW_REGISTRY_URI"] = "" EMR: Set environment variables; otherwise, experiments and model reporting may fail: import os os.environ["MLFLOW_REGISTRY_URI"] = ""

Data Science Overview

On this page:

Product Design Philosophy

MLOps Philosophy and Value

Our Insights and Advantages

Our Core Philosophy

Feature Overview

Core Modules

Auxiliary Modules

Prerequisites and Notes