Help & Documentation>Tencent Cloud TI-ONE Platform>Operation Guide>Online Services>Guide on Releasing Online Services Using Custom Images

Guide on Releasing Online Services Using Custom Images

Last updated: 2026-03-03 16:56:21

On this page:

Overview

This document introduces 2 methods for deploying online services on Tencent Cloud TI-ONE Platform (TI-ONE) using custom images, along with the specification constraints on deployment. Through typical cases, it demonstrates how to create custom images and release online services.

Operation Steps

Creating a Custom Image Based on tiinfer

Description of Base Images with the tiinfer Framework

The platform provides a base image for inference with the built-in tiinfer framework.
ccr.ccs.tencentyun.com/tione-public-images/ti-cloud-gpu-base-tiinfer:py38-cu111-1.0.0
The base image is created based on CentOS and contains the following software packages:
Software Package
Version
CUDA
11.1.1
python
3.9.13
cos-python-sdk-v5
1.9.14
coscmd
1.8.6.24
numpy
1.23.1
msgpack
1.0.5
opencv-python
4.6.0.66
opencv-contrib-python
4.6.0.66
pandas
1.4.3
Pillow
9.4.0
tiinfer
0.1.1
mosec-tiinfer
0.0.6
Startup command of the base image: /usr/local/service/ti-cloud-infer/entrypoint.sh.
The content of entrypoint.sh is:
#!/bin/bash
source /etc/profile
source /root/.bashrc
export LD_LIBRARY_PATH=/usr/local/python3/lib/python3.8/site-packages/torch/lib:/usr/local/openmpi/lib:/usr/local/nccl/lib:/usr/local/cuda/lib64:/usr/local/python3/lib:/usr/local/python3/lib64:/usr/local/openmpi/lib:/usr/local/gcc/lib:/usr/local/gcc/lib64


MODEL_DIR=/data/model

echo "================== code path ${MODEL_DIR}=========="
cd ${MODEL_DIR}

if [ -f "requirements.txt" ]; then
  echo "============== install python requirements ======================"
  echo "python3 -m pip install -r requirements.txt"
  python3 -m pip install -r requirements.txt
  echo "============== install python requirements done ================="
fi

echo "====================== start serving ============================"
echo "python3 -m tiinfer"
export TI_MODEL_DIR=${MODEL_DIR}
python3 -m tiinfer --timeout 30000
The startup logic is:
1.1 Read the requirements.txt file in the directory specified by the environment variable ${MODEL_DIR}, and use pip to install the specified dependency Python package.
1.2 The tiinfer framework will read the files in the directory specified by the environment variable ${MODEL_DIR}. After loading the model, it will start an HTTP service and listen to the port defined by the environment variable ${REST_PORT}.
1.3 When the tiinfer framework is started, the model will be loaded from the model_service.py file.

Specifications for Custom Images

1. Add a reference to the base image in the Dockerfile file. Example:
FROM ccr.ccs.tencentyun.com/tione-public-images/ti-cloud-gpu-base-tiinfer:py38-cu111-1.0.0
2. The custom logic is implemented mainly by modifying the model_service.py and entrypoint.sh files.
3. When Cloud File Storage (CFS), Cloud Object Storage (COS), or Data Accelerator Goose FileSystem (GooseFS) is used as the model source, the platform places the model files (including subdirectories) from the source path in the /data/model directory of the service instance by default. Therefore, custom code and data must not be placed in the /data/model directory. Otherwise, they will be overwritten by the platform.

Creating an Image

This case introduces a simple adder implemented based on a base image with the tiinfer framework by modifying the model_service.py and entrypoint.sh files.
Note: In this case, the managed model repository feature provided by the platform is not used to manage the model. Instead, the model and inference code are directly packaged into the image. Therefore, it is necessary to avoid placing the model and code in the /data/model directory.

Writing Code

Three files are involved in total:
File
Feature
model_service.py
Writes an adder model as required by tiinfer.
entrypoint.sh
Starts scripts and allows installing more dependency packages.
Dockerfile
Copies the first two files to the image.
1. Content of model_service.py:
from typing import Dict
import tiinfer

class AdderModel(tiinfer.Model):
 def __init__(self, model_dir: str):
     super().__init__(model_dir)

 def load(self) -> bool:
     self.ready = True
     return self.ready

 def preprocess(self, request: Dict) -> Dict:
     return request

 def predict(self, request: Dict) -> Dict:
     return {'result': request['a'] + request['b']}

 def postprocess(self, result: Dict) -> Dict:
     return result
2. Content of entrypoint.sh:
#!/bin/bash
source /etc/profile
source /root/.bashrc
export LD_LIBRARY_PATH=/usr/local/python3/lib/python3.8/site-packages/torch/lib:/usr/local/openmpi/lib:/usr/local/nccl/lib:/usr/local/cuda/lib64:/usr/local/python3/lib:/usr/local/python3/lib64:/usr/local/openmpi/lib:/usr/local/gcc/lib:/usr/local/gcc/lib64

MODEL_DIR=/opt/model

echo "================== code path ${MODEL_DIR}=========="
cd ${MODEL_DIR}

if [ -f "requirements.txt" ]; then
  echo "============== install python requirements ======================"
  echo "python3 -m pip install -r requirements.txt"
  python3 -m pip install -r requirements.txt
  echo "============== install python requirements done ================="
fi

echo "====================== start serving ============================"
echo "python3 -m tiinfer"
export TI_MODEL_DIR=${MODEL_DIR}
python3 -m tiinfer --timeout 30000
Note: The MODEL_DIR=/opt/model line of the above code changes the default /data/model of the startup directory to /opt/model to avoid being overwritten by the platform.
3. Content of Dockerfile:
FROM ccr.ccs.tencentyun.com/tione-public-images/ti-cloud-gpu-base-tiinfer:py38-cu111-1.0.0

COPY model_service.py /opt/model/model_service.py
COPY entrypoint.sh ./entrypoint.sh
RUN chmod +x ./entrypoint.sh
Note: The above code copies model_service.py to the /opt/model directory instead of the default /data/model directory to avoid being overwritten by the platform.

Packaging an Image

1. Overall steps:
Configure the local Docker environment and activate Tencent Container Registry (TCR).
Create a namespace and a personal image repository.
Compile a custom inference image and push it to the personal image repository.
When the model service is started, choose not to use the model file in the instance container column, select the running environment to enter the list of personal image repositories, and select the custom image environment pushed in the previous step.
Configure parameters and start the service.
2. Detailed description:
Run the following command for packaging:
docker build . --tag ccr.ccs.tencentyun.com/YOUR_NAMESPACE/YOUR_IMAGENAME
After packaging is completed, check whether the service is running normally in the local environment as follows:
Run docker run -d --name myinfer ccr.ccs.tencentyun.com/YOUR_NAMESPACE/YOUT_IMAGENAME to run the service.
Run docker exec -it myinfer bash to enter the container.
Run curl http://127.0.0.1:8501/v1/models/m:predict -d '{"a": 1, "b": 2}' in the container to obtain a correct result: {"result": 3}.
Exit the container, return to the local environment, and upload the image by running the following command: docker push ccr.ccs.tencentyun.com/YOUR_NAMESPACE/YOUR_IMAGENAME.

Creating a Custom Image with Other Inference Frameworks

The platform supports deploying the online services of models using custom images with other inference frameworks.

Specifications for Custom Images

1. The service must receive requests via the HTTP protocol, and only supports POST.
2. When CFS, COS, or GooseFSx is used as the model source, the platform places the model files (including subdirectories) from the source path in the /data/model directory of the service instance by default. Therefore, custom code and data must not be placed in the /data/model directory. Otherwise, they will be overwritten by the platform.
3. The image has been verified locally and can provide services normally.

Uploading a Custom Image and Releasing an Inference Service

Uploading a Custom Image

1. Log in to TCR.
2. On the Image Repository page, click Create.
3. Upload the image.
Click Quick Commands in the image repository to view the operation commands and upload the image.
image.png



Releasing an Online Service

On the Online Service page of TI-ONE, click Create Service.

Method 1: Packaging a Model for Use in an Image

If you have packaged a model in an image, you can use the image to release a service directly.
The configuration of the service replica is as follows:



Method 2: Mounting a Model in CFS to a Container for Use (Operation Steps for GooseFSx Are Similar to Those for CFS)

If your image only serves as a running environment for a service, you can upload a model to CFS and then mount it to a container. The model will be mounted in the /data/model directory, and your service can load the model from this directory.
The configuration of the service replica is as follows:


中国站 - English