This document takes the Qwen2-7B-Instruct model as an example to demonstrate how to import custom LLMs into Tencent Cloud TI-ONE Platform (TI-ONE) and deploy the LLM inference services using the built-in inference images.
Prerequisites
Applying for CFS or GooseFSx
Before deploying a custom LLM, you need to store LLM files in Cloud File Storage (CFS) or Data Accelerator Goose FileSystem extreme (GooseFSx) in advance. Therefore, you need to apply for CFS or GooseFSx in advance.
Operation Steps
1. Uploading LLM Files to CFS or GooseFSx
Log in to TI-ONE, choose Training Workshop > Dev Machines, and click Create. The instructions for specifying each field are as follows:
Image: Select any built-in image.
Billing Mode: Select either pay-as-you-go or yearly/monthly subscription billing mode. For billing rules supported by TI-ONE, see Billing Overview.
Storage Path Settings: Select CFS or GooseFSx. The path is the root directory / by default and is used to specify the storage location for your custom LLM.
Other Settings: Not required by default.
Note:
This dev machine is only used for uploading or downloading LLM files. Therefore, select CPU Computing Power for CVM Instance Specifications.
After the creation is completed and the dev machine is in the Running status, choose Open > Python3(ipykernel) to download the LLM files you need via scripts.
You can search for the LLMs you need in ModelScope or Hugging Face, download the models through Python scripts in ModelScope, and save them to CFS. This document takes the Qwen2-7B-Instruct model as an example, and the code for download is as follows:
!pip install modelscope
from modelscope import snapshot_download
#qwen/Qwen2-7B-Instruct is the name of the model to be downloaded, cache_dir is the address where the downloaded model will be saved, and ./ indicates saving the downloaded model in the root directory of CFS.
Specify the address cache_dir (for example, path/to/local/dir) for downloading the model. Then, specify the model address as /path/to/local/dir/qwen/Qwen2-7B-Instruct in the CFS instance of Online Services.
Copy the above download script, replace relevant content with the model to be downloaded, paste the modified script into the new .ipynb file, and click Run to start downloading the model.
Additionally, you can also download or make slight adjustments to the model locally. Then, save the model file to CFS through the upload channel of the dev machine. The upload API is shown below:
2. Creating an Online Service
Choose Model Services > Online Services on TI-ONE, and click New Service to start the inference service. The following is the guidance for service instance configuration.
Model Source: Choose Cloud Storage > CFS.
Model: Select the model file path stored in the CFS instance, which is /qwen/Qwen2-7B-Instruct.
Image: Select built-in/LLM/Angel-vLLM.
CVM Instance Specifications: Select resources based on the actual model size or your available resources. The Cloud Virtual Machine (CVM) instance resources required for LLM inference depend on the model parameter scale. It is recommended to configure the resources required for inference services according to the following rules.
Model Parameter Scale
GPU Card Type and Quantity
6 ~ 8B
L20 * 1 / A10 * 1 / A100 * 1 / V100 * 1
12 ~ 14B
L20 * 1 / A10 * 2 / A100 * 1 / V100 * 2
65 ~ 72B
L20 * 8 / A100 * 8
Advanced Settings > Environment Variables: You need to specify the model name MODEL_ID (the ID of an open-source model in ModelScope or Hugging Face) and the chat template name CONV_TEMPLATE (if MODEL_ID is the same as that of the open-source model, the CONV_TEMPLATE parameter can be omitted). Common chat template names are listed in the table below. This document uses the qwen-chat series, so set the chat template name to qwen-7b-chat.
Chat Template Name CONV_TEMPLATE
Supported Model Series MODEL_ID
generate
Non-chat models (direct generation, without chat template)
llama-3
llama-3-8b-instruct and llama-3-70b-instruct models
llama-2
llama-2-chat series models
qwen-7b-chat
qwen-chat series models (chatml format)
baichuan2-chat
baichuan2-chat series models
baichuan-chat
baichuan-13b-chat model
chatglm3
chatglm3-6b model
chatglm2
chatglm2-6b model
The configuration for creating an instance in the Online Services module is as follows:
Note:
If you have high requirements for inference speed, we recommend that you enable quantization acceleration by setting the environment variable QUANTIZATION . Valid values: "none", "ifq", "smoothquant", and "auto".
none: indicates that quantization acceleration is disabled.
ifq: indicates that online Int8 Weight-Only quantization is enabled, which can accelerate inference with basically no loss in performance and reduce the GPU memory occupied by model weights.
smoothquant: indicates that LayerwiseSearchSMQ quantization is enabled, which can further accelerate inference with slightly degraded performance (requiring pre-prepared quantized model files, currently only supported for specific models).
auto: indicates that the quantization mode is automatically determined:
If the GPU of an instance type does not support quantization, quantization is automatically disabled.
If the model directory contains the smoothq_model-8bit-auto.safetensors file, LayerwiseSearchSMQ quantization acceleration is automatically enabled.
In other cases, online Int8 Weight-Only quantization acceleration (ifq) is enabled by default.
If the log reports a CUDA out of memory error after the service is enabled, the default value, 32 KB, of the max-model-len parameter of the model is too large (the maximum number of context tokens supported by the inference service, which is the context length automatically read from the model configuration information by default). If the model is loaded with a large default context length, it may cause insufficient GPU memory. You can set a smaller value (such as 16 KB or 8 KB) via the MAX_MODEL_LEN environment variable or enable quantization acceleration to reduce the GPU memory occupied by model weights.
3. Experiencing Chatting with LLMs on the Frontend Online
Go to the details page of the created online service and chat with the deployed LLM by clicking the Online Demo tab.
4. Calling an API Service
You can access a service by clicking the Service Call tab and choosing API Information > Call Method (Online Testing). The API calling address is ${SERVER_URL}/v1/chat/completions, and the format of the request body is as follows:
{"messages":[{"role":"user","content":"Who are you"}]}
The content field indicates the specific message content.
The public network access address can be obtained from the Service Call tab of the online service instance. An API call example is as follows:
{"id":"chatcmpl-4aeRgYwnaYe4RzmmcyKtYs","object":"chat.completion","created":1698291242,"model":"baichuan-13b-chat","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! How can I assist you today?"},"finish_reason":"stop"}],"usage":{"prompt_tokens":4,"total_tokens":16,"completion_tokens":12}}
Alternatively, you can also use the service via the requests library that is commonly used in Python. Below is a demo example of a chat interaction between command lines and the Qwen2-7B-Instruct LLM inference service:
import argparse
import requests
import json
defchat(messages):
data ={
"messages": messages,
"temperature": args.temperature,
"max_tokens": args.max_tokens,
"top_p": args.top_p,
"stream":True,# Enable streaming output.
}
header ={
"Content-Type":"application/json",
}
if args.token:
header["Authorization"]=f"Bearer {args.token}"
response = requests.post(f"{args.server}/v1/chat/completions", json=data, headers=header, stream=True)# Set the stream=True parameter to obtain real-time data streams.
if response.status_code !=200:
print(response.json())
exit()
result =""
print("Assistant: ", end ="", flush =True)
for part in response.iter_lines():
if part:
if"content"in part.decode("utf-8"):
content = json.loads(part.decode("utf-8")[5:])["choices"][0]["delta"]["content"]# Filter the string to remove the data: prefix, convert the string into JSON format, and then extract the text.
