!pip install modelscope

from modelscope import snapshot_download
#qwen/Qwen2-7B-Instruct is the name of the model to be downloaded, and cache_dir is the address where the downloaded model will be saved. Here, /home/tione/notebook indicates saving the downloaded model in the root directory of the CFS mount directory.
model_dir = snapshot_download('qwen/Qwen2-7b-Instruct', cache_dir='/home/tione/notebook')

{"messages":[{"role":"user","content":"Hello"}],"max_tokens":128}

{
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "city": {
                            "type": "string",
                            "description": "The city to find the weather for, e.g. 'Beijing'"
                        }
                    },
                    "required": [
                        "city"
                    ]
                }
            }
        }
    ],
    "messages": [
        {"role": "user", "content": "what's the weather of Shanghai today?"}
    ],
    "max_tokens": 512
}

{
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://tione-public-cos-1308945662.cos.ap-shanghai.myqcloud.com/test/cat.jpg"
                    }
                }
            ]
        }
    ],
    "max_tokens": 512
}

#!/bin/bash
RESULTS_FOLDER="results"
HOST="https://ms-cxxxxxxx.ti.tencentcs.com/ms-czzfp5c9/"
MODEL="../Qwen2-7B-Instruct"

QPS="inf"
INPUT_LEN=128
OUTPUT_LEN=128
NUM_PROMPTS=20
CONCURRENCY=16

mkdir -p $RESULTS_FOLDER

client_command="python3 benchmark_serving.py \
--port 8501 \
--base-url $HOST \
--endpoint /v1/chat/completions \
--backend openai-chat \
--model $MODEL \
--dataset-name random \
--random-input-len $INPUT_LEN \
--random-output-len $OUTPUT_LEN \
--ignore-eos \
--request-rate $QPS \
--num-prompts $NUM_PROMPTS \
--save-result \
--result-dir $RESULTS_FOLDER \
--metadata input_len=$INPUT_LEN output_len=$OUTPUT_LEN qps=$QPS concurrency=$CONCURRENCY"

eval $client_command

python benchmark_serving.py \
        --port xxxx \
        --base-url http://xxxxxxxx/ms-rpbpr92c/ \
        --backend openai-chat \
        --model ../Qwen2-7B-Instruct \
        --dataset-name sharegpt \
        --endpoint /v1/chat/completions \
        --tokenizer ../Qwen2-7B-Instruct \
        --max-concurrency 10 \
        --save-result \
        --result-dir results \
        --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json

GPU memory required for model weights (G) ≈ Model weight parameter scale (B) x Model weight precision (byte)
GPU memory required for KV cache (G) = Model layer size (hidden_size) x Number of model layers (num_hidden_layers) x Number of model KV heads (num_key_value_heads)/Number of model attention heads (num_attention_heads) x Model context length (max_position_embeddings) x Model KV-cache precision (byte) x 2/1024/1024/1024
Required GPU memory > (GPU memory required for model weights + GPU memory required for KV cache)/GPU memory utilization (gpu_memory_utilization)
----
Minimum number of GPUs required = ceil (Required GPU memory/GPU memory per GPU card) 
With the requirement num_attention_heads % number of GPUs == 0