使用 DeepSeek-R1 蒸馏其他模型的全流程实践

import re
from typing import Union, Dict

class CodeChecker:
    def __init__(self):
        # 定义常见编程语言名称(可以扩展)
        self.languages = ["c++", "python", "javascript", "java", "c#", "typescript", "ruby", "php", "program", "tensorflow", "API", "json", "function", "algorithm", "script", "linux", "Kubernete"]

    def is_code_like(self, text: str) -> bool:
        # 将文本转为小写，以便不区分大小写
        lower_text = text.lower()
        
        # 检查文本中是否包含任何编程语言名称
        for lang in self.languages:
            if lang in lower_text:
                return True
        return False

# 注册过滤函数
@filter_func.FilterFunctionManager.register(zh_name="包含常见编程名称")
def check_code(text: str, config: Dict = None) -> Union[bool, str]:
    checker = CodeChecker()
    if checker.is_code_like(text):
        return True, "包含常见编程名称"
    return False, ""


# 注册过滤函数
@filter_func.FilterFunctionManager.register(zh_name="不包含5个以上数字")
def filter_three_or_more_digits(text: str, config: Dict = None) -> Union[bool, str]:
    # 统计文本中数字的个数
    num_count = len(re.findall(r'\\d', text))
    
    # 如果数字个数小于等于5，过滤掉
    if num_count <= 5:
        return True, "不包含5个以上数字"
    return False, ""

{"system": "您是腾讯云TI平台智能问答机器人，能理解用户问题并流利解答。", "question": "Anna goes trick-or-treating in a subdivision where she gets 14 pieces of candy per house. However, for every fifth house, she gets 2 mini chocolate bars in addition to the regular candy. Her brother Billy goes trick-or-tricking in a neighboring subdivision where he gets 11 pieces of candy per house. If Anna visits 75 houses and Billy visits 90 houses, how many more pieces of candy does Anna get?", "answer": "<Thought>\\nAlright, let's tackle this problem about Anna and Billy trick-or-treating. So, Anna is going to a subdivision where she gets 14 pieces of candy per house. But wait, there's a twist—every fifth house she visits, she gets 2 extra mini chocolate bars in addition to the regular candy. That's interesting. Her brother Billy, on the other hand, is going to a neighboring subdivision where he consistently gets 11 pieces of candy per house. The question is asking how many more pieces of candy Anna gets compared to Billy after visiting their respective number of houses: Anna visits 75 houses, and Billy visits 90 houses.\\n\\nFirst, let's break down Anna's candy collection. Normally, she gets 14 pieces per house, so if she visits 75 houses, the straightforward calculation would be 14 pieces multiplied by 75 houses. Let me do that: 14 times 75 equals 1050 pieces of candy. So, without considering the extra mini chocolate bars, Anna would have 1050 pieces.\\n\\nBut wait, there's more to the story. Every fifth house, she gets 2 extra mini chocolate bars. How many times does she hit every fifth house in her 75-house adventure? Well, 75 divided by 5 equals 15. That means she gets those extra 2 pieces of candy 15 times. So, 15 occasions multiplied by 2 extra pieces equal 30 extra pieces of candy. Adding that to her initial total, 1050 plus 30 equals 1080 pieces of candy. Anna ends up with 1080 pieces.\\n\\nNow, let's look at Billy. He gets 11 pieces of candy per house, and he visits 90 houses. So, 11 multiplied by 90 houses equals 990 pieces of candy for Billy.\\n\\nThe question is asking how many more pieces of candy Anna gets compared to Billy. So, I need to subtract Billy's total from Anna's total. That would be 1080 pieces (Anna) minus 990 pieces (Billy), which equals 90 pieces. Therefore, Anna gets 90 more pieces of candy than Billy.\\n\\nBut hold on a second—let me double-check to make sure I didn't make any mistakes. For Anna: 14 pieces per house times 75 houses is indeed 1050, plus 30 extra pieces for the 15 houses where she gets bonus candy equals 1080. For Billy: 11 pieces per house times 90 houses is definitely 990. Subtracting 990 from 1080 does give us 90. Yes, that seems correct.\\n\\nIs there another way to approach this? Maybe by calculating the average per house including the extras for Anna. Let's see: Anna gets an extra 2 pieces every 5 houses, so on average, that's an extra 0.4 pieces per house (since 2 divided by 5 is 0.4). So, her average per house would be 14 plus 0.4, which is 14.4 pieces per house. Multiplying that by 75 houses: 14.4 times 75 equals 1080. Yep, that matches the previous calculation.\\n\\nAlternatively, could there be a way to visualize this? If I imagine walking down a street with 75 houses, every fifth house has an extra candy incentive. Counting those extra pieces could be another way to confirm the total. But I've already done the math, and it seems consistent.\\n\\nAnother angle: What if I considered the total number of extra pieces Anna gets compared to Billy? Since Anna gets 2 extra pieces every 5 houses, and she visits 75 houses, that's 15 times 2 extra pieces, which is 30 extra pieces. Billy doesn't get any extra pieces, so all of Anna's advantage comes from that 30-piece difference, plus the difference in their per-house candy rates.\\n\\nComparing their per-house rates: Anna gets 14 per house, Billy gets 11. That's a difference of 3 pieces per house. Over 75 houses, that's 225 extra pieces from the per-house rate alone. Wait a minute, that doesn't match my earlier total difference. Did I make a miscalculation here?\\n\\nAh, I see the mistake. Billy visits 90 houses, not 75. So, for a fair comparison, Anna's 75 houses at 14 per house equals 1050, and with the extra 30 pieces, that's 1080. Billy's 90 houses at 11 per house is 990. The difference is still 90 pieces. So, my initial calculation stands.\\n\\nIn conclusion, after visiting their respective numbers of houses, Anna ends up with 90 more pieces of candy than Billy.\\n\\n</Thought>\\n<Output>\\nAnna gets 90 more pieces of candy than Billy.\\n</Output>"}


cd /home/tione/notebook/<nb-id>/single_round_qa_pipeline/
wget https://tione-public-cos-1308945662.cos.ap-shanghai.myqcloud.com/tools/deepseek_tools/deepseek_tokenizer.tgz
tar xf deepseek_tokenizer.tgz

from tools import data_tokenizer
from typing import Union, Dict
tokenizer = data_tokenizer.Tokenizer("./deepseek_tokenizer")
length_threshold = 4096

# 注册过滤函数
@filter_func.FilterFunctionManager.register(zh_name="token长度过滤")
def check_token_len(text: str, config: Dict = None) -> Union[bool, str]:
    text_len = tokenizer.tokenize_len(text)
    if text_len > length_threshold :
        return True, "长度短于{}".format(length_threshold)
    return False, ""

# 注册过滤函数
@filter_func.FilterFunctionManager.register(zh_name="检查思维链是否完整")
def check_think_tag(text: str, config: Dict = None) -> Union[bool, str]:
    if "<think>" not in text or "</think>" not in text:
        return True, "思维链不完整"
    return False, ""

{
  "system": "xxx",
  "conversation": [
    {
      "prompt": "xxx",
      "response": "xxx"
    }
  ]
}