开发自己的TCGA数据库下载器就是怎么简单

生信技能树

发布于 2019-05-08 16:49:53

9830

发布于 2019-05-08 16:49:53

如果你不懂代码，不懂网站规则，那么最简单的就是直接使用UCSC xena 浏览器啦！！！网站；https://xenabrowser.net/datapages/ 理论上也可以完成大部分数据探索的，甚至还有一些sci文章发表就是完完全全使用这个网页工具。

看到jimmy总结的如此有规律的下载地址链接，我尝试用python写几句脚本下载一下tcga数据。

1.尝试用爬虫获得所有疾病条目

尝试写爬虫发现网页需要javascript，暂时没有搞定，于是偷个懒把内容从https://xenabrowser.net/datapages/?host=https://tcga.xenahubs.net复制下来保存为TCGA_names.txt。一共是39 Cohorts, 903 Datasets。

TCGA Acute Myeloid Leukemia (LAML)
TCGA Adrenocortical Cancer (ACC)
......

2. 把这些文本解析一下，变成下载地址

https://gdc.xenahubs.net/download/TCGA-LAML/Xena_Matrices/TCGA-LAML.htseq_counts.tsv.gz
https://gdc.xenahubs.net/download/TCGA-ACC/Xena_Matrices/TCGA-ACC.htseq_counts.tsv.gz

绝大部分文件的下载地址中，网址的规律是:

https://gdc.xenahubs.net/download/TCGA-疾病名称/Xena_Matrices/TCGA-疾病名称-下载文件.tsv.gz

基本思路就是定义了两个函数完成这个事。第一个函数 get_name() 是用来从文本中获取疾病名的缩写，也就是这三十几种，因为下载地址就是用的缩写，利用 split() 函数获得。

import os

def get_name(file_in):
    name_list= []
    for line in file_in:
        if 'TCGA' in line:
            name_list.append(line.strip().split('(')\ [1].split(')')[0])
        return name_list

第二个函数是把下载地址补充完整，实现下载过程。首先看你需要的数据是哪几类，把网址里的文件名放在一个列表里。这里数据文件名和含义的对应关系是：

gene expression RNAseq(HTSeq)--htseq_counts
survival--survival
phenotype--GDC_phenotype
miRNA Expression Quantification--mirna
copy number--masked_cnv
MuSE Variant Aggregation and Masking--muse_snv
MuTect2 Variant Aggregation and Masking--mutect2_snv

然后是利用格式化字符串，实现网址的自动补全。最后，利用os.system()函数，调用wget命令完成，-c参数以实现断点续传。

def download_files(name_list):
    for name in name_list:
        file_list = ['htseq_counts', 'survival','GDC_phenotype', 'mirna', 'masked_cnv'，'muse_snv', 'mutect2_snv'] 
        for i in range(len(file_list)):
            cmd = 'wget -c https://gdc.xenahubs.net/download/TCGA-%s/Xena_Matrices/TCGA-%s.%s.tsv.gz' % (name, name, file_list[i])
            os.system(cmd)

最后，调用两个函数，完成下载过程。

file_in = open('TCGA_names.txt')
name_list = get_name(file_in)
download_files(name_list)

3.做个简陋的shiny控件

在循循善诱的群主启发下第一次学着用shiny，有点赶鸭子上架，哈哈。就参考着shiny的官方教程，做了个能有多简单就多简单的。

UI代码，就是按照官方示例框架，加上了内容：

library(shiny)

# Define UI for miles per gallon application
shinyUI(list(

  # Application title
  #titlePanel("TCGA下载地址获取工具"),

  fluidPage(
    titlePanel("TCGA下载地址获取工具"),
    sidebarPanel(
    selectInput("variable", "请选择一种癌症:",
                list("Acute Myeloid Leukemia" = "LAML", 
                     "Adrenocortical Cancer" = "ACC", 
                     "Bile Duct Cancer" = "CHOL",
                     "Bladder Cancer" = "BLCA",
                     "Breast Cancer" = "BRCA",
                     "Cervical Cancer" = "CESC",
                     "Colon and Rectal Cancer" = "COADREAD",
                     "Colon Cancer" = "COAD",
                     "Endometrioid Cancer" = "UCEC",
                     "Esophageal Cancer" = "ESCA",
                     "Formalin Fixed Paraffin-Embedded Pilot Phase II" = "FPPP",
                     "Glioblastoma" = "GBM",
                     "Head and Neck Cancer" = "HNSC",
                     "Kidney Chromophobe" = "KICH",
                     "Kidney Clear Cell Carcinoma" = "KIRC",
                     "Kidney Papillary Cell Carcinoma" = "KIRP", 
                     "Large B-cell Lymphoma" = "DLBC", 
                     "Liver Cancer" = "LIHC",
                     "Lower Grade Glioma" = "LGG",
                     "lower grade glioma and glioblastoma" = "GBMLGG",
                     "Lung Adenocarcinoma" = "LUAD",
                     "Lung Cancer" = "LUNG",
                     "Lung Squamous Cell Carcinoma" = "LUSC",
                     "Melanoma" = "SKCM",
                     "Mesothelioma" = "MESO",
                     "Ocular melanomas" = "UVM",
                     "Ovarian Cancer" = "OV",
                     "Pan-Cancer" = "PANCAN",
                     "PANCAN12" = "PANCAN12",
                     "Pancreatic Cancer" = "PAAD",
                     "Pheochromocytoma & Paraganglioma" = "PCPG", 
                     "Prostate Cancer" = "PRAD", 
                     "Rectal Cancer" = "READ",
                     "Sarcoma" = "SARC",
                     "Stomach Cancer" = "STAD",
                     "Testicular Cancer" = "TGCT",
                     "Thymoma" = "THYM",
                     "Thyroid Cancer" = "THCA",
                     "Uterine Carcinosarcoma" = "UCS"
                     )),
    selectInput("var", "选择一种数据:",
                list("gene expression RNAseq(HTSeq)" = "htseq_counts", 
                     "survival" = "survival", 
                     "phenotype" = "GDC_phenotype",
                     "miRNA Expression Quantification" = "mirna",
                     "copy number" = "masked_cnv",
                     "MuSE Variant Aggregation and Masking" = "muse_snv",
                     "MuTect2 Variant Aggregation and Masking" = "mutect2_snv"))

    #checkboxInput("outliers", "Show outliers", FALSE)
  ),

    mainPanel(h4("你选择的数据下载地址为:"),
              htmlOutput("mySite"))


)))

Server端代码就是同样把地址拼接了一下，同样是简单修改自模板，开始空格问题没搞定，后来百度了个解决方案，但是提示语却也不能有空格了，只好用下划线代替：

library(shiny)

# Define server logic required to plot various variables against mpg
shinyServer(
  function(input, output, session)
    {
  output$mySite <- renderUI( {
    url <- paste('\n', (gsub("([N ])", "", paste("https://gdc.xenahubs.net/download/TCGA-", input$variable,"/Xena_Matrices/TCGA-",input$variable,".",input$var,".tsv.gz"))))
    tags$a(href = url, url)

    })
  }
)

最后来看看简陋的效果：