我在我的谷歌项目中有一个BigQuery数据集-珠穆朗玛峰:evr_ dataset,我想将其表数据复制到我的另一个BigQuery数据集,该数据集位于另一个项目-alps:alp_ dataset中。
我尝试使用DTS数据传输服务来计划每天的摄取作业,但是我没有看到从另一个项目中选择目标数据集的任何选项?有没有人能告诉我如何开启项目间的DTS?
发布于 2021-09-28 07:09:55
您可以使用下面的python函数创建大型查询数据传输客户端,并通过指定源和目标项目id将数据集从一个项目复制到另一个项目。您还可以计划数据传输。在下面的方法中,它被设置为24小时(每天)。
def copy_dataset(override_values={}):
# [START bigquerydatatransfer_copy_dataset]
from google.cloud import bigquery_datatransfer
transfer_client = bigquery_datatransfer.DataTransferServiceClient()
destination_project_id = "my-destination-project"
destination_dataset_id = "my_destination_dataset"
source_project_id = "my-source-project"
source_dataset_id = "my_source_dataset"
# [END bigquerydatatransfer_copy_dataset]
# To facilitate testing, we replace values with alternatives
# provided by the testing harness.
destination_project_id = override_values.get(
"destination_project_id", destination_project_id
)
destination_dataset_id = override_values.get(
"destination_dataset_id", destination_dataset_id
)
source_project_id = override_values.get("source_project_id", source_project_id)
source_dataset_id = override_values.get("source_dataset_id", source_dataset_id)
# [START bigquerydatatransfer_copy_dataset]
transfer_config = bigquery_datatransfer.TransferConfig(
destination_dataset_id=destination_dataset_id,
display_name="Your Dataset Copy Name",
data_source_id="cross_region_copy",
params={
"source_project_id": source_project_id,
"source_dataset_id": source_dataset_id,
},
schedule="every 24 hours",
)
transfer_config = transfer_client.create_transfer_config(
parent=transfer_client.common_project_path(destination_project_id),
transfer_config=transfer_config,
)
print(f"Created transfer config: {transfer_config.name}")
# [END bigquerydatatransfer_copy_dataset]
return transfer_config
https://stackoverflow.com/questions/69356079
复制相似问题