blocks|key|2115494|text|Rsync不是不能在storage+rest+API中通过单个请求执行的操作，并且gsutil在云函数上不可用，因此无法通过python脚本实现两个buckets的rsync。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2115495|您可以创建一个函数，使用执行存储桶之间同步的startup+script启动preemptible+VM，并在完成同步操作后关闭实例。|2115496|通过使用VM而不是无服务器服务，您可以避免长rsync进程可能产生的任何超时。|2115497|一个可抢占的虚拟机在停止之前最多可以运行24小时，您只会在实例启动时进行计费(磁盘存储将独立于状态进行计费)|2115498|如果虚拟机在一分钟前关机，您将不会按使用量收费。|2115499|对于这种方法，首先需要在存储桶中创建bash脚本，这将由可抢占的VM在启动时执行，例如：|2115500|#!+/bin/bash
gstuil+rsync+-r+gs://mybucket1+gs://mybucket2+

sudo+init+0+#this+is+similar+to+poweroff,+halt+or+shutdown+-h+now|code-block|syntax|javascript|2115501|在此之后，您需要使用启动脚本创建一个可抢占的VM，我建议使用f1-micro实例，因为存储桶之间的rsync命令不需要太多资源。|2115502|1.-转到VM+Instances+page。|2115503|2.-单击创建实例。|2115504|3.-在创建新实例页面上，填写实例的属性。|2115505|4.-单击管理、安全、磁盘、网络、独占。|2115506|5.在身份和API访问部分，选择有权读取云存储中的启动脚本文件和要同步的存储桶的服务帐户|2115507|2115508|精选允许完全访问所有云API。|ordered-list-item|2115509|2115510|7.-在可用性策略下，将抢占选项设置为On。此设置禁用实例的自动重新启动，并将主机维护操作设置为终止。|2115511|8.-在元数据部分，提供startup-script-url作为元数据密钥。|2115512|9.-在值框中，以gs://存储桶/文件或https://storage.googleapis.com/BUCKET/FILE格式提供启动脚本文件的URL。|2115513|10.单击Create创建实例。|2115514|使用此配置，每次启动实例时，脚本也将被执行。|2115515|这是启动虚拟机的python函数(如果这是可抢占的，则独立)|2115516|def+power(request):
++++import+logging
++++#+this+libraries+are+mandatory+to+reach+compute+engine+api
++++from+googleapiclient+import+discovery
++++from+oauth2client.client+import+GoogleCredentials

++++#+the+function+will+take+the+service+account+of+your+function
++++credentials+=+GoogleCredentials.get_application_default()

++++#+this+line+is+to+specify+the+api+that+we+gonna+use,+in+this+case+compute+engine++++
++++service+=+discovery.build('compute',+'v1',+credentials=credentials,+cache_discovery=False)

++++#+set+correct+log+level+(to+avoid+noise+in+the+logs)
++++logging.getLogger('googleapiclient.discovery_cache').setLevel(logging.ERROR)

++++#+Project+ID+for+this+request.
++++project+=+"yourprojectID"++#+Update+placeholder+value.
++++zone+=+"us-central1-a"++#+update+this+to+the+zone+of+your+vm
++++instance+=+"myvm"++#+update+with+the+name+of+your+vm

++++response+=+service.instances().start(project=project,+zone=zone,+instance=instance).execute()

++++print(response)
++++return+("OK")|2115517|requirements.txt文件|2115518|google-api-python-client
oauth2client
flask|2115519|您可以通过Cloud+Scheduler计划您的功能：|2115520|在function|2115521|Create+new+Cloud+scheduler+job|unordered-list-item|2115522|
|2115523|Specify中使用|2115524|Create+a+service+account将cron+format.|2115525|Specify+HTTP中的频率作为目标类型。functions.invoker+permission+|2115526|从Auth+header下拉列表中添加您的云函数和方法的URL和令牌OIDC+|2115527|在服务帐户文本框中添加服务帐户电子邮件。|2115528|在audience+field中，您只需编写函数的URL，而不需要任何其他参数|2115529|<functions.invoker+permission>G235|2115530|2115531|在云调度器上，我使用以下URL访问我的函数|2115532|https://us-central1-yourprojectID.cloudfunctions.net/power|2115533|我利用了这些观众|2115534|2115535|请在代码、URL和区域us-central1中替换yourprojectID|2115536|entityMap|0|LINK|mutability|MUTABLE|url|https://cloud.google.com/storage/docs/json_api/v1|1|https://cloud.google.com/compute/docs/startupscript|2|https://cloud.google.com/preemptible-vms|3|https://console.cloud.google.com/compute/instances|4|https://storage.googleapis.com/BUCKET/FILE|5|https://cloud.google.com/scheduler/docs/creating|6|https://cloud.google.com/iam/docs/creating-managing-service-accounts#console-console|7|https://cloud.google.com/functions/docs/securing/managing-access-iam#console-console|8^0|15|6|A|G|0|0|M|E|1|12|E|2|0|0|0|0|0|0|U|8|0|5|H|3|0|0|0|0|0|0|0|0|0|0|L|16|4|0|0|0|0|0|0|0|0|0|0|U|5|0|0|0|0|O|6|0|N|S|7|0|0|0|1|E|0|1|S|8|0|0|0|0|1M|0|0|0|1M|0|B|B|P|D|0^^$0|@$1|2|3|4|5|6|7|3D|8|@$9|3E|A|3F|B|C]]|D|@$9|3G|A|3H|1|3I]]|E|$]]|$1|F|3|G|5|6|7|3J|8|@]|D|@$9|3K|A|3L|1|3M]|$9|3N|A|3O|1|3P]]|E|$]]|$1|H|3|I|5|6|7|3Q|8|@]|D|@]|E|$]]|$1|J|3|K|5|6|7|3R|8|@]|D|@]|E|$]]|$1|L|3|M|5|6|7|3S|8|@]|D|@]|E|$]]|$1|N|3|O|5|6|7|3T|8|@]|D|@]|E|$]]|$1|P|3|Q|5|R|7|3U|8|@]|D|@]|E|$S|T]]|$1|U|3|V|5|6|7|3V|8|@$9|3W|A|3X|B|C]]|D|@]|E|$]]|$1|W|3|X|5|6|7|3Y|8|@]|D|@$9|3Z|A|40|1|41]]|E|$]]|$1|Y|3|Z|5|6|7|42|8|@]|D|@]|E|$]]|$1|10|3|11|5|6|7|43|8|@]|D|@]|E|$]]|$1|12|3|13|5|6|7|44|8|@]|D|@]|E|$]]|$1|14|3|15|5|6|7|45|8|@]|D|@]|E|$]]|$1|16|3|-4|5|6|7|46|8|@]|D|@]|E|$]]|$1|17|3|18|5|19|7|47|8|@]|D|@]|E|$]]|$1|1A|3|-4|5|6|7|48|8|@]|D|@]|E|$]]|$1|1B|3|1C|5|6|7|49|8|@]|D|@]|E|$]]|$1|1D|3|1E|5|6|7|4A|8|@]|D|@]|E|$]]|$1|1F|3|1G|5|6|7|4B|8|@]|D|@$9|4C|A|4D|1|4E]]|E|$]]|$1|1H|3|1I|5|6|7|4F|8|@]|D|@]|E|$]]|$1|1J|3|1K|5|6|7|4G|8|@]|D|@]|E|$]]|$1|1L|3|1M|5|6|7|4H|8|@]|D|@]|E|$]]|$1|1N|3|1O|5|R|7|4I|8|@]|D|@]|E|$S|T]]|$1|1P|3|1Q|5|6|7|4J|8|@]|D|@]|E|$]]|$1|1R|3|1S|5|R|7|4K|8|@]|D|@]|E|$S|T]]|$1|1T|3|1U|5|6|7|4L|8|@]|D|@]|E|$]]|$1|1V|3|1W|5|6|7|4M|8|@]|D|@]|E|$]]|$1|1X|3|1Y|5|1Z|7|4N|8|@]|D|@$9|4O|A|4P|1|4Q]]|E|$]]|$1|20|3|21|5|6|7|4R|8|@]|D|@]|E|$]]|$1|22|3|23|5|1Z|7|4S|8|@]|D|@]|E|$]]|$1|24|3|25|5|19|7|4T|8|@]|D|@$9|4U|A|4V|1|4W]]|E|$]]|$1|26|3|27|5|19|7|4X|8|@]|D|@$9|4Y|A|4Z|1|50]]|E|$]]|$1|28|3|29|5|19|7|51|8|@]|D|@]|E|$]]|$1|2A|3|2B|5|19|7|52|8|@]|D|@]|E|$]]|$1|2C|3|2D|5|19|7|53|8|@$9|54|A|55|B|C]]|D|@]|E|$]]|$1|2E|3|2F|5|6|7|56|8|@]|D|@$9|57|A|58|1|59]]|E|$]]|$1|2G|3|-4|5|6|7|5A|8|@]|D|@]|E|$]]|$1|2H|3|2I|5|6|7|5B|8|@]|D|@]|E|$]]|$1|2J|3|2K|5|6|7|5C|8|@$9|5D|A|5E|B|C]]|D|@]|E|$]]|$1|2L|3|2M|5|6|7|5F|8|@]|D|@]|E|$]]|$1|2N|3|2K|5|6|7|5G|8|@$9|5H|A|5I|B|C]]|D|@]|E|$]]|$1|2O|3|2P|5|6|7|5J|8|@$9|5K|A|5L|B|C]|$9|5M|A|5N|B|C]]|D|@]|E|$]]|$1|2Q|3|-4|5|6|7|5O|8|@]|D|@]|E|$]]]|2R|$2S|$5|2T|2U|2V|E|$2W|2X]]|2Y|$5|2T|2U|2V|E|$2W|2Z]]|30|$5|2T|2U|2V|E|$2W|31]]|32|$5|2T|2U|2V|E|$2W|33]]|34|$5|2T|2U|2V|E|$2W|35]]|36|$5|2T|2U|2V|E|$2W|37]]|38|$5|2T|2U|2V|E|$2W|39]]|3A|$5|2T|2U|2V|E|$2W|3B]]|3C|$5|2T|2U|2V|E|$2W|3B]]]]

Rsync is not an operation that can't be performed via a single request in the <a href="https://cloud.google.com/storage/docs/json_api/v1" rel="nofollow noreferrer">storage rest API</a>, and <code>gsutil</code> is not available on Cloud Functions, for this reason rsync both buckets via a python script is not possible.
You can create a function to start a <a href="https://cloud.google.com/preemptible-vms" rel="nofollow noreferrer">preemptible VM</a> with a <a href="https://cloud.google.com/compute/docs/startupscript" rel="nofollow noreferrer">startup script</a> that executes the rsync between buckets and shut down the instance after finalizing the rsync operation.
By using a VM instead of a serverless service you can avoid any timeout that could be generated by a long rsync process.
A preemptible VM can run for up to 24Hours before been stopped and you only will charged by the time that the instance is turned on (the disk storage will be charged independently of the status)
If the VM is powered off before a minute you won't be charged by the usage.
For this approach first is necessary to create a bash script in a bucket, this will be executed by the preemptible VM at the startup time for example:
<pre class="lang-sh prettyprint-override"><code>#! /bin/bash
gstuil rsync -r gs://mybucket1 gs://mybucket2 

sudo init 0 #this is similar to poweroff, halt or shutdown -h now
</code></pre>
After that, you need to create a preemptible VM with a Startup script, I recommend an <code>f1-micro</code> instance since the rsync command between buckets doesn't require so much resources.
1.- go to the <a href="https://console.cloud.google.com/compute/instances" rel="nofollow noreferrer">VM Instances page</a>.
2.- Click Create instance.
3.- On the Create a new instance page, fill in the properties for your instance.
4.- Click Management, security, disks, networking, sole tenancy.
5.In the Identity and API access section, select a service account that has access to read your startup script file in Cloud Storage and the buckets to be synced
<ol start="6">
<li>Select Allow full access to all Cloud APIs.</li>
</ol>
7.- Under Availability policy, set the Preemptibility option to On. This setting disables automatic restart for the instance, and sets the host maintenance action to Terminate.
8.- In the Metadata section, provide startup-script-url as the metadata key.
9.- In the Value box, provide a URL to the startup script file, either in the gs://BUCKET/FILE or <a href="https://storage.googleapis.com/BUCKET/FILE" rel="nofollow noreferrer">https://storage.googleapis.com/BUCKET/FILE</a> format.
10.Click Create to create the instance.
With this configuration every time that your instance will be started the script also will be executed.
This is the python function to start a VM (independently if this is preemptible)
<pre><code>def power(request):
 import logging
 # this libraries are mandatory to reach compute engine api
 from googleapiclient import discovery
 from oauth2client.client import GoogleCredentials

 # the function will take the service account of your function
 credentials = GoogleCredentials.get_application_default()

 # this line is to specify the api that we gonna use, in this case compute engine 
 service = discovery.build('compute', 'v1', credentials=credentials, cache_discovery=False)

 # set correct log level (to avoid noise in the logs)
 logging.getLogger('googleapiclient.discovery_cache').setLevel(logging.ERROR)

 # Project ID for this request.
 project = &quot;yourprojectID&quot; # Update placeholder value.
 zone = &quot;us-central1-a&quot; # update this to the zone of your vm
 instance = &quot;myvm&quot; # update with the name of your vm

 response = service.instances().start(project=project, zone=zone, instance=instance).execute()

 print(response)
 return (&quot;OK&quot;)
</code></pre>
requirements.txt file
<pre><code>google-api-python-client
oauth2client
flask
</code></pre>
And you can schedule your function by Cloud Scheduler:
<ol>
<li><a href="https://cloud.google.com/iam/docs/creating-managing-service-accounts#console-console" rel="nofollow noreferrer">Create a service account</a> with <a href="https://cloud.google.com/functions/docs/securing/managing-access-iam#console-console" rel="nofollow noreferrer">functions.invoker permission</a> within your function</li>
<li><a href="https://cloud.google.com/scheduler/docs/creating" rel="nofollow noreferrer">Create new Cloud scheduler job</a></li>
<li>Specify the frequency in cron format.</li>
<li>Specify HTTP as the target type.</li>
<li>Add the URL of your cloud function and method as always.</li>
<li>Select the token OIDC from the Auth header dropdown</li>
<li>Add the service account email in the Service account text box.</li>
<li>In <code>audience field</code> you must only need to write the URL of the function without any additional parameter</li>
</ol>
On cloud scheduler, I hit my function by using these URL
<code>https://us-central1-yourprojectID.cloudfunctions.net/power</code>
and I used this audience
<code>https://us-central1-yourprojectID.cloudfunctions.net/power</code>
please replace <code>yourprojectID</code> in the code and in the URLs and the zone <code>us-central1</code>

blocks|key|900443|text|执行此操作的python脚本将变得非常慢()。我将使用Dataflow+(apache+bream)批处理进程来完成此操作。您可以很容易地在python3中对此进行编码。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|900444|基本上，您需要：|900445|900446|One+Operation可列出所有文件。|unordered-list-item|900447|One+shuffle()操作可在多个工作进程之间分配负载。|900448|One+operation可实际从源复制到目标。|900449|900450|好的部分是Google将为您扩展工人，并且不会花费太多时间。您将为存储操作和移动所有数据所需的to+%2B+cpu付费。|900451|entityMap^0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|S|8|@]|9|@]|A|$]]|$1|E|3|F|5|G|7|T|8|@]|9|@]|A|$]]|$1|H|3|I|5|G|7|U|8|@]|9|@]|A|$]]|$1|J|3|K|5|G|7|V|8|@]|9|@]|A|$]]|$1|L|3|-4|5|6|7|W|8|@]|9|@]|A|$]]|$1|M|3|N|5|6|7|X|8|@]|9|@]|A|$]]|$1|O|3|-4|5|6|7|Y|8|@]|9|@]|A|$]]]|P|$]]

A python script to do this will be really slow.
I would use a Dataflow (apache bream) batch process to do this. You can code this in python3 easily.
Basically you need:
<ul>
<li>One Operation to list all files.</li>
<li>One shuffle() operation to distribute the load among several workers.</li>
<li>One Operation to actually copy from source to destination.</li>
</ul>
The good part is Google will scale the workers for you and won't take much time.
You'll be billed for the storage operations and the gigabytes + cpu that takes to move al data.

Google have Cloud Storage Data Transfer option to copy from one bucket to another but this will only work if both the buckets are in the same project. Using <code>gutil -m rsync -r -d</code> is an easy option to run as cron but we are migrating all bash to python3. So I need a python 3 script to use it as google cloud function to do a weekly copy whole bucket from project1 to another bucket in project2.
<pre><code>Language: python 3
app : Cloud Function
Process : Copy one bucket to another
Source Project: project1
Source bucket : bucket1
Dest Project: project2
Dest Bucket: bucket2
pseudo cmd: rsync -r gs://project1/bucket1 gs://project2/bucket2
</code></pre>
Any quick and readable python 3 code script to do that.

Cloud Function in Python3 - copy from Google Cloud Bucket to another Google Cloud Bucket

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

Google有云存储数据传输选项，可以从一个存储桶复制到另一个存储桶，但只有当两个存储桶都在同一个项目中时，这才会起作用。作为cron运行，使用gutil -m rsync -r -d是一个简单的选择，但是我们将所有的bash迁移到python3。因此，我需要一个Python3脚本来使用它作为google cloud函...

问Python3中的云函数-从Google Cloud Bucket复制到另一个Google Cloud Bucket
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Python3中的云函数-从Google Cloud Bucket复制到另一个Google Cloud BucketEN