blocks|key|492934|text|目前，Cloudera的备份和灾难恢复不支持局限性中列出的Google。请通过此链接查看整个文档以获得配置Google云存储连接。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|492935|上述方法将有效。我们只需要添加几个步骤就可以了：|492936|我们首先需要使用云互连或云VPN在or网络和Google网络之间建立一个私有链接。|ordered-list-item|492937|数据传输需要Dataproc集群。|492938|使用Google连接到主实例。|492939|最后，您可以运行DistCp命令来移动数据。|style|CODE|492940|有关更详细的信息，您可以查看有关使用DistCp将数据复制到云存储的完整文档。|492941|谷歌也有自己的BDR，你可以检查这个数据恢复规划指南。|492942|请注意，不能成为集群的默认文件系统。|492943|您还可以检查以下链接：与Google合作伙伴合作|492944|您可以使用以下连接器：|492945|在Spark+(或PySpark)或Hadoop应用程序中使用gs://前缀。|unordered-list-item|492946|hadoop：hadoop+fs+-ls+gs://bucket/dir/file。|492947|云控制台云存储浏览器。|492948|使用gsutil+cp或gsutil+rsync命令。|492949|您可以在使用连接器上查看这个完整的文档。|492950|如果你有问题请告诉我。|492951|entityMap|0|LINK|mutability|MUTABLE|url|https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_gcs_config.html#gcs_limitations|1|https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_gcs_config.html|2|https://cloud.google.com/architecture/hadoop/hadoop-gcp-migration-data?hl=en#using_distcp_to_copy_your_data_to_cloud_storage|3|https://cloud.google.com/architecture/dr-scenarios-planning-guide|4|https://cloud.google.com/architecture/hadoop/hadoop-gcp-migration-data?hl=en#working_with_cloud_storage_partners|5|https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage#using_the_connector^0|M|3|0|1F|D|1|0|0|0|0|0|8|6|0|G|H|2|0|I|8|3|0|0|B|D|4|0|0|V|5|0|7|Y|0|0|2|9|C|C|0|4|5|5|0|0^^$0|@$1|2|3|4|5|6|7|1V|8|@]|9|@$A|1W|B|1X|1|1Y]|$A|1Z|B|20|1|21]]|C|$]]|$1|D|3|E|5|6|7|22|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|23|8|@]|9|@]|C|$]]|$1|I|3|J|5|H|7|24|8|@]|9|@]|C|$]]|$1|K|3|L|5|H|7|25|8|@]|9|@]|C|$]]|$1|M|3|N|5|H|7|26|8|@$A|27|B|28|O|P]]|9|@]|C|$]]|$1|Q|3|R|5|6|7|29|8|@]|9|@$A|2A|B|2B|1|2C]]|C|$]]|$1|S|3|T|5|6|7|2D|8|@]|9|@$A|2E|B|2F|1|2G]]|C|$]]|$1|U|3|V|5|6|7|2H|8|@]|9|@]|C|$]]|$1|W|3|X|5|6|7|2I|8|@]|9|@$A|2J|B|2K|1|2L]]|C|$]]|$1|Y|3|Z|5|6|7|2M|8|@]|9|@]|C|$]]|$1|10|3|11|5|12|7|2N|8|@$A|2O|B|2P|O|P]]|9|@]|C|$]]|$1|13|3|14|5|12|7|2Q|8|@$A|2R|B|2S|O|P]]|9|@]|C|$]]|$1|15|3|16|5|12|7|2T|8|@]|9|@]|C|$]]|$1|17|3|18|5|12|7|2U|8|@$A|2V|B|2W|O|P]|$A|2X|B|2Y|O|P]]|9|@]|C|$]]|$1|19|3|1A|5|6|7|2Z|8|@]|9|@$A|30|B|31|1|32]]|C|$]]|$1|1B|3|1C|5|6|7|33|8|@]|9|@]|C|$]]|$1|1D|3|-4|5|6|7|34|8|@]|9|@]|C|$]]]|1E|$1F|$5|1G|1H|1I|C|$1J|1K]]|1L|$5|1G|1H|1I|C|$1J|1M]]|1N|$5|1G|1H|1I|C|$1J|1O]]|1P|$5|1G|1H|1I|C|$1J|1Q]]|1R|$5|1G|1H|1I|C|$1J|1S]]|1T|$5|1G|1H|1I|C|$1J|1U]]]]

As of the moment, Cloudera Manager’s Backup and Disaster Recovery does not support Google Cloud Storage it is listed in <a href="https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_gcs_config.html#gcs_limitations" rel="nofollow noreferrer">limitations</a>. Please check the whole documentation through this link for <a href="https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_gcs_config.html" rel="nofollow noreferrer">Configuring Google Cloud Storage Connectivity</a>.
The above approach will work. We just need to add a few steps to begin with:
<ol>
<li>We first need to establish a private link between on-prem network and Google network using Cloud Interconnect or Cloud VPN.</li>
<li>Dataproc cluster is needed for data transfer.</li>
<li>Use Google CLI to connect to your master's instance.</li>
<li>Finally, you can run <code>DistCp</code> commands to move your data.</li>
</ol>
For more detailed information, you may check this full documentation on <a href="https://cloud.google.com/architecture/hadoop/hadoop-gcp-migration-data?hl=en#using_distcp_to_copy_your_data_to_cloud_storage" rel="nofollow noreferrer">Using DistCp to copy your data to Cloud Storage</a>.
Google also has its own BDR and you can check this <a href="https://cloud.google.com/architecture/dr-scenarios-planning-guide" rel="nofollow noreferrer">Data Recovery planning guide</a>.
Please be advised that Google Cloud Storage cannot be the default file system for the cluster.
You can also check this link: <a href="https://cloud.google.com/architecture/hadoop/hadoop-gcp-migration-data?hl=en#working_with_cloud_storage_partners" rel="nofollow noreferrer">Working with Google Cloud partners</a>
You could either use the following connectors:
<ul>
<li>In a Spark (or PySpark) or Hadoop application using the <code>gs://</code> prefix.</li>
<li>The hadoop shell: <code>hadoop fs -ls gs://bucket/dir/file</code>.</li>
<li>The Cloud Console Cloud Storage browser.</li>
<li>Using the <code>gsutil cp</code> or <code>gsutil rsync</code> commands.</li>
</ul>
You can check this full documentation on <a href="https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage#using_the_connector" rel="nofollow noreferrer">using connectors</a>.
Let me know if you have questions.

In the hope of achieving Cloudera Backup and Disaster Recovery to AWS-like functionality in GCP, I am searching for some alternatives.
Will the below approach work?
<ol>
<li>adding GCP connector to an on-prem Cloudera cluster</li>
<li>then copying with <code>hadoop dist-cp</code></li>
<li>then syncing hdfs source directory to gcs directory with <code>gsutil rsync [OPTION]... src_url dst_url</code></li>
</ol>
If the above approach is not possible then is there any other alternative to achieve Cloudera BDR in Google Cloud Storage (GCS)?

Syncing HDFS data and google cloud storage data (for BDR)

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

为了在GCP中实现Cloudera备份和AWS类功能的灾难恢复，我正在寻找一些替代方案。下面的方法会有效吗？将GCP连接器添加到on-prem Cloudera集群中然后用hadoop dist-cp复制然后用gsutil rsync [OPTION]... src_url dst_url同步hdfs源目录到gcs目录。如果上面的方法是不可能的，那么在Google云存储(GCS)中实现Cloude

问同步HDFS数据和google云存储数据(用于BDR)
EN

回答 1

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问同步HDFS数据和google云存储数据(用于BDR)EN