blocks|key|2299511|text|我把问题解决了，也许有点傻，但很管用。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2299512|使用python，我在一个.sh文件上编写多行AWS下载命令，然后在终端上执行它。|2299513|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

I got the problem solved, may be a little bit stupid, but it works.

Using python, I write multiple line of AWS download commands on one single .sh file, then I execute it on the terminal.

blocks|key|1177812|text|有一个bash脚本可以从一个文件filename.txt读取所有文件名。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1177813|#!/bin/bash++
set+-e++
while+read+line++
do++
++aws+s3+cp+s3://bucket-name/$line+dest-path/++
done+<filename.txt|code-block|syntax|javascript|1177814|entityMap^0|G|C|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|P|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Q|8|@]|D|@]|E|$]]]|L|$]]

There is a bash script which can read all the filenames from a file <code>filename.txt</code>. 

<pre><code>#!/bin/bash 
set -e 
while read line 
do 
 aws s3 cp s3://bucket-name/$line dest-path/ 
done &lt;filename.txt
</code></pre>

blocks|key|1175258|text|您可能希望使用“同步”而不是"cp“。下面只下载/同步本地文件夹中扩展名为".txt“的文件：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1175259|aws+s3+sync+--exclude="*"+--include="*.txt"+s3://mybucket/mysubbucket+.|code-block|syntax|javascript|1175260|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

You might want to use "sync" instead of "cp". The following will download/sync only the files with the ".txt" extension in your local folder:

<pre><code>aws s3 sync --exclude="*" --include="*.txt" s3://mybucket/mysubbucket .
</code></pre>

blocks|key|2299583|text|还可以使用--recursive选项，如文档+for+cp命令中所描述的那样。它将递归地复制指定前缀下的所有对象。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2299584|示例：|2299585|aws+s3+cp+s3://folder1/folder2/folder3+.+--recursive|code-block|syntax|javascript|2299586|将获取folder3+1/folder3+2/folder3+3下的所有文件，并将它们复制到本地目录中。|2299587|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.aws.amazon.com/cli/latest/reference/s3/cp.html^0|5|B|R|2|K|2|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|W|8|@$9|X|A|Y|B|C]|$9|Z|A|10|B|C]]|D|@$9|11|A|12|1|13]]|E|$]]|$1|F|3|G|5|6|7|14|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|15|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|16|8|@]|D|@]|E|$]]|$1|O|3|-4|5|6|7|17|8|@]|D|@]|E|$]]]|P|$Q|$5|R|S|T|E|$U|V]]]]

Also one can use the <code>--recursive</code> option, as described in the <a href="http://docs.aws.amazon.com/cli/latest/reference/s3/cp.html" rel="noreferrer">documentation</a> for <code>cp</code> command. It will copy all objects under a specified prefix recursively. 

Example:

<pre><code>aws s3 cp s3://folder1/folder2/folder3 . --recursive
</code></pre>

will grab all files under folder1/folder2/folder3 and copy them to local directory.

blocks|key|1547108|text|根据文档，您也可以在s3+cp中使用include和exclude过滤器。所以你可以这样做：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1547109|aws+s3+cp+s3://bucket/folder/+.+--recursive+--exclude="*"+--include="2017-12-20*"|code-block|syntax|javascript|1547110|确保exclude和include过滤器的顺序正确，因为这可能会改变整个含义。|1547111|entityMap|0|LINK|mutability|MUTABLE|url|https://docs.aws.amazon.com/cli/latest/reference/s3/index.html#use-of-exclude-and-include-filters^0|A|5|I|7|Q|7|2|2|0|0|0|2|7|A|7|0^^$0|@$1|2|3|4|5|6|7|U|8|@$9|V|A|W|B|C]|$9|X|A|Y|B|C]|$9|Z|A|10|B|C]]|D|@$9|11|A|12|1|13]]|E|$]]|$1|F|3|G|5|H|7|14|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|15|8|@$9|16|A|17|B|C]|$9|18|A|19|B|C]]|D|@]|E|$]]|$1|M|3|-4|5|6|7|1A|8|@]|D|@]|E|$]]]|N|$O|$5|P|Q|R|E|$S|T]]]]

As per the <a href="https://docs.aws.amazon.com/cli/latest/reference/s3/index.html#use-of-exclude-and-include-filters" rel="noreferrer">doc</a> you can use <code>include</code> and <code>exclude</code> filters with <code>s3 cp</code> as well. So you can do something like this:

<pre><code>aws s3 cp s3://bucket/folder/ . --recursive --exclude="*" --include="2017-12-20*"
</code></pre>

Make sure you get the order of <code>exclude</code> and <code>include</code> filters right as that could change the whole meaning.

blocks|key|1547113|text|试过以上所有的方法。joy不多。最后，将@rajan的回答改编为一句话：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1547114|for+file+in+whatever*.txt;+do+{+aws+s3+cp+$file+s3://somewhere/in/my/bucket/;+}+done|code-block|syntax|javascript|1547115|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Tried all the above. Not much joy. Finally, adapted @rajan's reply into a one-liner:

<pre><code>for file in whatever*.txt; do { aws s3 cp $file s3://somewhere/in/my/bucket/; } done
</code></pre>

blocks|key|1175329|text|我想从文本文件中读取s3对象密钥，并将它们并行下载到我的机器上。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1175330|我用了这个命令|1175331|cat+<filename>.txt+%7C+parallel+aws+s3+cp+{}+<output_dir>|code-block|syntax|javascript|1175332|我的文本文件的内容如下：|1175333|s3://bucket-name/file1.wav
s3://bucket-name/file2.wav
s3://bucket-name/file3.wav|1175334|请确保您的文本文件末尾没有空行。您可以了解更多关于GNU并行这里的信息。|offset|length|1175335|entityMap|0|LINK|mutability|MUTABLE|url|https://www.gnu.org/software/parallel/^0|0|0|0|0|0|U|2|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Z|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|10|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|11|8|@]|9|@]|A|$]]|$1|K|3|L|5|F|7|12|8|@]|9|@]|A|$G|H]]|$1|M|3|N|5|6|7|13|8|@]|9|@$O|14|P|15|1|16]]|A|$]]|$1|Q|3|-4|5|6|7|17|8|@]|9|@]|A|$]]]|R|$S|$5|T|U|V|A|$W|X]]]]

I wanted to read s3 object keys from a text file and download them to my machine parallelly.
I used this command
<pre><code>cat &lt;filename&gt;.txt | parallel aws s3 cp {} &lt;output_dir&gt;
</code></pre>
The contents of my text file looked like this:
<pre><code>s3://bucket-name/file1.wav
s3://bucket-name/file2.wav
s3://bucket-name/file3.wav
</code></pre>
Please make sure you don't have an empty line at the end of your text file. You can learn more about GNU parallel <a href="https://www.gnu.org/software/parallel/" rel="nofollow noreferrer">here</a>

blocks|key|1547156|text|@Rajan的答案是一个非常好的答案，但是当在*.txt文件和源s3桶中找不到匹配时，它就失败了，但是下面的代码也解决了这个问题：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1547157|#!/bin/bash
while+IFS=+read+-r+line;+do
aws+s3+cp+s3://your-s3-source-bucket/folder/$line+s3://your-s3-destination/folder/
done+<try.txt|code-block|syntax|javascript|1547158|唯一需要的就是运行aws笔记本中的bash文件。|1547159|!chmod+%2Bx+YOUR-BASH-NAME.sh
!./YOUR-BASH-NAME.sh|1547160|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

@Rajan's answer is a very good one, however it fails when there is no match found in the *.txt file and the source s3 bucket, however below code resolves also this issue:
<pre><code>#!/bin/bash
while IFS= read -r line; do
aws s3 cp s3://your-s3-source-bucket/folder/$line s3://your-s3-destination/folder/
done &lt;try.txt
</code></pre>
The only thing you need is to run the bash file inside you aws notebook.
<pre><code>!chmod +x YOUR-BASH-NAME.sh
!./YOUR-BASH-NAME.sh
</code></pre>

I am having trouble downloading multiple files from AWS S3 buckets to my local machine. 

I have all the filenames that I want to download and I do not want others. How can I do that ? Is there any kind of loop in aws-cli I can do some iteration ?

There are couple hundreds files I need to download so that it seems not possible to use one single command that takes all filenames as arguments.

Copy multiple files from s3 bucket

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我在将多个文件从AWS S3桶下载到本地机器时遇到困难。我有我想下载的所有文件名，我不想要其他文件名。我怎么能这么做？在aws中有什么循环，我可以做一些迭代吗？我需要下载几百个文件，这样就不可能使用一个将所有文件名作为参数的命令。

问从s3桶复制多个文件
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问从s3桶复制多个文件EN