blocks|key|3200031|text|为此，您必须使用SDK，使用您最喜欢的语言和脚本来递归地筛选出您想要的文件格式的对象。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3200032|然后将列表导出为csv或json，只要您更喜欢更易读。|3200033|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

You will have to use the SDK for that, with your favorite language and a script to filter out objects recursively for the file formats you want
and then export the list as csv or json whatever you prefer more readable

blocks|key|1654138|text|以下是如何使用awscli和其他几个命令行工具(grep和awk，在Mac和Linux上可以免费获得)来解决这个问题的想法。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1654139|aws+s3+ls+s3://mybucket+--recursive+\
++++%7C+grep+-v+-E+'%5E.%2B/$'+\
++++%7C+awk+'{na=split($NF,+a,+".");+tot[a[na]]+%2B=+$3;+num[a[na]]%2B%2B;}+END+{for+(e+in+tot)+printf+"%2515d+%256d+%25s\n",+tot[e],+num[e],+e};'|code-block|syntax|javascript|1654140|一步一步，aws+s3+ls+s3://mybucket+--recursive的结果如下所示：|offset|length|style|CODE|1654141|2021-11-24+12:45:39++57600+cat.png
2021-09-29+13:15:48++93651+dog.png
2021-09-29+14:16:06+++1448+names.csv
2021-02-15+15:09:56++++++0+pets/
2021-02-15+15:09:56++++135+pets/pets.json|1654142|通过grep+-v+-E+'%5E.%2B/$'管道删除文件夹，结果如下所示：|1654143|2021-11-24+12:45:39++57600+cat.png
2021-09-29+13:15:48++93651+dog.png
2021-09-29+14:16:06+++1448+names.csv
2021-02-15+15:09:56++++135+pets/pets.json|1654144|最后，为每一行调用AWK脚本。它分割句号字符(split($NF,+a,+"."))上每一行的最后一个字，这样就可以计算出文件扩展名是什么(存储在a[na]中)。然后，它根据tot[extension]中的扩展名聚合文件大小，并在num[extension]中按扩展名进行文件计数。它最终按扩展名打印出聚合文件大小和文件计数，如下所示：|1654145|+++++151251++++++2+png
+++++++1448++++++1+csv
++++++++135++++++1+json|1654146|您也可以很简单地解决这个问题，例如在Python中使用boto3+SDK。|1654147|entityMap^0|0|0|5|Z|0|0|2|I|0|0|N|I|21|5|2F|E|37|E|0|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|11|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|12|8|@$I|13|J|14|K|L]]|9|@]|A|$]]|$1|M|3|N|5|D|7|15|8|@]|9|@]|A|$E|F]]|$1|O|3|P|5|6|7|16|8|@$I|17|J|18|K|L]]|9|@]|A|$]]|$1|Q|3|R|5|D|7|19|8|@]|9|@]|A|$E|F]]|$1|S|3|T|5|6|7|1A|8|@$I|1B|J|1C|K|L]|$I|1D|J|1E|K|L]|$I|1F|J|1G|K|L]|$I|1H|J|1I|K|L]]|9|@]|A|$]]|$1|U|3|V|5|D|7|1J|8|@]|9|@]|A|$E|F]]|$1|W|3|X|5|6|7|1K|8|@]|9|@]|A|$]]|$1|Y|3|-4|5|6|7|1L|8|@]|9|@]|A|$]]]|Z|$]]

Here's an idea for how to solve this with the awscli and a couple of other command lines tools (grep and awk, freely available on Mac and Linux).
<pre><code>aws s3 ls s3://mybucket --recursive \
 | grep -v -E '^.+/$' \
 | awk '{na=split($NF, a, &quot;.&quot;); tot[a[na]] += $3; num[a[na]]++;} END {for (e in tot) printf &quot;%15d %6d %s\n&quot;, tot[e], num[e], e};'
</code></pre>
Step by step, <code>aws s3 ls s3://mybucket --recursive</code> results in output like this:
<pre><code>2021-11-24 12:45:39 57600 cat.png
2021-09-29 13:15:48 93651 dog.png
2021-09-29 14:16:06 1448 names.csv
2021-02-15 15:09:56 0 pets/
2021-02-15 15:09:56 135 pets/pets.json
</code></pre>
Piping that through <code>grep -v -E '^.+/$'</code> removes the folders, and the result looks like this:
<pre><code>2021-11-24 12:45:39 57600 cat.png
2021-09-29 13:15:48 93651 dog.png
2021-09-29 14:16:06 1448 names.csv
2021-02-15 15:09:56 135 pets/pets.json
</code></pre>
Finally, the AWK script is called for each line. It splits the last word of each line on the period character (<code>split($NF, a, &quot;.&quot;)</code>) so it can work out what the file extension is (stored in <code>a[na]</code>). It then aggregates the file size by extension in <code>tot[extension]</code> and the file count by extension in <code>num[extension]</code>. It finally prints out the aggregated file size and file count by extension, which looks something like this:
<pre><code> 151251 2 png
 1448 1 csv
 135 1 json
</code></pre>
You could also solve this fairly simply e.g. in Python using the boto3 SDK.

blocks|key|2891403|text|下面是一个Python脚本，它将按扩展对对象进行计数，并按扩展计算总大小：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2891404|import+boto3

s3_resource+=+boto3.resource('s3')

sizes+=+{}
quantity+=+{}

for+object+in+s3_resource.Bucket('jstack-a').objects.all():
++if+not+object.key.endswith('/'):
++++extension+=+object.key.split('.')[-1]
++++sizes[extension]+=+sizes.get(extension,+0)+%2B+object.size
++++quantity[extension]+=+quantity.get(extension,+0)+%2B+1
++++

for+extension,+size+in+sizes.items():
++print(extension,+quantity[extension],+size)|code-block|syntax|javascript|2891405|如果有一个没有扩展的对象，那就有点好笑了。|2891406|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

Here's a Python script that will count objects by extension and compute the total size by extension:
<pre class="lang-py prettyprint-override"><code>import boto3

s3_resource = boto3.resource('s3')

sizes = {}
quantity = {}

for object in s3_resource.Bucket('jstack-a').objects.all():
 if not object.key.endswith('/'):
 extension = object.key.split('.')[-1]
 sizes[extension] = sizes.get(extension, 0) + object.size
 quantity[extension] = quantity.get(extension, 0) + 1
 

for extension, size in sizes.items():
 print(extension, quantity[extension], size)
</code></pre>
It goes a bit funny if there is an object without an extension.

We have multiple files on our S3 bucket with the same file extensions.
I would like to find a way to list all these file extensions with the amount of space they're taking up in our bucket in human readable format.
For example, instead of just listing out all the files with <code>aws s3 ls s3://ebio-rddata --recursive --human-readable --summarize</code>
I'd like to list only the file extensions with the total size they're taking:
<pre><code>.basedon.peaks.l2inputnormnew.bed.full | total size: 100 GB
.adapterTrim.round2.rmRep.sorted.rmDup.sorted.bam | total size: 200 GB
.logo.svg | total size: 400 MB
</code></pre>

Search S3 bucket for file extension and size

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我们的S3桶上有多个文件，具有相同的文件扩展名。我想找到一种方法来列出所有这些文件扩展名，并在我们的桶中以人类可读的格式占用大量的空间。例如，与其仅仅列出使用aws s3 ls s3://ebio-rddata --recursive --human-readable --summarize的所有文件我只想列出文件扩展...

问在S3桶中搜索文件扩展名和大小
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在S3桶中搜索文件扩展名和大小EN