blocks|key|1763913|text|wget+-S+-i+*file*将从文件中的每个url中获取标题。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1763914|通过grep过滤特定的状态代码。|1763915|entityMap^0|0|H|0|2|4|0^^$0|@$1|2|3|4|5|6|7|J|8|@$9|K|A|L|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|M|8|@$9|N|A|O|B|C]]|D|@]|E|$]]|$1|H|3|-4|5|6|7|P|8|@]|D|@]|E|$]]]|I|$]]

<code>wget -S -i *file*</code> will get you the headers from each url in a file.

Filter though <code>grep</code> for the status code specifically.

blocks|key|1763946|text|使用curl只获取HTTP-header+(而不是整个文件)并解析它：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1763947|$+curl+-I++--stderr+/dev/null+http://www.google.co.uk/index.html+%7C+head+-1+%7C+cut+-d'+'+-f2
200|code-block|syntax|javascript|1763948|entityMap|0|LINK|mutability|MUTABLE|url|http://curl.haxx.se/^0|2|4|2|4|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]]|D|@$9|V|A|W|1|X]]|E|$]]|$1|F|3|G|5|H|7|Y|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Z|8|@]|D|@]|E|$]]]|L|$M|$5|N|O|P|E|$Q|R]]]]

Use <a href="http://curl.haxx.se/"><code>curl</code></a> to fetch the HTTP-header only (not the whole file) and parse it:

<pre><code>$ curl -I --stderr /dev/null http://www.google.co.uk/index.html | head -1 | cut -d' ' -f2
200
</code></pre>

blocks|key|1977573|text|wget+--spider+-S+"http://url/to/be/checked"+2>&1+%7C+grep+"HTTP/"+%7C+awk+'{print+$2}'|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|1977574|仅为您打印状态代码|unstyled|1977575|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$B|C]]|$1|D|3|E|5|F|7|J|8|@]|9|@]|A|$]]|$1|G|3|-4|5|F|7|K|8|@]|9|@]|A|$]]]|H|$]]

<pre><code>wget --spider -S "http://url/to/be/checked" 2&gt;&amp;1 | grep "HTTP/" | awk '{print $2}'
</code></pre>

prints only the status code for you

blocks|key|1764229|text|扩展了Phil已经提供的答案。如果您使用xargs进行调用，那么在bash中添加并行性是不需要动脑筋的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1764230|代码如下：|1764231|xargs+-n1+-P+10+curl+-o+/dev/null+--silent+--head+--write-out+'%25{url_effective}:+%25{http_code}\n'+<+url.lst|code-block|syntax|javascript|1764232|-n1:只使用一个值(来自列表)作为curl调用的参数|1764233|-P10:随时保持10个curl进程活动(即10个并行连接)|1764234|查看curl手册中的write_out参数，了解可以使用它提取的更多数据(时间等)。|offset|length|style|CODE|1764235|如果它对某人有帮助，这是我目前使用的呼叫：|1764236|xargs+-n1+-P+10+curl+-o+/dev/null+--silent+--head+--write-out+'%25{url_effective};%25{http_code};%25{time_total};%25{time_namelookup};%25{time_connect};%25{size_download};%25{speed_download}\n'+<+url.lst+%7C+tee+results.csv|1764237|它只是将一堆数据输出到csv文件中，该文件可以导入到任何办公工具中。|1764238|entityMap^0|0|0|0|0|0|A|9|0|0|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|11|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|12|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|13|8|@]|9|@]|A|$]]|$1|K|3|L|5|6|7|14|8|@]|9|@]|A|$]]|$1|M|3|N|5|6|7|15|8|@$O|16|P|17|Q|R]]|9|@]|A|$]]|$1|S|3|T|5|6|7|18|8|@]|9|@]|A|$]]|$1|U|3|V|5|F|7|19|8|@]|9|@]|A|$G|H]]|$1|W|3|X|5|6|7|1A|8|@]|9|@]|A|$]]|$1|Y|3|-4|5|6|7|1B|8|@]|9|@]|A|$]]]|Z|$]]

Extending the answer already provided by Phil. Adding parallelism to it is a no brainer in bash if you use xargs for the call.

Here the code:

<pre><code>xargs -n1 -P 10 curl -o /dev/null --silent --head --write-out '%{url_effective}: %{http_code}\n' &lt; url.lst
</code></pre>

-n1: use just one value (from the list) as argument to the curl call

-P10: Keep 10 curl processes alive at any time (i.e. 10 parallel connections)

Check the <code>write_out</code> parameter in the manual of curl for more data you can extract using it (times, etc).

In case it helps someone this is the call I'm currently using:

<pre><code>xargs -n1 -P 10 curl -o /dev/null --silent --head --write-out '%{url_effective};%{http_code};%{time_total};%{time_namelookup};%{time_connect};%{size_download};%{speed_download}\n' &lt; url.lst | tee results.csv
</code></pre>

It just outputs a bunch of data into a csv file that can be imported into any office tool.

blocks|key|1977897|text|由于https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P+(+xargs中并行作业的输出可能会被混合)，我会使用GNU+parallel而不是xargs来并行化：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1977898|cat+url.lst+%7C
++parallel+-P0+-q+curl+-o+/dev/null+--silent+--head+--write-out+'%25{url_effective}:+%25{http_code}\n'+>+outfile|code-block|syntax|javascript|1977899|在这种情况下，使用xargs可能是安全的，因为输出很短，所以使用xargs的问题是，如果后来有人为了做更大的事情而更改了代码，它将不再安全。或者如果有人读到这个问题，认为他可以用其他东西取代curl，那么这也可能是不安全的。|1977900|entityMap|0|LINK|mutability|MUTABLE|url|https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P^0|25|5|39|5|2|20|0|0|0|9|5|W|5|2N|4|0^^$0|@$1|2|3|4|5|6|7|U|8|@$9|V|A|W|B|C]|$9|X|A|Y|B|C]]|D|@$9|Z|A|10|1|11]]|E|$]]|$1|F|3|G|5|H|7|12|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|13|8|@$9|14|A|15|B|C]|$9|16|A|17|B|C]|$9|18|A|19|B|C]]|D|@]|E|$]]|$1|M|3|-4|5|6|7|1A|8|@]|D|@]|E|$]]]|N|$O|$5|P|Q|R|E|$S|T]]]]

Due to <a href="https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P" rel="nofollow noreferrer">https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P</a> (output from parallel jobs in <code>xargs</code> risks being mixed), I would use GNU Parallel instead of <code>xargs</code> to parallelize:

<pre><code>cat url.lst |
 parallel -P0 -q curl -o /dev/null --silent --head --write-out '%{url_effective}: %{http_code}\n' &gt; outfile
</code></pre>

In this particular case it may be safe to use <code>xargs</code> because the output is so short, so the problem with using <code>xargs</code> is rather that if someone later changes the code to do something bigger, it will no longer be safe. Or if someone reads this question and thinks he can replace <code>curl</code> with something else, then that may also not be safe.

blocks|key|1764373|text|我发现了一个用Python语言编写的工具“webchk”。返回urls列表的状态代码。https://pypi.org/project/webchk/|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1764374|输出如下所示：|1764375|▶+webchk+-i+./dxieu.txt+%7C+grep+'200'
http://salesforce-case-status.dxi.eu/login+...+200+OK+(0.108)
https://support.dxi.eu/hc/en-gb+...+200+OK+(0.389)
https://support.dxi.eu/hc/en-gb+...+200+OK+(0.401)|code-block|syntax|javascript|1764376|希望这能有所帮助！|1764377|entityMap|0|LINK|mutability|MUTABLE|url|https://pypi.org/project/webchk/^0|17|W|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@$A|V|B|W|1|X]]|C|$]]|$1|D|3|E|5|6|7|Y|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|Z|8|@]|9|@]|C|$I|J]]|$1|K|3|L|5|6|7|10|8|@]|9|@]|C|$]]|$1|M|3|-4|5|6|7|11|8|@]|9|@]|C|$]]]|N|$O|$5|P|Q|R|C|$S|T]]]]

I found a tool "webchk” written in Python. Returns a status code for a list of urls.
<a href="https://pypi.org/project/webchk/" rel="nofollow noreferrer">https://pypi.org/project/webchk/</a>

Output looks like this:

<pre><code>▶ webchk -i ./dxieu.txt | grep '200'
http://salesforce-case-status.dxi.eu/login ... 200 OK (0.108)
https://support.dxi.eu/hc/en-gb ... 200 OK (0.389)
https://support.dxi.eu/hc/en-gb ... 200 OK (0.401)
</code></pre>

Hope that helps!

blocks|key|1978334|text|请记住，curl并不总是可用的(特别是在容器中)，此解决方案存在问题：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1978335|wget+--server-response+--spider+--quiet+"${url}"+2>&1+%7C+awk+'NR==1{print+$2}'|code-block|syntax|javascript|1978336|即使URL不存在，它也会返回0的退出状态。|1978337|或者，下面是使用wget的一个合理的容器健康检查：|1978338|wget+-S+--spider+-q+-t+1+"${url}"+2>&1+%7C+grep+"200+OK"+>+/dev/null|1978339|虽然它可能不会给你确切的状态，但它至少会给你一个有效的基于退出代码的健康响应(即使在端点上有重定向)。|1978340|entityMap^0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|S|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|T|8|@]|9|@]|A|$]]|$1|K|3|L|5|D|7|U|8|@]|9|@]|A|$E|F]]|$1|M|3|N|5|6|7|V|8|@]|9|@]|A|$]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

Keeping in mind that curl is not always available (particularly in containers), there are issues with this solution:
<pre><code>wget --server-response --spider --quiet &quot;${url}&quot; 2&gt;&amp;1 | awk 'NR==1{print $2}'
</code></pre>
which will return exit status of 0 even if the URL doesn't exist.
Alternatively, here is a reasonable container health-check for using wget:
<pre><code>wget -S --spider -q -t 1 &quot;${url}&quot; 2&gt;&amp;1 | grep &quot;200 OK&quot; &gt; /dev/null
</code></pre>
While it may not give you exact status out, it will at least give you a valid exit code based health responses (even with redirects on the endpoint).

I have a list of URLS that I need to check, to see if they still work or not. I would like to write a bash script that does that for me.

I only need the returned HTTP status code, i.e. 200, 404, 500 and so forth. Nothing more.

EDIT Note that there is an issue if the page says "404 not found" but returns a 200 OK message. It's a misconfigured web server, but you may have to consider this case.

For more on this, see <a href="https://stackoverflow.com/questions/6439993/check-if-a-url-goes-to-a-page-containing-the-text-404">Check if a URL goes to a page containing the text &quot;404&quot;</a>

Script to get the HTTP status code of a list of urls?

服务器

容器

Python

我有一个需要检查的URLS列表，看看它们是否仍然有效。我想写一个bash脚本来帮我做到这一点。我只需要返回的HTTP状态码，即200、404、500等等。没别的了。编辑请注意，如果页面显示"404未找到“，但返回200 OK消息，则会出现问题。这是一个配置错误的web服务器，但您可能必须考虑这种情况。有关这方面的更多信...

问获取urls列表的HTTP状态代码的脚本？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问获取urls列表的HTTP状态代码的脚本？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问获取urls列表的HTTP状态代码的脚本？
EN