blocks|key|866492|text|该文件将允许所有爬行器访问|type|unstyled|depth|inlineStyleRanges|entityRanges|data|866493|User-agent:+*
Allow:+/|code-block|syntax|javascript|866494|这基本上允许所有用户代理(+*)访问站点的所有部分(+/)。|866495|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

<p>That file will allow all crawlers access</p>

<pre><code>User-agent: *
Allow: /
</code></pre>

<p>This basically allows all user agents (the *) to all parts of the site (the /).</p>


blocks|key|866528|text|这意味着您允许每个(*)用户代理/爬虫程序访问您站点的根目录(/)。你会好起来的。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|866529|entityMap^0|A|1|V|1|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]|$9|K|A|L|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|M|8|@]|D|@]|E|$]]]|G|$]]

<p>It means you allow every (<code>*</code>) user-agent/crawler to access the root (<code>/</code>) of your site. You're okay.</p>


blocks|key|866549|text|我知道这是一个相当古老的问题，并且有一些很好的答案。但是，为了完整起见，这里是我的两点意见。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|866550|根据官方documentation，有四种方式，你可以允许机器人完全访问你的网站。|offset|length|866551|干净：|866552|指定带有@unor提到的不允许段的全局匹配器。所以你的/robots.txt看起来像这样。|style|CODE|866553|User-agent:+*
Disallow:|code-block|syntax|javascript|866554|黑客：|866555|创建一个不包含任何内容的/robots.txt文件。这将默认为允许所有类型的Bots的全部。|866556|我才不管呢：|866557|请勿完全创建/robots.txt。这应该会产生与上面两个完全相同的结果。|866558|丑陋的：|866559|在robots+documentation+for+meta+tags中，您可以在站点上的所有页面上使用以下元标记，让Bots知道这些页面不应该被索引。|866560|<META+NAME="ROBOTS"+CONTENT="NOINDEX">|866561|为了将其应用于整个站点，您必须为所有页面添加此meta标记。这个标签应该严格地放在页面的HEAD标签下。关于这个元标签here的更多信息。|866562|entityMap|0|LINK|mutability|MUTABLE|url|http://www.robotstxt.org/robotstxt.html|1|http://www.robotstxt.org/faq/noindex.html|2|https://www.metatags.org/meta_name_robots^0|0|4|D|0|0|0|R|B|0|0|0|C|B|12|4|0|0|6|B|0|0|1N|4|1|Y|1|0|0|18|4|1N|4|2|0^^$0|@$1|2|3|4|5|6|7|1I|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|1J|8|@]|9|@$D|1K|E|1L|1|1M]]|A|$]]|$1|F|3|G|5|6|7|1N|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|1O|8|@$D|1P|E|1Q|J|K]]|9|@]|A|$]]|$1|L|3|M|5|N|7|1R|8|@]|9|@]|A|$O|P]]|$1|Q|3|R|5|6|7|1S|8|@]|9|@]|A|$]]|$1|S|3|T|5|6|7|1T|8|@$D|1U|E|1V|J|K]|$D|1W|E|1X|J|K]]|9|@]|A|$]]|$1|U|3|V|5|6|7|1Y|8|@]|9|@]|A|$]]|$1|W|3|X|5|6|7|1Z|8|@$D|20|E|21|J|K]]|9|@]|A|$]]|$1|Y|3|Z|5|6|7|22|8|@]|9|@]|A|$]]|$1|10|3|11|5|6|7|23|8|@$D|24|E|25|J|K]]|9|@$D|26|E|27|1|28]]|A|$]]|$1|12|3|13|5|N|7|29|8|@]|9|@]|A|$O|P]]|$1|14|3|15|5|6|7|2A|8|@$D|2B|E|2C|J|K]]|9|@$D|2D|E|2E|1|2F]]|A|$]]|$1|16|3|-4|5|6|7|2G|8|@]|9|@]|A|$]]]|17|$18|$5|19|1A|1B|A|$1C|1D]]|1E|$5|19|1A|1B|A|$1C|1F]]|1G|$5|19|1A|1B|A|$1C|1H]]]]

<p>I understand that this is fairly old question and has some pretty good answers. But, here is my two cents for the sake of completeness.</p>

<p>As per the official <a href="http://www.robotstxt.org/robotstxt.html" rel="nofollow noreferrer">documentation</a>, there are four ways, you can allow complete access for robots to access your site.</p>

<h2>Clean:</h2>

<p>Specify a global matcher with a disallow segment as mentioned by @unor. So your <code>/robots.txt</code> looks like this.</p>

<pre><code>User-agent: *
Disallow:
</code></pre>

<h2>The hack:</h2>

<p>Create a <code>/robots.txt</code> file with no content in it. Which will default to allow all for all type of <code>Bots</code>.</p>

<h2>I don't care way:</h2>

<p>Do not create a <code>/robots.txt</code> altogether. Which should yield the exact same results as the above two.</p>

<h2>The ugly:</h2>

<p>From the <a href="http://www.robotstxt.org/faq/noindex.html" rel="nofollow noreferrer">robots documentation for meta tags</a>, You can use the following meta tag on all your pages on your site to let the <code>Bots</code> know that these pages are not supposed to be indexed.</p>

<pre><code>&lt;META NAME="ROBOTS" CONTENT="NOINDEX"&gt;
</code></pre>

<p>In order for this to be applied to your entire site, You will have to add this meta tag for all of your pages. And this tag should strictly be placed under your <code>HEAD</code> tag of the page. More about this meta tag <a href="https://www.metatags.org/meta_name_robots" rel="nofollow noreferrer">here</a>.</p>


blocks|key|866582|text|我认为你做得很好，你允许所有的页面抓取User-agent：*+allow:/|type|unstyled|depth|inlineStyleRanges|entityRanges|data|866583|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

<p>I think you are good, you're allowing all pages to crawling
User-agent: *
allow:/</p>


<p>My <code>robots.txt</code> in Google Webmaster Tools shows the following values:</p>

<pre><code>User-agent: *
Allow: /
</code></pre>

<p>What does it mean? I don't have enough knowledge about it, so looking for your help. I want to allow all robots to crawl my website, is this the right configuration?</p>


How to configure robots.txt to allow everything?

谷歌站长工具中的My robots.txt显示以下值：User-agent: *Allow: /什么意思？我对这方面的知识还不够，所以寻求你的帮助。我想要允许所有的机器人爬行我的网站，这是正确的配置吗？

问如何将robots.txt配置为允许所有内容？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何将robots.txt配置为允许所有内容？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何将robots.txt配置为允许所有内容？
EN