Robots 检测
使用方法
- 输入域名
- 查看 robots.txt 规则和 Sitemap 检测结果
常见问题
Disallow: / 是什么意思?
表示禁止所有搜索引擎抓取整站,上线后务必检查是否误设置。
检测robots.txt规则与sitemap可用性
| 地址 | https://www.volcengine.com/robots.txt |
|---|---|
| 状态 | 200 存在 |
| 规则条数 | 13 |
| 声明 Sitemap | 3 |
| 全站屏蔽 | 否 |
| User-agent | 指令 | 路径 |
|---|---|---|
| * | Disallow | /search$ |
| * | Disallow | /search? |
| * | Disallow | /search/ |
| * | Disallow | /*?*infrom=* |
| * | Disallow | /*?*_vtm_=* |
| * | Disallow | /*?*utm_source=* |
| * | Disallow | /*?*utm_medium=* |
| * | Disallow | /*?*utm_campaign=* |
| * | Disallow | /*?*utm_term=* |
| * | Disallow | /*?*utm_content=* |
| * | Disallow | /signin/ |
| * | Disallow | /api/ |
| CCBot | Allow | / |
# Robots.txt for www.volcengine.com User-agent: * Disallow: /search$ Disallow: /search? Disallow: /search/ Disallow: /*?*infrom=* Disallow: /*?*_vtm_=* Disallow: /*?*utm_source=* Disallow: /*?*utm_medium=* Disallow: /*?*utm_campaign=* Disallow: /*?*utm_term=* Disallow: /*?*utm_content=* Disallow: /signin/ Disallow: /api/ # Allow AI crawlers to access User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: MistralAI-Index User-agent: MistralAI-User User-agent: Google-Agent User-agent: Google-GeminiNotebook User-agent: Google-NotebookLM User-agent: Applebot User-agent: Doubao-User User-agent: Kimi-SearchBot User-agent: Kimi-User User-agent: TongyiBot User-agent: Qwen-User User-agent: YiyanBot User-agent: Toutiao User-agent: YuanbaoBot User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: MistralAI-Training User-agent: Bytespider User-agent: Doubaobot User-agent: QwenBot User-agent: CCBot Allow: / # Sitemap Sitemap: https://www.volcengine.com/sitemap/volcengine/index.xml Sitemap: https://www.volcengine.com/sitemap/article/index.xml Sitemap: https://www.volcengine.com/sitemap/docs/index.xml
| Sitemap 地址 | 状态 | 内容 |
|---|---|---|
| https://www.volcengine.com/sitemap/volcengine/index.xml | 200 | 索引文件,含 1 个子 sitemap |
| https://www.volcengine.com/sitemap/article/index.xml | 200 | 索引文件,含 138 个子 sitemap |
| https://www.volcengine.com/sitemap/docs/index.xml | 200 | 索引文件,含 2 个子 sitemap |
Disallow: / 是什么意思?
表示禁止所有搜索引擎抓取整站,上线后务必检查是否误设置。