www.volcengine.com 的Robots检测结果

检测robots.txt规则与sitemap可用性

实时查询 · 2026-11-21 07:05:36

robots.txt 概况

地址https://www.volcengine.com/robots.txt
状态200 存在
规则条数13
声明 Sitemap3
全站屏蔽否

抓取规则

User-agent指令路径
*Disallow/search$
*Disallow/search?
*Disallow/search/
*Disallow/*?*infrom=*
*Disallow/*?*_vtm_=*
*Disallow/*?*utm_source=*
*Disallow/*?*utm_medium=*
*Disallow/*?*utm_campaign=*
*Disallow/*?*utm_term=*
*Disallow/*?*utm_content=*
*Disallow/signin/
*Disallow/api/
CCBotAllow/

robots.txt 原文

展开查看
# Robots.txt for www.volcengine.com

User-agent: *
Disallow: /search$
Disallow: /search?
Disallow: /search/
Disallow: /*?*infrom=*
Disallow: /*?*_vtm_=*
Disallow: /*?*utm_source=*
Disallow: /*?*utm_medium=*
Disallow: /*?*utm_campaign=*
Disallow: /*?*utm_term=*
Disallow: /*?*utm_content=*
Disallow: /signin/
Disallow: /api/


# Allow AI crawlers to access
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: Google-Agent
User-agent: Google-GeminiNotebook
User-agent: Google-NotebookLM
User-agent: Applebot
User-agent: Doubao-User
User-agent: Kimi-SearchBot
User-agent: Kimi-User
User-agent: TongyiBot
User-agent: Qwen-User
User-agent: YiyanBot
User-agent: Toutiao
User-agent: YuanbaoBot
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: Doubaobot
User-agent: QwenBot
User-agent: CCBot
Allow: /

# Sitemap
Sitemap: https://www.volcengine.com/sitemap/volcengine/index.xml
Sitemap: https://www.volcengine.com/sitemap/article/index.xml
Sitemap: https://www.volcengine.com/sitemap/docs/index.xml

Sitemap 检测

Sitemap 地址状态内容
https://www.volcengine.com/sitemap/volcengine/index.xml200索引文件,含 1 个子 sitemap
https://www.volcengine.com/sitemap/article/index.xml200索引文件,含 138 个子 sitemap
https://www.volcengine.com/sitemap/docs/index.xml200索引文件,含 2 个子 sitemap

Robots 检测

使用方法

  1. 输入域名
  2. 查看 robots.txt 规则和 Sitemap 检测结果

常见问题

Disallow: / 是什么意思?
表示禁止所有搜索引擎抓取整站,上线后务必检查是否误设置。