Robots.txt - Graph View

A text file placed at the root of a website that tells web crawlers which pages or sections to crawl or skip.

View concept details

Related Concepts

Crawl Budget
Web Crawler
XML Sitemap
Technical SEO
Search Engine Optimization (SEO)
Indexability

← Back to full graph