Build a robots.txt file visually. Control which search engine bots can crawl which parts of your website. Add sitemap URL and custom rules. Download ready to upload.
A robots.txt file sits at the root of your website (e.g. https://example.com/robots.txt) and tells search engine crawlers which pages they can and cannot access. It is part of the Robots Exclusion Protocol โ a standard followed by all major search engines including Google, Bing, and Yandex.
Allow vs Disallow: "Allow" explicitly permits access to a path (useful when a parent path is disallowed). "Disallow" blocks crawlers from accessing that path and all its sub-pages. Disallow: / blocks everything; Disallow: /admin/ blocks only the admin section.
Important: robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in search results if other pages link to it. Use the noindex meta tag to prevent indexing.
Updated: May 2025 ยท Works on Chrome, Firefox, Safari, Edge ยท Mobile friendly