ToolSphere

Robots.txt Generator

More tools

Generate customized robots.txt files for Googlebot, Bingbot, and other web crawlers. Specify allow rules, disallowed directory paths, and sitemap URLs.

Robots.txt generator

Guide search spiders on which directories to crawl and indexing restrictions.

Crawl rules & permissions
User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Yandex Allow: / User-agent: * Disallow: /admin/ Disallow: /api/ Allow: / Sitemap: https://toolsphere.in/sitemap.xml

XML sitemap generator

Generate structured sitemap.xml layouts to submit to Google Search Console.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://toolsphere.in/</loc>
    <lastmod>2026-11-12</lastmod>
    <changefreq>daily</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://toolsphere.in/about</loc>
    <lastmod>2026-11-12</lastmod>
    <changefreq>daily</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://toolsphere.in/contact</loc>
    <lastmod>2026-11-12</lastmod>
    <changefreq>daily</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://toolsphere.in/tools/word-counter</loc>
    <lastmod>2026-11-12</lastmod>
    <changefreq>daily</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://toolsphere.in/tools/password-generator</loc>
    <lastmod>2026-11-12</lastmod>
    <changefreq>daily</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

On this browser only

Give feedback

Did this tool help?

Tell us what worked, what didn’t, or what you’d like next.

Opens Google Forms in a new tab with this tool’s name filled in. No sign-in or email required. Feedback is optional.

About Robots.txt Generator

What robots.txt Actually Controls (and Doesn't)

A robots.txt file tells well-behaved crawlers which parts of a site they're welcome to crawl — but it's fundamentally an honor-system convention, not an access control mechanism. Any crawler can simply ignore it, and in fact malicious scrapers routinely do; robots.txt only affects the behavior of crawlers that voluntarily choose to respect it, which major search engines generally do, but nothing enforces compliance.

This distinction matters because robots.txt is frequently misunderstood as a security or privacy tool — disallowing a path in robots.txt doesn't password-protect it, encrypt it, or hide it from anyone who already has (or guesses) the URL; it only asks compliant crawlers not to index it. A truly private path needs actual authentication, not a robots.txt entry. Blocking a path also doesn't guarantee it stays out of search results either — if other pages link to a disallowed URL, search engines can still show that URL (without crawling its content) based on the link alone, which is a common source of "why is this blocked page still showing up in search" confusion.

Different crawlers (Googlebot, Bingbot, AI-training crawlers like GPTBot) can be given different rules via separate User-agent blocks, which is increasingly relevant now that many sites want to allow search indexing while opting out of AI training crawls specifically.

Frequently asked questions

Does disallowing a path in robots.txt actually hide it from the public?

No — it only asks compliant crawlers not to crawl it; anyone with the direct URL can still access it, and non-compliant crawlers can ignore the directive entirely. Real privacy requires authentication, not a robots.txt entry.

Can a disallowed page still appear in search results?

Yes — if other pages link to it, a search engine can list the URL based on those links even without crawling the page's actual content, which is a common point of confusion about what "disallow" actually prevents.

Why would a site want different rules for different crawlers?

To allow search engine indexing while opting out of specific uses — for example, permitting Googlebot for search visibility while disallowing an AI-training-specific crawler, which requires separate User-agent blocks with different rules.

Have a suggestion for Robots.txt Generator?Share your feedback ↗