robots.txt, sitemap और क्रॉल नियंत्रण

Your site's front-door policy

robots.txt tells crawlers what they may fetch; meta robots and X-Robots-Tag decide what may be indexed; the XML sitemap lists what you want found and when it changed; canonical tags consolidate duplicates into one URL that earns all the credit.

These files are pure leverage: a one-line mistake can remove a site from results, and a one-line fix can bring it back. AI crawlers respect the same robots.txt, so your crawl policy is now also your AI-visibility policy.

What this site measures

Whether robots.txt exists and what it blocks (for search bots and for each major AI crawler separately), noindex directives, sitemap presence and entry count, canonical declarations, redirect chains, and whether missing pages return real 404s.

How to improve

• Keep robots.txt minimal: block admin paths, never whole sites by accident.
• Regenerate the sitemap automatically on publish; reference it in robots.txt.
• Use canonicals on every paginated, filtered or parameterized URL family.
• Keep redirect chains to one hop; update internal links to the final URL.
• Audit yearly: robots.txt archaeology finds rules nobody remembers writing.

Common mistakes

The staging Disallow that launched. Sitemaps full of redirected or dead URLs. Canonicals pointing every page at the homepage. Blocking CSS/JS so Google renders your pages broken.