Free Tool

Robots.txt Checker

Validate your robots.txt file and check for crawling issues. Instantly see every directive, detect misconfigurations, and get specific recommendations to fix them.

Enter your website URL — we'll find and parse your robots.txt automatically

Related tools

Noindex Detector

Indexing

Find exactly why a page may be missing from search results

Open tool

Meta Tags Analyzer

SEO

Analyze title, meta description, and Open Graph tags

Open tool

Page Speed Test

Performance

Analyze load time and Core Web Vitals for any URL

Open tool

Schema Generator

Markup

Generate structured data markup for better search visibility

Open tool

DNS Email Health

DNS

Check SPF, DKIM, DMARC and MX records for deliverability

Open tool

Shipcheck

Found an issue? Run a full pre-launch audit with Shipcheck before you deploy.

Shipcheck detects robots.txt changes automatically and alerts your team if a deploy accidentally blocks search engines.

Start free audit

No credit card required

FAQ

Frequently asked questions

What is robots.txt and why does it matter?

What happens if I block Googlebot in robots.txt?

Should I include a sitemap in robots.txt?

Does crawl-delay affect SEO?

Can I have different rules for different search engines?