FAQ
- Does Disallow keep a page out of Google?
- No. It stops crawling, but a blocked URL can still be indexed from links. Use a noindex tag on a crawlable page to keep it out.
- Do I need a robots.txt?
- Not strictly, but it's the standard place to list your sitemap, and without one some crawlers log errors.
More site tools
- WHOIS Lookup
- DNS Lookup
- SSL Certificate Checker
- HTTP Headers Checker
- Meta Tags Checker
- Email DNS Checker
- Domain Availability
- IP Lookup
Need the full picture? Run a complete check: