OpenVibeNet

About the Robots.txt Tester

Read a site's robots.txt, see the rules that apply to each crawler, and test whether a specific URL would be allowed or blocked — the file where an accidental line can remove a whole site from search.

• Rules grouped by user-agent
• Test any URL against the live rules
• Flags the accidental site-wide disallow

Frequently asked questions

Does robots.txt remove a page from Google?

No. It stops crawling, not indexing — a blocked URL can still appear without a description. Use a noindex meta tag to remove a page properly.

Should robots.txt list my sitemap?

Yes. A Sitemap line is the simplest way to point every crawler at your sitemap without registering anywhere.

Related tools

• TCP Latency TestMeasure TCP connect latency to any host from our server: how long a connection to port 443 takes, repeated ten times to show the spread. No install.
• Port CheckerCheck whether a TCP port is open on any host, from outside your network. Test common services or any custom port. Instant.
• SSL Certificate CheckerCheck any site's SSL certificate: expiry date, issuer, chain completeness and supported TLS versions. SSL checker, instant results.
• Website Uptime CheckerCheck whether a website is up right now from our server: the HTTP status, where redirects lead and how long the server took to answer. "Is it down" checker.
• DNS Propagation CheckerCheck whether a DNS change has propagated by asking ten public resolvers (Google, Cloudflare, Quad9, OpenDNS and more) and comparing their answers. Instant.
• MTR ReportRun an MTR report from our server to any host: ten probes per hop with loss, last, best, average and worst time and jitter (StDev), plus each hop's network. IPv4 and IPv6.

Browse all network tools · OpenVibe home · Live streaming