Check
robots.txt checker
Public pages only, one at a time
Free, up to 30 requests per 10 minutes per visitor.
Public only, entered by you
Snoei only reads pages that anyone can see without logging in, and only the address you enter. It skips login pages.
No email addresses or phone numbers
Snoei refuses to collect them. Links to mailto: and tel: are only counted, never shown or exported.
Copyright stays
Texts, tables and images belong to whoever made them. Use them to check or to quote with a source, not to copy.
One page at a time
No crawling of whole sites. Snoei reads what you give, at most checks the links or language versions of that one page, follows robots.txt and keeps the pace low.
How it works
robots.txt checker in three steps
Paste the address
Of the site or a page on it; Snoei looks at that site’s /robots.txt.
Read the rules
Snoei shows the groups per bot, the sitemaps and any rules it does not understand, with line numbers.
Test a path
Pick a bot and type a path. You see right away whether it is allowed and which rule decides; that rule lights up in the file.
Questions
Good to know
Which rule wins if several match?
The longest matching rule, according to the RFC 9309 standard. If an Allow and a Disallow are equally long, Allow wins. A * stands for anything and a $ for the end of the address.
Is robots.txt a lock on the door?
No, it is a request to polite bots. Anyone who really wants to keep something secret puts it behind a login. Snoei always follows robots.txt.
What if a site has no robots.txt?
Then everything is allowed. If the server returns an error (5xx), the standard says nothing is allowed for now; Snoei then does not read that site.