Crawler
PurlockBot
PurlockBot is the crawler behind Purlock, a website fact checker. It reads public pages so site owners can find expired offers, contradictions, placeholders, broken links and missing pages.
User agent
PurlockBot/1.0 (+https://purlock.com/crawler)
Plain HTML only
Fetches pages over HTTP. Doesn't run JavaScript, fill in forms, log in, or write anything to your site.
Small footprint
Up to 20 pages per free scan (100 to 300 for verified subscribers), plus quick status checks on links found on them.
Respects robots.txt
Including rules for PurlockBot specifically. If your robots.txt says no, we don't scan.
Strict limits
Stops after 10 seconds or 2 MB per page, and follows at most 5 redirects.
Rate limited
At most one fresh free scan per domain every 24 hours. Ongoing monitoring only for owners who verified the site.
Public pages only
Never accesses private areas or internal network addresses.
How to block it
Add this to your robots.txt:
User-agent: PurlockBot Disallow: /
Skip demos and examples
Product mockups, sample dashboards and examples often contain made-up prices or placeholder text on purpose. PurlockBot already skips code samples, hidden elements and obvious mockups. To tell it to skip any other section, add this attribute to it:
<div data-purlock-ignore>
…your demo or example…
</div>Google's data-nosnippet attribute is respected too.
Contact
Questions or problems: hello@purlock.com