Should I block AI crawlers?

That's your call. Blocking search and answer crawlers keeps you out of AI answers. Blocking training crawlers doesn't, so many sites allow the first and block the second.

Does robots.txt really stop them?

It's a request. The large companies say they follow it. Some crawlers don't.

An optional plain text summary of your site for AI tools. It's a young idea and not every tool reads it, but it's quick to add.

Can a firewall block crawlers too?

Yes, and this check can't see that. It reads your robots.txt, so a firewall rule can still turn a crawler away.

Free tool · AI crawler checker

Can AI search find your site?

Many sites block AI crawlers without knowing it, often through a setting at their host. See which ones can read yours, and which are for training and which are for answering people's questions.

Free, with no account needed.

What it checks

What you get from the ai crawler checker

13 crawlers
From OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Meta and others.
Training or answers
You can block training and still appear in AI answers, so they're shown separately.
A draft llms.txt
If you don't have one, we draft it from your homepage.

For developers

Three lines from your own code

Every tool is also an API, with one simple answer format. Keep your key on your server.

JavaScript

const res = await fetch("https://www.purlock.com/v1/tools/ai-crawler-checker?target=yourshop.com", {
  headers: { authorization: `Bearer ${process.env.PURLOCK_API_KEY}` },
});
const report = await res.json();

Prompt for Cursor or Lovable

Add a call to the Purlock AI crawler checker API on the server side of this app. Send GET https://www.purlock.com/v1/tools/ai-crawler-checker?target=VALUE with the header "Authorization: Bearer" and the PURLOCK_API_KEY environment variable. The JSON answer has status ("good", "attention" or "bad"), headline, checks (each with level, title, detail, fix) and data. If the call fails or times out after 3 seconds, carry on as if it wasn't there. Never put the API key in client code.

Free plan: 100 calls a month. Individual: 5,000. Business: 25,000.

FAQ

Questions, answered

Should I block AI crawlers?: That's your call. Blocking search and answer crawlers keeps you out of AI answers. Blocking training crawlers doesn't, so many sites allow the first and block the second.
Does robots.txt really stop them?: It's a request. The large companies say they follow it. Some crawlers don't.
What's llms.txt?: An optional plain text summary of your site for AI tools. It's a young idea and not every tool reads it, but it's quick to add.
Can a firewall block crawlers too?: Yes, and this check can't see that. It reads your robots.txt, so a firewall rule can still turn a crawler away.

More free tools

Use it from your own code

Get a free API key with 100 calls a month. No card needed.

Get your API key