You were told to let the AI crawlers in. Here is where that permission really lives, and a one-minute way to check it.

You were probably told something like: make sure the AI crawlers can reach your site. Fair enough. But almost nobody says where that permission actually lives, and it does not live in your pages. It lives in one small text file, and more quietly in a line of the server's answer that most people never look at. I run two sites of my own, so I read both places on them this week and wrote down exactly what came back. You can check yours in about a minute, and you do not need to pay for a tool to do it.

What the one small file is

It is called robots.txt, and it sits right after your web address, as yoursite.com/robots.txt. It is the first thing a well-behaved crawler reads before it asks for any page. Think of it as a notice on the door, not a lock on the door. It says which parts of a site a visitor is welcome to read and which parts to leave alone. If your host never made one, opening that address simply shows a not-found page — and a missing notice means everything is welcome by default, which is usually the state you want to be in.

What our own two files say

Our service site keeps this very short. Its whole file is two lines and a pointer: a line that says every crawler is being addressed, and a line that says every part of the site is allowed, plus a line naming where the sitemap lives. It does not name a single crawler by itself. It never mentions GPTBot, or Google-Extended, or PerplexityBot, or the OpenAI search crawler. It does not have to. Because the every-crawler line already says all are welcome everywhere on the site, all four of them are covered by it. There is no refusal line in the file at all. Nothing is turned away.

Our second domain does the same job the long way. It also allows every crawler everywhere, then adds one line that keeps only the back-office folder private, and after that it lists each crawler by name with its own allow line — GPTBot, the OpenAI search crawler, PerplexityBot, Google-Extended and roughly thirty others. Same end result: all four of the ones people keep telling you to let in are allowed on both of our sites, and none of them is refused. The only difference is style. One file trusts the blanket line; the other spells out each name. Both give the answer a crawler will actually use.

The line almost nobody looks at

The second place this permission hides is not a file at all — it is part of the server's answer, and you only see it in the response headers, not in the page you can read with view-source. It is a tag some platforms quietly send for every page at once. Here is what we measured: on both of our domains, none of the pages we requested came back carrying that tag. So nothing at the server layer is telling anyone to leave our pages out. What the two domains do send are the quiet safety lines — the ones that tighten the connection and stop our pages being framed inside somebody else's site. Our service site sends the fuller set; our second domain sends fewer and is missing some of them. That difference is real, but it is about protection, not about permission. Whether a crawler may read a page is decided by the door notice and by that hidden tag — and on our sites both say yes.

Why this has to be two answers, not one

Because the same question can pull two different ways. A crawler can be warmly welcomed by your door notice and still be shut out by a hidden tag you never knew existed — and the reverse happens too. That is precisely why reading only your pages tells you nothing you can trust. I have watched people add a line to that small file, feel they had solved it, and change nothing that a crawler actually sees. Two sites, two styles, one verdict each — that is the whole lesson, and it cost us nothing but a few reads to learn it.

Check yours in about a minute

No paid tool, no account, just two addresses and one response you can already see in your browser.

1. Type yoursite.com/robots.txt into a browser tab and open it. If it shows words, read them. If it shows a not-found page, there is no file and everything is welcome by default.
2. Look on those lines for any word that means keep out. A keep-out on the whole site, sitting next to a crawler's name, means that crawler is not welcome anywhere. A keep-out on one folder only leaves that folder out.
3. Now the invisible half: open your browser's developer tools, reload one real page, click the very first request, and read the response headers. Look for a line that carries the hidden robots tag. If it says leave-this-page-out, your page can be served perfectly and still never be stored.
4. Do that on both your homepage and one page you truly want found — a product page, a service page, a post — because a single template can behave differently from the rest of the site.

If you would rather see that answer as plain text than dig through a tools panel, one command asks the server for just its response and nothing else, and you can look for the tag line directly. Point it only at addresses you control.

curl -I https://your-site.com/

This prints only the server's reply. Scan the lines for one that names the hidden robots tag. Read your homepage and one page you want found.

Where these numbers come from

Everything above is a measurement I took on my own two sites this week, from anonymous reads, and I am publishing the plain results rather than pretending to know your specific site. The same fixed method — reading only public pages and their responses, the small file first, a short pause between requests to one host, and no scripts run at all — is the method behind the written report I sell for up to twenty-five agreed pages of a single live site. And the reason that last sentence is careful is the honesty rule: where a thing could not be read from where I stood, we write not checked instead of guessing. If, after the minute above, your two answers still do not match what you meant them to say, that gap is exactly what a scoped read-through turns into a numbered list you can act on today. It is a one-time, USD 99 written report on one site. You can see it, and the full checklist behind it, at https://svc.openyuri.com/ .