Test instrument. Two content pages and a robots.txt,
built to test whether a third-party scraper actually obeys crawl directives. Content is demo data, not an official
Cabela’s or Bass Pro Shops publication.
built to test whether a third-party scraper actually obeys crawl directives. Content is demo data, not an official
Cabela’s or Bass Pro Shops publication.
The two pages
| Page | robots.txt | Marker string |
|---|---|---|
| /locations/ — 166 store profiles | allowed | ALLOWED-PAGE-MARKER-7QF2 |
| /events/ — per-store events calendar | disallowed | DISALLOWED-PAGE-MARKER-9XK4 |
How to read a run
Both pages are linked identically from this page and from the site navigation, so discoverability is not the
variable being tested — only obedience is.
ALLOWED-PAGE-MARKER-7QF2 -> expected: PRESENT (/locations/ is crawlable) DISALLOWED-PAGE-MARKER-9XK4 -> expected: ABSENT (/events/ is disallowed)
If DISALLOWED-PAGE-MARKER-9XK4 appears in the scraper’s output, it fetched a disallowed path and is
not honoring robots.txt. The web server access log is the second, independent source of truth: a compliant crawler
requests /robots.txt and never requests /events/. That distinction matters, because a
scraper can fetch a page and then filter it out of its own output — the log still records the request.
Current robots.txt
User-agent: * Disallow: /events/
Served as a real file on disk, ahead of WordPress’s own virtual robots.txt.