Skip to content

Robots.txt — validity

Wygard parses your robots.txt and alerts you when it contains syntax a search engine can't reliably interpret.

Scope Tier Default Alert
Site-wide Basic On — one robots.txt per site Depends on the issue

Why it matters

Crawlers are forgiving parsers — which is exactly the problem. A malformed line doesn't throw an error; it's silently ignored or misread, and a rule you thought was protecting a section simply doesn't apply. A stray directive, two rules crammed on one line, or a Crawl-delay that Google flat-out ignores can leave paths exposed or crawlable in ways you never intended, with no warning from anyone.

The severity depends on the issue Wygard finds — most validity slips are worth reviewing rather than an instant emergency, so they arrive in the daily digest.

What Wygard checks

On every run, the crawler parses the file and confirms:

  1. Only allowed tokens appear — comments (#), Allow, Disallow, User-agent, Crawl-delay[int], Sitemap[url], and blank lines.
  2. Crawl-delay never sits under a User-agent: Googlebot block — Google ignores it entirely, so its presence there signals a misconfiguration.
  3. At most one directive per line.
User-agent: *
Disallow: /cart/
Sitemap: https://example.com/sitemap_index.xml

Common alerts

  • Unknown directive — a token that isn't part of the robots.txt syntax appeared.
  • Crawl-delay under Googlebot — a Crawl-delay was placed in a User-agent: Googlebot block, where it does nothing.
  • Multiple directives on one line — two rules share a single line, which parsers may split incorrectly.

Responding to an alert

  1. Open the alert and review the line Wygard flagged.
  2. Decide whether the syntax was intended (you meant something the parser can't express) or a mistake (a typo, a merged line, a misplaced directive).
  3. Correct the file so each directive is on its own line and every token is valid.
  4. The next crawl re-parses the file and turns the test green.

Pair it with content changes

Validity catches a broken rule; the content changes test catches any edit — including a valid line that quietly disallows a section you wanted crawled. The two together cover both "is it well-formed" and "did someone change it".