Skip to content

Robots.txt — content changes

Wygard keeps a copy of your robots.txt and alerts you on any textual change between runs, line by line.

Scope Tier Default Alert
Site-wide Basic On — one robots.txt per site Depends on the issue

Why it matters

robots.txt is small, powerful, and edited by hand or by plugins far more often than anyone tracks. A single line — Disallow: / slipped in by a staging config, a Sitemap: line removed by an update, an Allow rule quietly deleted — can reshape how your entire site is crawled. The other robots tests catch specific failure modes; this one catches the thing they can't anticipate: someone changed the file and didn't tell you.

Because the impact of a change ranges from harmless to catastrophic, the severity depends on the issue — but you always see that the file moved.

What Wygard checks

On every run, the crawler:

  1. Fetches /robots.txt.
  2. Compares its full content against the copy stored from the previous run.
  3. Fires an alert on any textual difference and highlights the added and removed lines.

The first crawl stores the file as the baseline; every later run is diffed against it until you accept a new version.

Common alerts

  • robots.txt content changed — the file differs from the baseline; the alert shows exactly which lines were added or removed.
  • Disallow rule added or removed — a path's crawlability changed.
  • Sitemap directive changed — a Sitemap: line was edited, added, or dropped.

Responding to an alert

  1. Open the alert and read the diff — Wygard highlights every added and removed line, so you can see the change at a glance.
  2. Decide whether it was intended (you or a teammate edited the file on purpose) or accidental (a plugin update, a staging config that leaked to production).
  3. If intended, adopt the new file as the baseline from the alert card.
  4. If accidental, restore the previous content and let the next crawl turn the test green.

Pair it with validity

Content changes tells you that the file changed; the validity test tells you whether the new version is still well-formed. Together they catch both an unexpected edit and a broken one.