Skip to content

Robots.txt — live sitemap included

Wygard checks that every sitemap your robots.txt declares actually resolves, and alerts you when a Sitemap: line points at a dead URL.

Scope Tier Default Alert
Site-wide Basic On — one robots.txt per site Depends on the issue

Why it matters

The Sitemap: directive in robots.txt is one of the primary ways search engines discover your sitemap. If that line points at a URL that no longer resolves — a path that changed on deploy, a generator that moved its output — the reference is broken even though robots.txt itself is perfectly healthy. Google is left without the map it was pointed to, and discovery of new and updated pages slows down with no visible symptom.

What Wygard checks

On every run, the crawler:

  1. Parses robots.txt and collects every Sitemap: directive.
  2. Requests each declared sitemap URL.
  3. Confirms it resolves (returns a live 200, not a 404/5xx).
Sitemap: https://example.com/sitemap_index.xml

Common alerts

  • Declared sitemap is unreachable — a Sitemap: line points at a URL that returns 404 or 5xx.
  • Sitemap path changed — the file moved, but the old URL is still declared in robots.txt.

Responding to an alert

  1. Open the alert and check which declared sitemap URL failed.
  2. Decide whether it's intended (you retired that sitemap) or accidental (the path changed, the generator broke).
  3. If accidental, restore the sitemap or update the Sitemap: line to the new, live URL.
  4. If the sitemap is genuinely gone, remove its Sitemap: line. The next crawl turns the test green.

Pair it with the SitemapGroup tests

This test only confirms the sitemap URL declared in robots.txt resolves. The SitemapGroup — status code, Size & Limit, and validity & format tests then check that the sitemap and its children are healthy inside.