Skip to content

Robots.txt — status code

Wygard fetches your /robots.txt and alerts you the moment it stops returning HTTP 200.

Scope Tier Default Alert
Site-wide Basic On — one robots.txt per site 🔴 Danger

Why it matters

robots.txt is the first file a crawler asks for. It's where you allow and disallow paths and point search engines at your sitemaps. If it stops returning 200 — a 404 after a deploy, a 5xx from an overloaded server — search engines don't fall back gracefully. A 5xx in particular can make Google treat the whole site as temporarily off-limits and pause crawling until the file recovers.

Because the site itself keeps loading and nothing visible breaks, this failure is easy to miss. The default severity is Danger: the alert fires immediately after the crawl batch.

What Wygard checks

On every run, the crawler:

  1. Requests /robots.txt at your domain root.
  2. Records the HTTP status code it returns.
  3. Flags anything that isn't 200.

This test is one of five that make up the Robots.txt group — together they guard the file's availability, size, syntax, and content.

Common alerts

  • robots.txt returned a non-200 status — the file now returns 404 (removed or route broken) or 5xx (server error, timeout).
  • robots.txt unreachable — the request failed entirely (DNS, TLS, or connection error at the root).

Why the default is Danger

A 5xx on robots.txt can make Googlebot back off crawling the entire site until the file responds again. That's a site-wide crawl slowdown with no on-page symptom — Danger severity surfaces it within the hour instead of at the next digest.

Responding to an alert

  1. Open the alert and check the status the file now returns.
  2. Decide whether it's intended (rare — you almost never want robots.txt to stop responding) or accidental (a deploy that 404s the route, a server error).
  3. Fix the source so /robots.txt returns 200 again — restore the file, correct the route, or resolve the server error.
  4. The next crawl turns the test green automatically.

Pair it with the rest of the Robots.txt group

A 200 only means the file responds. The validity, max file size, live sitemap included, and content changes tests confirm that what's inside the file is correct and hasn't drifted.