Skip to content

Robots.txt — max file size

Wygard measures your robots.txt and alerts you if it exceeds Google's 500 KB limit.

Scope Tier Default Alert
Site-wide Basic On — one robots.txt per site 🔴 Danger

Why it matters

Google reads only the first 500 KB of a robots.txt file. Anything past that cap is ignored entirely — including Disallow rules and Sitemap: lines that live at the bottom of an oversized file. A bloated robots.txt (usually from a generator appending rules unchecked) can therefore silently drop the exact protections or sitemap references you were counting on, while the file still returns a healthy 200.

Because the truncation is invisible — no error, no warning, the file loads fine — the default severity is Danger.

What Wygard checks

On every run, the crawler:

  1. Fetches /robots.txt.
  2. Measures its total size.
  3. Flags the file if it exceeds 500 KB.

Common alerts

  • robots.txt exceeds 500 KB — the file has grown past the cap, so any rules or sitemap lines beyond the limit are being ignored by Google.

Why the default is Danger

Once the file passes 500 KB, Google stops reading — and the rules most likely to be cut are the ones appended last. A protection you added recently could already be silently inactive. Danger severity surfaces the overflow immediately.

Responding to an alert

  1. Open the alert and check the reported file size.
  2. Trim the file below 500 KB — consolidate repetitive Disallow patterns, remove obsolete rules, and rely on wildcards (*, $) instead of listing paths individually.
  3. Confirm the rules and Sitemap: lines you depend on sit within the first 500 KB.
  4. The next crawl re-measures the file and turns the test green.

Pair it with validity

A large file is often a sign of a generator gone wrong. The validity test catches the malformed lines such a generator tends to produce, while this test catches the sheer bulk that pushes real rules past Google's cutoff.