Skip to content

Meta robots

Wygard watches the content of <meta name="robots"> on every monitored URL and alerts you the moment it changes.

Scope Tier Default Alert
Per URL Basic Enabled on every URL where a meta robots tag is found 🔴 Danger

Why it matters

The meta robots tag controls whether a page can be indexed (index / noindex) and whether its links pass equity (follow / nofollow). A single accidental noindex — typically copied from staging to production, or set by a CMS plugin during a routine update — drops the page out of Google within days. The site still loads, internal links still work, and there is no visible symptom until organic traffic collapses.

The default severity is Danger because nothing else on this list has a faster path from "everything looks fine" to "the page is gone from search."

What Wygard checks

On every run, the crawler:

  1. Fetches the monitored URL.
  2. Extracts the content attribute of <meta name="robots"> from the page source.
  3. Compares the found value against the stored desired value.
<meta name="robots" content="index,follow">

The first crawl's value becomes the baseline. Any deviation — a new directive added, an existing one flipped, the tag removed entirely — triggers an alert.

Common alerts

  • Meta robots value changed — e.g. index,follow flipped to noindex,nofollow or noindex was appended.
  • Meta robots not found — the tag was removed from the page (often a regression after a template refactor).
  • Multiple meta robots tags — more than one tag was detected. Search engines combine them in restrictive ways that are easy to misjudge.

Why the default is Danger

noindex on a money page deindexes it within a few crawls — usually a matter of days, sometimes hours on high-priority URLs. The Danger severity ensures you see the alert immediately after the batch finishes, not at the next daily digest.

Responding to an alert

  1. Open the alert and review the diff — Wygard shows the previous and the new directive string.
  2. If the new value contains noindex on a page that should be indexed, treat it as a production incident. Roll back the change at its source (template, plugin, CMS setting) rather than just toggling it back in the UI.
  3. If the change was intended (e.g. you really do want a page noindexed), click Set found value as desired to adopt it as the new baseline.
  4. Re-run the test from the alert card to confirm the fix is live.

Pair it with Status code

Meta robots and HTTP status codes are the two main mechanisms a page can silently fall out of search. The Status code test catches the other half — a page returning 404 or 5xx instead of 200, or a redirect chain pointing somewhere unexpected.