Audit your own site the way a crawler does, with curl

Nine checks, a terminal, and about twenty minutes. No tools, no account, no trial. I ran every one of these against my own site today and found six real problems, one of which had been quietly suppressing nine pages.

The reason to do it with curl rather than a crawler product is not price. It is that you see the raw response, which is what a search engine sees, rather than a product's interpretation of it.

1. Does every page canonicalise to itself

This is the one that cost me nine pages. A canonical tag pointing somewhere else says "I am a duplicate, index that instead," and Google does exactly that.

curl -sL https://example.com/some/page | grep -o 'rel="canonical"[^>]*'

If the href is not the URL you just requested, that page has handed its value to another one. Check a page from every section, not just the homepage: the usual cause is a shared layout handing its own canonical to every child route. Long version here.

2. Is anything accidentally noindex

curl -sIL https://example.com/page | grep -i x-robots-tag
curl -sL  https://example.com/page | grep -o '<meta name="robots"[^>]*'

Check the header as well as the tag. An X-Robots-Tag set at the edge or in middleware is invisible in the HTML and overrides everything you can see in the source, and Google treats the two as equivalent directives.

3. Does your sitemap contradict your robots.txt

A URL that appears in the sitemap and is also disallowed in robots.txt is a contradiction, and Google resolves it badly: the page can still surface, as a bare link with no description, because the crawler was never allowed to read it. The sitemap documentation is explicit that a sitemap is a request to crawl, not permission to.

curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*' | sed 's/<loc>//' > urls.txt
curl -s https://example.com/robots.txt | grep -i '^disallow'

Then check each sitemap path against each disallow prefix. I had five.

4. Are any two pages sharing a title

Duplicate titles waste crawl budget and make a search engine choose between your own pages. Mine came from three section layouts handing their title to every child.

while read u; do
  printf '%s\t' "$u"
  curl -sL "$u" | grep -o '<title>[^<]*' | head -1
done < urls.txt | sort -k2 | uniq -f1 -D

5. Does your structured data parse

Not "is it present." Does it parse. A JSON-LD block with a trailing comma is skipped in silence, and the page behaves exactly as if you never wrote it. It is also worth reading what Google says it is actually assessing, because no amount of valid markup substitutes for it.

curl -sL https://example.com/page | python3 -c '
import sys, re, json
h = sys.stdin.read()
for b in re.findall(r"<script[^>]*ld\+json[^>]*>(.*?)</script>", h, re.S):
    try:
        d = json.loads(b)
        print("ok  ", d.get("@type"))
    except Exception as e:
        print("FAIL", e)
'

6. Does your entity have one identity or several

If your site names a person or an organisation in several places, every one of those nodes needs the same @id. Without it you are describing several different people who happen to share a name, on your own domain, which is the exact ambiguity you are presumably trying to remove.

Extract every @type: Person across the site and count the distinct identifiers. Mine was twelve, three of them anonymous. It should be one.

7. Does the title actually contain the thing you want to rank for

Embarrassingly basic and worth checking anyway. The title tag is the strongest on-page signal for a query, and my homepage did not contain my own name, which meant the highest-authority page on my domain could not rank for it.

while read u; do
  t=$(curl -sL "$u" | grep -o '<title>[^<]*' | head -1)
  case "$t" in *"Your Thing"*) : ;; *) echo "MISSING: $u  $t";; esac
done < urls.txt

8. How fast is it, really

curl -sL -o /dev/null -w 'ttfb %{time_starttransfer}s  total %{time_total}s  %{size_download} bytes\n' \
  https://example.com/page

Time to first byte is the number worth watching, because it is the part your hosting and your server code control. If it is under about 200 milliseconds you have a design problem, not a speed problem, and you should go and fix something else.

9. What is actually indexed

Search site:yourdomain.com in a browser, signed out. Count the results and compare that number to your sitemap.

I had 7 pages indexed against 28 in the sitemap. That gap is the single most useful number in this entire list, and it is the one people never look at because it takes ten seconds and tells you something you would rather not know.

Do it in a browser, not through an API

One warning, learned expensively today. A search API is not a window onto Google. It is a separate index with separate ranking, and for a low-volume query the two can disagree completely.

I optimised against an API result set for half a day and named the wrong competitor on three live pages before checking in an actual browser and finding that the person I was distinguishing myself from does not appear in the top twenty at all.

Anything you are about to publish about where you rank, check it in a browser first, signed out, and write down the date.

Common questions

How do I check if a page has the wrong canonical tag?

Fetch the page and print the canonical link: curl -sL https://example.com/page | grep -o 'rel="canonical"[^>]*'. If the href is not the URL you requested, that page is telling search engines to index a different one instead. Check a page from every section, because the usual cause is a shared layout handing its canonical to every nested route.

Why are my pages not showing up in Google even though they are in my sitemap?

Being in the sitemap only asks for a crawl. Common causes of pages still not appearing: a canonical tag pointing at another page, a noindex directive in an X-Robots-Tag header rather than in the HTML, a robots.txt rule blocking the URL you also listed in the sitemap, or duplicate titles causing the engine to choose between your own pages.

What is the difference between a noindex tag and a canonical tag?

A noindex directive says do not index this page at all. A canonical tag says this page is a duplicate of another one, so index that other one and give it the credit. The second is easier to set by accident, because it usually looks correct in the file where it is written and only misbehaves through inheritance.

Can I check my SEO without paying for a tool?

Yes, for most of the technical checks. curl plus grep will tell you about canonicals, robots directives, titles, structured data validity and response times. A paid crawler saves time at scale, but every check in this article runs from a terminal with no account.

Why does a search API give different results from Google in a browser?

A search API is a separate index with its own ranking, not a view into Google's results. For low-volume queries the two can disagree entirely. Verify any claim about where something ranks in a real browser, signed out, and record the date you checked.