Check your sitemap & robots.txt.

robots.txt lives at the root of your domain (/robots.txt) and tells crawlers which paths they're allowed or not allowed to visit.

It uses a simple format: User-agent specifies which crawler, and Disallow / Allow specify paths. A Sitemap: directive tells crawlers where your sitemap lives.

Important: robots.txt is a suggestion, not a security mechanism. Malicious bots can ignore it. Never use it to hide sensitive content — use authentication instead.

A sitemap tells search engines about pages on your site that they might not otherwise discover through normal crawling. It lists URLs, optional last-modified dates, change frequency, and priority.

The spec allows up to 50,000 URLs per sitemap file, with a maximum file size of 50 MB (uncompressed). If you have more, use a sitemap index that references multiple sitemap files.

• Reference your sitemap in robots.txt with a Sitemap: directive.
• Submit it directly in Google Search Console and Bing Webmaster Tools.
• Use lastmod dates so crawlers know which pages changed recently.

1. Disallow: / blocks everything — if this is under User-agent: *, no search engine will crawl your site.
2. No Sitemap: directive in robots.txt — search engines may still find /sitemap.xml, but referencing it explicitly is safer.
3. Stale sitemap — URLs that return 404 waste crawl budget and confuse Google about your site quality.
4. Missing lastmod — without dates, search engines have to re-crawl everything to find changes.
5. Sitemap too large — over 50,000 URLs or 50 MB. Split into a sitemap index.

Manually maintaining sitemaps breaks the moment your CMS adds a new category page or your product catalog changes. Linkraft generates sitemaps dynamically from your URL patterns, so they're always in sync with your actual routes.