Sitemap Strategy for Thousands of Pages
A sitemap strategy for thousands of pages splits URLs into segmented XML sitemaps under a sitemap index, keeps lastmod accurate, and uses per-sitemap indexing reports to diagnose discovery problems.
Respect the limits and use a sitemap index
A single XML sitemap can hold up to 50,000 URLs and must stay under the uncompressed size limit, so any large site needs multiple sitemap files referenced by a sitemap index. The index is a small file that lists all your sitemaps, and it is what you submit to Search Console and reference in robots.txt. This structure is mandatory once you cross the per-file limit and is good practice well before that.
Include only canonical, indexable, 200-status URLs in your sitemaps. Do not list redirects, noindexed pages, canonicalized duplicates, or blocked URLs, because a sitemap full of non-indexable URLs wastes crawl attention and muddies the signal. The sitemap is a statement of the pages you want indexed, so keep it clean and let your generation pipeline exclude anything that should not be there.
Segment sitemaps to diagnose problems
Split your sitemaps along meaningful lines, such as by page type, category, or template, rather than arbitrarily. When each sitemap corresponds to a logical section, Search Console's per-sitemap indexing counts become a diagnostic tool: you can see that one template is indexing at ninety percent while another sits at thirty percent and act on that specific section.
This segmentation turns a large opaque site into a set of measurable cohorts. If a whole segment indexes poorly, you know to look at that template's content quality, internal linking, or crawl waste rather than searching the entire site. Keep the segmentation stable over time so you can track each section's indexation trend across releases.
Keep lastmod accurate and generate dynamically
Set the lastmod value to the genuine date a page's content last changed, and make sure it is accurate, because search engines use it to prioritize recrawling and will discount it if it is unreliable. Do not stamp every URL with today's date on every build; that teaches crawlers to ignore the field entirely.
Generate sitemaps automatically from the same data source that builds your pages, so new URLs appear and removed URLs drop off without manual work. For a site that publishes continuously, a dynamic sitemap that regenerates on each build is the only way to keep discovery current across thousands of frequently changing pages.
Submit, monitor, and iterate
Submit the sitemap index in Search Console and reference it in robots.txt so crawlers find it independently. Then treat the indexing reports as an ongoing monitor rather than a launch-day check: watch discovered-versus-indexed counts per sitemap and investigate any segment where the gap widens.
When a segment shows many discovered-but-not-indexed URLs, resubmitting the sitemap will not fix it; the cause is usually content quality, duplication, or crawl waste in that section. Use the sitemap segmentation to localize the problem, fix the underlying issue, and confirm the indexed count recovers on subsequent crawls.
Frequently asked questions
How many URLs can one sitemap contain?
A single XML sitemap can list up to 50,000 URLs and must stay under the uncompressed size limit. Beyond that, use multiple sitemaps referenced by a sitemap index file.
Should I split sitemaps by page type?
Yes. Segmenting sitemaps by page type or template turns Search Console's per-sitemap indexing counts into a diagnostic, letting you see which sections index well and which need attention.
Does submitting a sitemap guarantee indexing?
No. A sitemap aids discovery but does not force indexing. Pages must still be crawlable, unique, and useful. If URLs are discovered but not indexed, address content quality and crawl waste.
Keep reading
Want this machine pointed at your site?
This page is one of a swarm we generated for ourselves. We build the same for you.
Get Started - $10,000