XML Sitemap and Robots.txt in WordPress: Beginner Guide

An XML sitemap and robots.txt WordPress setup helps search engines discover the URLs you consider important while controlling which parts of the site crawlers may request. These files serve different purposes: a sitemap provides preferred URL discovery signals, while robots.txt manages crawler access.

Neither file can guarantee crawling, indexing, or rankings. A technically correct sitemap still needs useful canonical pages, crawlable internal links, stable responses, and appropriate indexing directives. A robots.txt file should not be used as a privacy system or as the primary method for removing normal web pages from Google Search.

This beginner guide explains how WordPress generates sitemaps and robots.txt, how to inspect the files, what to include or exclude, how to submit a sitemap in Google Search Console, and how to diagnose common conflicts. For the wider technical framework, start with Technical SEO for WordPress.

Quick Answer

For a normal WordPress blog, use one working sitemap system, include only preferred public and indexable URLs, submit the sitemap index in Google Search Console, and reference it in robots.txt when appropriate. Keep robots.txt at the root of the host, allow crawlers to access public pages and the resources required to render them, and use page-level noindex directives or authentication when content should not appear in search. Test both files after migrations, plugin changes, redesigns, and visibility-setting updates.

Key Takeaways

• A sitemap supports discovery; it does not force indexing.
• Robots.txt controls crawling; it is not a reliable method for hiding a web page from search results.
• WordPress core provides a sitemap index at /wp-sitemap.xml unless another system replaces or disables it.
• An SEO plugin may generate a different sitemap index, such as /sitemap_index.xml.
• Use one intentional sitemap system instead of submitting duplicate systems without a reason.
• List canonical, indexable URLs rather than redirects, errors, noindex pages, and duplicate variants.
• Do not block a page in robots.txt when Google needs to crawl it to see a noindex directive.
• Private content needs authentication or another real access control.

XML Sitemap vs Robots.txt

Google describes a sitemap as a file containing information about pages and other files and the relationships between them. It describes robots.txt as a file that tells crawlers which URLs they can request. Review Google’s sitemap overview and robots.txt introduction.

How XML Sitemaps Work in WordPress

WordPress added core XML sitemap functionality in version 5.5. A standard current installation can expose the main sitemap index at:

https://example.com/wp-sitemap.xml

The index links to smaller sitemap files for supported public post types, taxonomies, users, and other providers. WordPress core originally set a default limit of 2,000 entries per individual sitemap, with the system creating additional sitemap pages when needed.

The official WordPress core announcement explains that the feature is extensible and that plugins can add, remove, replace, or customize sitemap providers. Review the WordPress core sitemap documentation.

WordPress core sitemap vs SEO plugin sitemap

An SEO plugin may replace the native sitemap with its own index. For example, the active sitemap may use a location such as:

https://example.com/sitemap_index.xml

The exact URL depends on the active system. Open the sitemap in a browser and verify what the site currently serves rather than assuming a specific location.

• Use the sitemap generated by the system that controls the site’s intended indexation settings.
• Avoid submitting both a native and plugin sitemap when they contain competing or duplicate URL sets.
• Confirm that disabling or replacing an SEO plugin does not remove the only submitted sitemap unexpectedly.
• Record the active sitemap URL before migrations and plugin changes.

How to Find the WordPress Sitemap

1. Open the website homepage and confirm the preferred HTTPS domain.
2. Try the native WordPress location: /wp-sitemap.xml.
3. Check the sitemap setting in the active SEO plugin.
4. Open /robots.txt and look for a Sitemap: line.
5. Review the submitted sitemap in Google Search Console.
6. Check the website source or plugin documentation when the location remains unclear.

During an XML sitemap and robots.txt WordPress review, a working sitemap should return a normal success response and valid XML. Its child sitemap URLs should also open without authentication, redirect loops, server errors, or HTML error pages.

What Should an XML Sitemap Include?

Google recommends including fully qualified absolute URLs that represent the canonical versions you want to show in search results.

Usually include

• Published canonical blog posts
• Published canonical pages
• Useful category or taxonomy pages intended for search
• Public custom post types that provide distinct value
• Author archives when they are useful and intended for indexing
• Image, video, or news information when the sitemap system supports it correctly

Usually exclude

• Redirecting URLs
• 404 and 410 URLs
• Pages carrying a noindex directive
• Private, password-protected, or draft content
• Internal search-result URLs
• Tracking and session parameters
• Duplicate HTTP, WWW, print, feed, or filtered variants
• Thin taxonomies and archives with no search purpose
• Staging and development URLs

The sitemap should support the same preferred URL as the canonical tag, internal links, redirects, and HTTPS configuration. Conflicting signals make diagnosis more difficult.

XML Sitemap Best Practices

Use absolute canonical URLs

Use complete URLs such as:

https://example.com/example-post/

Do not list a relative path such as /example-post/. Google states that sitemap URLs should be fully qualified and that it attempts to crawl the URLs exactly as listed.

Keep the preferred host and protocol consistent

• Do not mix HTTP and HTTPS.
• Do not mix WWW and non-WWW unless the site intentionally uses both as separate hosts.
• Use the same trailing-slash convention as the canonical URLs.
• Remove URLs copied from staging or preview environments.

Use accurate modification dates

A lastmod value should represent a meaningful page change, not the time a sitemap is generated. Avoid updating modification dates across the entire site without material changes.

Stay within sitemap limits

Google currently supports a maximum of 50,000 URLs or 50 MB uncompressed for one sitemap file. Larger collections should be divided into multiple files and can be organized through a sitemap index. WordPress and SEO plugins normally handle this division automatically.

Keep important pages internally linked

A sitemap should supplement navigation, not replace it. Google can often discover properly linked pages by following menus and contextual links. Important pages should not remain orphaned with only a sitemap entry.

How to Submit a WordPress Sitemap in Search Console

1. Open the correct Search Console property.
2. Select Sitemaps under Indexing.
3. Enter the active sitemap index path.
4. Submit the file.
5. Check whether the status becomes successful.
6. Review the last-read date and discovered URL count.
7. Investigate fetch and parsing errors.
8. Monitor Page indexing separately.

Submitting a sitemap is a discovery hint. Google does not guarantee that it will download, crawl, or index every submitted URL. Use the Google Search Console for Bloggers guide for property verification and report workflows.

Common XML Sitemap Problems

Sitemap shows a blank page or error

• Clear WordPress, server, and CDN caches.
• Resave Permalinks only after recording the current structure.
• Check for a plugin or theme conflict.
• Review PHP and server logs.
• Confirm the active sitemap provider.
• Ask the hosting provider when the error is server-side.

Sitemap contains unwanted content

Review the post type, taxonomy, author, and archive settings in the active sitemap system. Do not exclude a content type only because it is unfamiliar; first decide whether its URLs provide useful distinct pages.

How Robots.txt Works in WordPress

A robots.txt file must be available at the root of the relevant host, for example:

https://example.com/robots.txt

In an XML sitemap and robots.txt WordPress setup, WordPress can generate a virtual robots.txt response when no physical file overrides it. WordPress also exposes the robots_txt filter for developers to modify the output. The correct implementation depends on the server, host, plugins, and whether the website uses a physical file.

Review the official WordPress robots_txt hook documentation before adding custom code.

Basic Robots.txt Syntax

User-agent: *
Disallow:

Sitemap: https://example.com/wp-sitemap.xml

This example does not disallow a path for crawlers matched by User-agent: * and provides the sitemap location. The empty Disallow: line is not required when no restriction is needed, but it makes the example’s intent clear.

User-agent

The user-agent line identifies which crawler the group applies to.

User-agent: *

The asterisk represents crawlers that do not have a more specific matching group under the applicable protocol rules.

Disallow

The Disallow line identifies paths that the matched crawler should not request.

Disallow: /example-private-path/

Do not use robots.txt to protect confidential information. A public URL can still be discovered through links, and not all crawlers are required to obey the file. Use authentication for private content.

Allow

An Allow rule can permit a more specific path inside a broader disallowed path when supported by the crawler.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Do not copy a rule merely because another WordPress site uses it. Check what the path does on your own website and whether important resources depend on it.

Sitemap

A robots.txt file can contain one or more absolute sitemap references:

Sitemap: https://example.com/sitemap_index.xml

Google states that multiple sitemap lines are supported. Use the actual preferred sitemap files rather than old, redirected, or staging locations.

Robots.txt vs Noindex vs Password Protection

Google explains that robots.txt is mainly for managing crawler traffic and is not a mechanism for keeping an ordinary web page out of Google. A blocked URL can still appear without a normal snippet if Google discovers it through other sources.

A noindex directive can be read only when the crawler is allowed to access the page. Review the official robots meta and X-Robots-Tag specifications.

WordPress Robots.txt Setup Checklist

• Open the exact root robots.txt URL.
• Confirm it returns a normal text response.
• Review all user-agent groups.
• Check broad disallow rules.
• Confirm public articles and pages remain crawlable.
• Confirm required CSS, JavaScript, and image resources are not blocked.
• Check for staging or maintenance rules.
• Check whether a physical file overrides WordPress output.
• Check host, CDN, firewall, and security-plugin rules.
• Confirm the sitemap line uses the current HTTPS domain.
• Retest after migrations and plugin changes.

Robots.txt Rules to Review Carefully

Blocking the whole site

User-agent: *
Disallow: /

This rule blocks crawlers covered by the group from requesting the site. It may be appropriate for a temporary test environment but is dangerous on a public production site.

Blocking scripts and styles broadly

Blocking important resources can make rendered pages harder for search engines to understand. Test the live rendered page before restricting resource directories.

Blocking a page that also uses noindex

Google needs to crawl the page to read its noindex instruction. Remove the crawl block when noindex is the intended exclusion method.

Blocking parameter URLs without testing

A broad parameter rule can affect navigation, filtering, pagination, search, scripts, or other useful functionality. Crawl and indexing strategies for large parameter systems require careful testing.

WordPress Search Engine Visibility

WordPress includes a “Discourage search engines from indexing this site” option under Settings → Reading. It is intended as a request and can affect the robots directives WordPress outputs.

• Check it before launching a production site.
• Check it after restoring or cloning the website.
• Check it after moving from staging to production.
• Do not depend on it to protect private content.
• Review the live HTML and robots.txt output after changing it.

Use the WordPress Settings After Installation guide and the Blog Launch Checklist before making the site public.

How to Test the Sitemap and Robots.txt Setup

1. Open the sitemap index in a logged-out browser.
2. Open several child sitemaps.
3. Check sample URLs for correct protocol, host, status, and canonical.
4. Confirm redirects and noindex pages are absent.
5. Open the root robots.txt file.
6. Review every disallow rule and sitemap reference.
7. Inspect representative URLs in Search Console.
8. Run Test Live URL for an important page.
9. Check the Sitemaps report for processing errors.
10. Review Page indexing for unexpected exclusion patterns.

When an important URL remains outside the index, use How to Fix Page Indexing Issues in Google Search Console to diagnose the exact page.

Sitemap and Robots.txt Checks After a Migration

• Replace old domain references.
• Remove staging sitemap URLs.
• Remove staging-wide crawl blocks from production.
• Confirm old URLs redirect to relevant new URLs.
• Confirm the new sitemap contains final canonical URLs.
• Submit the new sitemap in the correct Search Console property.
• Check DNS, HTTPS, CDN, and firewall behavior.
• Review server logs and indexing reports.
• Keep the old property available for comparison when appropriate.
• Record all sitemap and robots changes.

Common XML Sitemap and Robots.txt Mistakes

Submitting two conflicting sitemap systems

A native sitemap and plugin sitemap may contain different content types and indexation decisions. Choose one intentional system and verify its output.

Including noindex pages in the sitemap

This sends inconsistent signals. Remove the page from the sitemap or change the indexation decision when the page belongs in search.

Including redirected and broken URLs

List final canonical pages. Replace old URLs after migrations, merges, and slug changes.

Expecting the sitemap to replace internal links

Important pages should be reachable through normal site navigation and contextual links.

Using robots.txt as security

Robots rules are public and voluntary. Protect sensitive information with authentication and proper server controls.

Blocking the page before Google can read noindex

Allow crawling when the goal is for Google to process a noindex directive.

Copying another website’s robots.txt

Paths, plugins, themes, applications, and crawl requirements differ. Write rules for the actual site.

Editing files without a backup or record

Robots and sitemap changes can affect many URLs. Preserve a recovery point and document the old and new configurations.

XML Sitemap and Robots.txt WordPress Audit

Maintenance Schedule

Integrate these checks into the Blog Maintenance Checklist so changes are reviewed regularly rather than only after traffic declines.

Printable Sitemap and Robots.txt Checklist

• □ Preferred HTTPS domain confirmed
• □ Active sitemap generator identified
• □ Sitemap index opens successfully
• □ Child sitemap files open successfully
• □ Sitemap contains absolute canonical URLs
• □ No HTTP or incorrect host URLs
• □ No redirects in the sitemap
• □ No 404 or 410 URLs in the sitemap
• □ No noindex URLs in the sitemap
• □ No private or draft URLs
• □ Important pages appear in the correct sitemap
• □ Important pages have crawlable internal links
• □ Sitemap submitted in Search Console
• □ Sitemap status reviewed
• □ Root robots.txt opens successfully
• □ User-agent groups reviewed
• □ Broad disallow rules reviewed
• □ Public pages are crawlable
• □ Required CSS and JavaScript are crawlable
• □ Sitemap line uses the current sitemap URL
• □ Staging rules removed from production
• □ Noindex pages are crawlable when required
• □ Private pages use real access control
• □ Search visibility setting reviewed
• □ Sample URLs tested in Search Console
• □ Changes backed up and recorded

Frequently Asked Questions

Does WordPress create an XML sitemap automatically?

Yes. WordPress core has included native sitemap functionality since version 5.5 and normally exposes an index at /wp-sitemap.xml. An SEO plugin can replace or disable that output.

Which WordPress sitemap should I submit?

Submit the active sitemap index that represents the site’s intended canonical and indexable content. Avoid submitting duplicate systems without a clear reason.

Does a sitemap guarantee indexing?

No. It helps discovery and communicates preferred URLs but does not guarantee crawling, indexing, or ranking.

Should noindex pages be in the sitemap?

Normally no. A sitemap should list canonical URLs intended for search. Including a noindex page creates conflicting signals.

Can robots.txt remove a page from Google?

Not reliably. A blocked web-page URL can still be discovered and appear without a normal snippet. Use noindex for an accessible page, authentication for private content, or removal and proper status codes when the content no longer exists.

Where should robots.txt be located?

It should be available at the root of the relevant host, such as https://example.com/robots.txt.

Should I block wp-admin in robots.txt?

WordPress commonly discourages crawling of administrative paths while allowing resources needed by public pages. Review the actual generated file and avoid copying rules without testing the site’s behavior.

Should CSS and JavaScript be blocked?

Do not block resources required to render and understand important pages. Test the page through URL Inspection before restricting resource paths.

Can I list more than one sitemap in robots.txt?

Yes. Google supports multiple sitemap references, but each should be intentional, current, and relevant to the host.

Why is my WordPress sitemap returning 404?

Possible causes include an incorrect sitemap URL, disabled sitemap feature, permalink or rewrite problem, plugin conflict, cache, server error, or a sitemap system that has been replaced. Identify the active provider before troubleshooting.

Conclusion

A dependable XML sitemap and robots.txt WordPress configuration uses one clear sitemap system and a minimal, intentional robots.txt file. The sitemap should list preferred public URLs, while robots.txt should manage crawler access without blocking the pages or resources search engines need to process.

Submit the sitemap in the correct Search Console property, review its processing status, and keep important pages internally linked. Use noindex rather than robots.txt when an accessible HTML page should not appear in search, and use authentication for confidential content.

Return to Technical SEO for WordPress for the parent audit framework, use Google Search Console for Bloggers for submission and monitoring, and follow How to Fix Page Indexing Issues when an important URL remains excluded.