Robots.txt and XML Sitemaps: How to Set Up Both Correctly
- Difficulty
- Beginner to moderate
- You need
- Access to your site root or CMS settings
- Test with
- Search Console robots.txt and Sitemaps reports
On this page
A robots.txt file tells crawlers which URLs on your site they may fetch, and an XML sitemap lists the URLs you want search engines to find and index. Robots.txt controls crawling, not indexing, so never use it to hide a page. Put both files at the root of your site, keep the sitemap to canonical URLs, and test each in Search Console. This guide is part of our technical SEO guide.
You'll need access to your site's root folder or your CMS's search settings. Many platforms generate both files for you; in that case, use this guide to check what they produce.
What robots.txt and a sitemap each do
The two files work in opposite directions. Robots.txt says where crawlers shouldn't go. A sitemap says which pages you consider important. Neither forces Google to do anything: robots.txt rules are followed by reputable crawlers, and a sitemap is a hint.
| robots.txt | XML sitemap | |
|---|---|---|
| Purpose | Limits which URLs crawlers fetch | Lists URLs you want found and indexed |
| Location | yourdomain.com/robots.txt only | Usually the root; any URL if submitted |
| Keeps a page out of search? | No: use noindex or a password | No: a listed URL may still not be indexed |
| Format | Plain text, UTF-8 | XML, UTF-8 |
AI crawlers read the same robots.txt file, each under its own user agent. If you want to allow or block them separately, see our guide to robots.txt for AI crawlers.
A minimal robots.txt and sitemap
Here is a simple robots.txt for a site that wants internal search results and cart pages left alone, with the sitemap location at the end:
User-agent: *
Disallow: /search
Disallow: /cart
Sitemap: https://www.example.com/sitemap.xmlAnd here is the start of a matching XML sitemap with one URL entry:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/services/roof-repair</loc>
<lastmod>2026-10-01</lastmod>
</url>
</urlset>Steps
- Decide what, if anything, to block.
List URL patterns that waste crawling and have no value in search, such as internal search results, cart and checkout pages, or endless filter combinations. If nothing qualifies, you may not need any Disallow rules. Don't block pages to hide them: Google can still index a blocked URL if other pages link to it, just without a description. Use noindex or a password for that.
- Write the rules.
Start each group with a User-agent line, where an asterisk means all crawlers. Add Disallow and Allow lines with paths that start with a slash. Paths are case-sensitive, so /Cart and /cart are different. When rules conflict, Google follows the most specific matching rule, and if two match equally, the least restrictive one. Keep the file short and readable.
- Leave CSS, JavaScript and images crawlable.
Google renders pages much like a browser. If robots.txt blocks the files a page needs to display, Google may not understand the page properly. Check that no rule matches your theme, script or image folders. Only block a resource file if the page still makes sense without it.
- Publish robots.txt at the root and test it.
Save the file as plain UTF-8 text named robots.txt and upload it to the top level of the host, so it loads at yourdomain.com/robots.txt. Each subdomain, protocol and port needs its own file. Open the robots.txt report in Search Console to confirm Google fetched it without errors.
- Build the sitemap from canonical, indexable URLs.
Include only full, absolute URLs that return a 200 status, aren't noindexed and are the version you want shown. Leave out redirects, duplicates and parameter versions. A single sitemap can hold up to 50,000 URLs or 50MB uncompressed. Larger sites split it into several files and list them in a sitemap index.
- Use lastmod honestly.
Google ignores the priority and changefreq tags. It uses lastmod only when the dates are consistently accurate. Update lastmod when the main content, structured data or links on a page change significantly, not when a footer date ticks over. Leave it out rather than filling it with the current date on every page.
- Tell Google where the sitemap is.
Submit the sitemap URL in Search Console's Sitemaps report, and add a Sitemap line with the full URL to robots.txt so other crawlers can find it too. Submitting is only a hint: Google may not crawl or index every URL. Our explainer on canonical URLs covers what belongs in it.
- Check the reports and keep both files current.
In the Sitemaps report, look for the status Success. "Has errors" means some entries couldn't be read, and "Couldn't fetch" means Google couldn't load the file. Update the sitemap whenever pages are added, moved or removed, and recheck robots.txt after every redesign or migration.
Both files are among the first things we check in a technical SEO audit. If you'd rather have them set up and maintained for you, our in-house SEO team handles technical SEO, including site structure and metadata.
Rules Google ignores
Some robots.txt and sitemap instructions you'll see online have no effect on Google. Google doesn't support the crawl-delay rule; it sets its own crawl rate from how your server responds, as our explainer on crawl budget explains. It doesn't support noindex inside robots.txt either. To keep a page out of the index, put a noindex meta tag or header on the page itself, and leave the page crawlable so Google can see it.
In sitemaps, Google ignores priority and changefreq. Other search engines may treat these differently, but none of them make a page rank higher.
Troubleshooting
'Indexed, though blocked by robots.txt'
Cause: the page is disallowed in robots.txt, but Google found links to it and indexed the URL anyway, without crawling its content. Robots.txt stops crawling, not indexing. The result often shows in search with no description.
Fix: decide what you want. If the page should appear in search, remove or narrow the Disallow rule so Google can crawl it, then request indexing in URL Inspection. If the page should stay out of search, remove the robots.txt block and add a noindex tag instead. Google has to crawl the page to see the noindex, so the block must go first. Once the page drops out, you can leave it crawlable.
The Sitemaps report says 'Couldn't fetch'
Cause: Google couldn't load the sitemap file at the URL you submitted. Common reasons are a typo in the URL, a sitemap blocked by robots.txt, a login or firewall in front of it, or a server error when Google requested it.
Fix: open the exact submitted URL in a private browser window and confirm it loads as XML without a login. Check that no robots.txt rule matches its path. Use URL Inspection's live test on the sitemap URL to see what Google receives. If it loads, resubmit it in the Sitemaps report. If your CMS generates the sitemap, check that the feature is switched on and the URL hasn't changed after an update.
Crawling stopped after a robots.txt change
Cause: a new rule blocks far more than intended, often a leftover "Disallow: /" from staging or a wildcard pattern that matches every URL. A broken robots.txt file can also cause it: if Google finds the file but can't fetch it because of a server error (5xx), it stops crawling the site for the first 12 hours while it keeps retrying. If it still can't fetch a new version, it uses the last good copy for the next 30 days.
Fix: load yourdomain.com/robots.txt and read it line by line. Remove the blocking rule, or fix the server so the file returns a 200 status. Then open the robots.txt report in Search Console and request a recrawl of the file, since Google generally caches it for up to a day. Inspect a few key URLs to confirm they are crawlable again.
Common questions
Do I need a robots.txt file?
Not always. If Google requests robots.txt and gets a 404, it assumes there are no crawl restrictions and crawls normally. Many small sites run fine without one. You need a robots.txt file when you want to stop crawlers fetching certain URLs, such as endless filter combinations, or when you want a simple place to list your sitemap location for any crawler that reads it.
Should my sitemap list every page on the site?
No. List the canonical URLs you want to appear in search results, and leave out redirects, duplicates, noindexed pages and error pages. Google says a site of about 500 pages or fewer that is well linked internally might not need a sitemap at all. A sitemap still helps new sites with few outside links, large sites, and sites with lots of video or images.
How quickly does Google notice a robots.txt change?
Google generally caches robots.txt for up to 24 hours, and sometimes longer if it can't fetch a fresh copy. So a change may take about a day to apply. If you fixed a rule that blocked important pages by mistake, you can ask Google to recrawl the file from the robots.txt report in Search Console instead of waiting.