Robots.txt and XML Sitemaps Explained: A Beginner's Guide to Controlling How Google Crawls Your Site
A developer launches a beautiful new website for a coaching institute. Weeks pass, and the site does not appear in Google at all, even for its own name. After much confusion, someone opens a small text file on the server and finds a single line telling every search engine to stay away. It had been added on the staging site to keep it private during development and was never removed when the site went live.
That small file is robots.txt, and together with the XML sitemap, it plays an important role in how search engines discover and crawl your website. Neither file is complicated, but mistakes in either can quietly undermine your SEO.
What robots.txt does
Robots.txt is a plain text file placed at the root of your domain, for example yoursite.com/robots.txt. It gives instructions to search engine crawlers about which parts of your site they may or may not crawl. A typical file contains rules naming a crawler, or all crawlers, followed by paths that are disallowed or allowed. It can also include a line pointing to your XML sitemap.
The file is useful for preventing crawlers from wasting time on areas that offer no value in search, such as admin sections, internal search results, shopping cart and checkout pages, or endless filtered URL combinations on large e-commerce sites. On very large websites, steering crawlers away from low-value URLs helps them focus on important pages.
What robots.txt does not do
This is where many businesses go wrong. Robots.txt controls crawling, not indexing. If a page is blocked in robots.txt but other websites link to it, Google may still index the URL, sometimes showing it in results without a description. To keep a page out of search results, the correct approach is usually a noindex tag on the page, which requires that Google can crawl the page to see the tag. Blocking a page in robots.txt and adding noindex at the same time can therefore backfire, because Google cannot read the noindex instruction on a page it is not allowed to crawl.
Robots.txt is also not a security tool. The file is public, and anyone can read it. Sensitive areas should be protected with passwords and proper access controls, not hidden behind a robots.txt rule that effectively advertises where they are.
Common robots.txt mistakes
The most damaging mistake is the one in the opening story: a rule disallowing the entire site, left over from development. Another common error is blocking CSS and JavaScript files, which prevents Google from rendering pages properly and understanding their layout and content. Blocking important sections by accident, for example by using a pattern that matches more URLs than intended, can remove key pages from crawling. Some sites also block resources from third-party plugins without realising those resources are needed to display content.
Always test changes. Google Search Console provides a robots.txt report that shows the file Google has fetched and highlights problems, and the URL Inspection tool reveals whether a specific page is blocked.
What an XML sitemap does
An XML sitemap is a file that lists the URLs on your website you want search engines to know about, often with additional information such as when each page was last updated. It acts as a guide, helping search engines discover pages, particularly new pages, pages deep within the site structure or pages with few internal links.
Most modern website platforms, including WordPress with common SEO plugins, Shopify and many custom systems, generate sitemaps automatically. Large sites may use a sitemap index file that links to several smaller sitemaps, for example one for blog posts, one for products and one for service pages.
Sitemap best practices
Include only pages you genuinely want indexed: canonical URLs that return a successful status and are not set to noindex. Remove redirected URLs, broken pages, duplicate versions and thin pages that offer little value. A clean sitemap sends a consistent signal about which pages matter, while a sitemap full of errors and duplicates sends a confusing one.
Keep the sitemap updated automatically as you add or remove pages. Submit it in Google Search Console and Bing Webmaster Tools, and reference it in your robots.txt file. Then check the Sitemaps report in Search Console periodically to see how many submitted URLs are indexed, which can reveal quality or technical problems.
Remember that a sitemap does not guarantee indexing. It helps discovery, but Google still decides whether pages are valuable enough to index. Strong content and good internal linking remain essential.
A quick health check
Open yoursite.com/robots.txt and confirm it does not block your whole site or important sections. Check that your sitemap loads, lists your key pages and excludes redirects and duplicates. Confirm both are submitted and error-free in Search Console. Review them after any redesign, migration or platform change, because those are the moments when mistakes most often creep in.
Frequently asked questions
Does every website need a robots.txt file?
It is not mandatory, but having a simple, correct file is good practice and helps point crawlers to your sitemap.
How often should I update my sitemap?
Ideally it updates automatically whenever pages are added, changed or removed.
Can robots.txt remove a page from Google?
Not reliably. Use a noindex tag, removal tools or proper redirects instead.
Make sure search engines can find your best pages
OM Enterprises audits robots.txt files, sitemaps and crawlability as part of technical SEO reviews, fixing issues that quietly hold websites back. Request a technical SEO audit.