SEO basics

How to fix: robots.txt and XML sitemap

robots.txt tells crawlers which parts of your site not to fetch. An XML sitemap lists the URLs you want search engines to know about.

Not sure whether your site has this problem? Check it in seconds.

Run a free audit

Why it matters

A sitemap helps search engines discover pages, especially new or poorly linked ones. robots.txt keeps crawlers out of areas you don't want fetched, such as admin pages. Neither guarantees a page will be indexed.

How to check

Open https://yourdomain.com/robots.txt and https://yourdomain.com/sitemap.xml. Submit the sitemap in Google Search Console to see which URLs were read and indexed.

How to fix it

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://www.example.com/sitemap.xml

sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-01-15</lastmod>
  </url>
</urlset>

Next.js

// app/robots.ts
export default function robots() {
  return { rules: { userAgent: '*', disallow: ['/admin/', '/api/'] }, sitemap: 'https://www.example.com/sitemap.xml' };
}
// app/sitemap.ts
export default function sitemap() {
  return [{ url: 'https://www.example.com/', lastModified: new Date() }];
}

Common mistakes

Using robots.txt to hide a page from search results. A blocked URL can still be indexed if other sites link to it. To keep a page out of results, allow crawling and use a noindex robots meta tag instead.
Accidentally shipping Disallow: / from a staging environment to production.
Listing redirecting, noindex or error URLs in the sitemap.

Frequently asked questions

Does having a sitemap improve rankings?

Not directly. It helps discovery; ranking depends on content, links and many other signals.

Related guides

Guidance is general. Test header and DNS changes on a staging copy first, and check your hosting provider's documentation for platform-specific steps.