# robots.txt

URL: https://xodia.ai/glossary/robots-txt
Description: What robots.txt is and how Xodia writes it on every build: switches for search engines, AI training and AI search bots, your own rules and the sitemap line.
Last updated: 2026-11-10

robots.txt is a plain text file at a site's root, /robots.txt, that tells crawlers which parts of the site they may read and where its sitemap is.

How Xodia writes robots.txt: Search engines settings, free on every plan, make it on every build from three switches, for search engines, AI training bots and AI search bots, plus your own rules and the sitemap's address.

Updated November 10, 2026

## What it is

A crawler that follows the rules reads robots.txt before anything else. Each group starts with User-agent, naming one crawler or all of them with *, and lists the paths it may not read with Disallow, or may read with Allow. A Sitemap line points crawlers at the site's sitemap.xml.

robots.txt asks; it doesn't lock anything. A page that is blocked can still show in search results if other sites link to it, so a page that must stay out of search needs a noindex tag instead, and private pages need a password.

## In Xodia

Project settings, Search engines writes robots.txt for your site on every build. With search engines on, every crawler may read the site. AI training bots and AI search bots each have their own switch, and turning one off adds the rules that ask those bots to stay away. List the sitemap in robots.txt adds the Sitemap line, and Your own rules are added as written.

A site that already has its own robots.txt keeps it, and the card lets you edit it in place or hand it over to Xodia. On Xodia hosting, the Preview and publish previews always ask every crawler to stay away.

## Read more

- [SEO for React sites: Search engines settings](https://xodia.ai/product/seo#search-engines)
- [Docs: robots.txt settings](https://docs.xodia.ai/seo/search-engines#robots)

## Related terms

- [Sitemap A sitemap is a map of a website's pages: for planning, a tree of pages and how they nest; for search engines, an XML file that lists every page a crawler should find.](https://xodia.ai/glossary/sitemap)
- [llms.txt llms.txt is a plain Markdown file at a site's root, /llms.txt, that tells AI tools what the site is and links to its most useful pages with a line about each.](https://xodia.ai/glossary/llms-txt)
- [Canonical URL A canonical URL is the one address a page declares as its main version, with a link rel="canonical" tag, so search engines combine duplicates under it.](https://xodia.ai/glossary/canonical-url)

[See every term in the glossary](https://xodia.ai/glossary)

Glossary