robots.txt
robots.txt is a plain text file at a site's root, /robots.txt, that tells crawlers which parts of the site they may read and where its sitemap is.
How Xodia writes robots.txt: Search engines settings, free on every plan, make it on every build from three switches, for search engines, AI training bots and AI search bots, plus your own rules and the sitemap's address.
Updated
On this page
What it is
A crawler that follows the rules reads robots.txt before anything else. Each group starts with User-agent, naming one crawler or all of them with *, and lists the paths it may not read with Disallow, or may read with Allow. A Sitemap line points crawlers at the site's sitemap.xml.
robots.txt asks; it doesn't lock anything. A page that is blocked can still show in search results if other sites link to it, so a page that must stay out of search needs a noindex tag instead, and private pages need a password.
In Xodia
Project settings, Search engines writes robots.txt for your site on every build. With search engines on, every crawler may read the site. AI training bots and AI search bots each have their own switch, and turning one off adds the rules that ask those bots to stay away. List the sitemap in robots.txt adds the Sitemap line, and Your own rules are added as written.
A site that already has its own robots.txt keeps it, and the card lets you edit it in place or hand it over to Xodia. On Xodia hosting, the Preview and publish previews always ask every crawler to stay away.
Read more
Related terms
- SitemapA sitemap is a map of a website's pages: for planning, a tree of pages and how they nest; for search engines, an XML file that lists every page a crawler should find.
- llms.txtllms.txt is a plain Markdown file at a site's root, /llms.txt, that tells AI tools what the site is and links to its most useful pages with a line about each.
- Canonical URLA canonical URL is the one address a page declares as its main version, with a link rel="canonical" tag, so search engines combine duplicates under it.