Your own robots.txt and llms.txt

Every published site has two small text files at the top of its address:

• robots.txt tells search engines which pages they may read. It lives at yourdomain.com/robots.txt.
• llms.txt is a short guide to your site for AI assistants such as ChatGPT. It lives at yourdomain.com/llms.txt.

ContentMetric writes both for you each time you publish. You only need this page if you want to write your own.

Your own files are on the Pro plan and up. On the Free plan your site keeps the files written for you.

Write your own

1. Open Settings for the site and go to Files, under Hosting.
2. Each box shows, in grey, the file your site publishes today. Type or paste your own text into the box.
3. Press Save.
4. Publish your site. Your file goes live with the publish.

Your AI assistant can set these files for you too. Ask it to write your robots.txt or llms.txt.

What we always keep in robots.txt

Your text replaces ours, with two lines kept so nothing breaks by accident:

• Your sitemap address. If your text has no line starting with Sitemap:, we add one, so search engines can still find all your pages. If you wrote your own, we do not add a second.
• Members-only pages stay hidden. If your site has pages only members can open, we add a line that keeps search engines away from them, in every section of your file.

Your llms.txt goes live exactly as you type it.

Go back to the automatic file

Press Use the automatic file above the box, then Save and publish. Your site goes back to the file ContentMetric writes for you.

Common questions

I wrote a robots.txt but Google still does not show my site. robots.txt only works on your own domain. A site on a free contentmetric.com address is always hidden from Google, whatever its robots.txt says. Connect your own domain first; see Connect a custom domain.

My changes do not show at yourdomain.com/robots.txt. Publish again, then wait a minute and reload. Browsers and search engines can keep an older copy for a short while.

What happens if I move to the Free plan? Your text is kept, but your site goes back to the automatic files from its next publish. Move back to Pro and publish, and your own files return.