Category: Technical SEO

robots.txt

A file controlling bot access to the site.

robots.txt is a file in the domain root that tells bots which URLs they should not crawl (Disallow rules). Note: a robots.txt block does not remove a page from the index — that is what the noindex tag is for. It is also worth listing your sitemap here.

🛠️ Check it in practice

We have a free tool related to this concept — test it on your own website.

Open the tool

🔗 Related terms

D

Disallow

A robots.txt rule blocking access to URLs.

N

noindex

Instruction: do not show this page in results.

S

sitemap.xml

A sitemap listing URLs for the search engine.

C

Crawl budget

How many pages a bot crawls in a given time.

Back to glossary

Check your website's SEO

Run a free, comprehensive SEO analysis and see which of these elements need fixing on your website.

Start free analysis