What is a Robots.TXT File?
A robots.txt file which is also known as the robots exclusion protocol is a tool used by websites to communicate with search engine crawlers and other bots. The robots.txt tell the bots which areas of the website should not be processed or scanned.
Why is a Robots.TXT File Important?
A robots.txt file can be used to block specific areas of a website, an entire website, and even point links to an XML sitemap to help tell the bots where they can find a complete list of each page of the website. This helps improve the indexation process.
How Do I Find MY Robots.TXT File?
Every website should have a robot.txt File. If yours does not, its important to create one as soon as possible. You can either upload a robots.txt file into your root directory on your server, or use a plugin like Yoast to create one.
What Should Be In A Robots.TXT File?
- Its up to the creator to determine how their website should be crawled and by what bots. However, below are some common codes found
- “User-agent:” will be used to identify all bots or a bot.
- “Disallow:” will be used to disallow a bot or all bots from an entire website or directory
- “Allow:” will be used to allow bots to see a specific page or directory
- “Sitemap:” will show bots how to find your XML Sitemap