You Inc, Entrepreneur, Homepreneur, Globalpreneur & Infopreneur

A robots.txt file provides search engines with the necessary information to properly crawl and index a website. Search engines such as Google, Bing, Yahoo, etc all have bots that crawl websites on a periodic basis in order to collect existing and/or new information such as web pages, blog articles, images, etc. Once these resources are published via the website it is up to the search engines to determine what will be indexed.

A robots.txt file can help you better define what you want the search bots to crawl and therefore index. Doing this is useful for a variety of reasons including controlling crawl traffic to help ensure that the crawler does not overwhelm your server. The robots.txt file, however, should not be used to hide web pages from Google search results.

Views: 14


You need to be a member of MillionaireX3 to add comments!

Join MillionaireX3

© 2021   Created by Millionairex3.   Powered by

Report an Issue  |  Terms of Service