robots.txt file and ranking?
Pear asked:
I just read this on http://www.caffeinemarketing.com/?p=96 “If you’ve submitted your web site to the search engines than the spider robots will be searching for your robots.txt file. If they do not find one the robot generates a ‘404 File Not Found’ error everytime. These robots might be searching for that file several times a day. Adding the file to your root directory will help eliminate these errors and boost your rankings.”
I just read this on http://www.caffeinemarketing.com/?p=96 “If you’ve submitted your web site to the search engines than the spider robots will be searching for your robots.txt file. If they do not find one the robot generates a ‘404 File Not Found’ error everytime. These robots might be searching for that file several times a day. Adding the file to your root directory will help eliminate these errors and boost your rankings.”
Is this true? Can robots.txt help with search engine optimization?
Internet Marketing Las Vegas
Las Vegas Internet Consulting
YES, YES, YES and YES!
You should add a robots.txt file to the root directory of all your websites to help control the indexing of your site by robots that ignore the convention. In this file you specifically list any pages that you DO NOT want walked and indexed (such as password protected folders and folders which contain only images, etc.). The robots.txt file is very simple yet very powerful and every website should have a robots.txt file on the root directory.
Create a new file with Notepad and call it robots.txt
The two conventions used in robots.txt file are User-agent: and Disallow: /
User-agent: * By using the * or wild card you are addressing ALL robots. If you wish to address individual robots you need to list each robot separately with an individual User-agent: statement. They must be listed by their specific name or IP Address, along with a separate Disallow: / statement listing the folders and files you DO NOT want the specified robot to index.
Tip: Use the * wild card to address all robots….. it is the safest way
Disallow: / List any folders that you do not want to have indexed by robots.
Warning: Disallow: / used without any folder name tells the robot do not index ANY page of the website.
ALL Files and folders in the directory named in the Disallow: / statement as well as all of those under it will NOT be indexed by robots.
Sample of Folders that could be in this website that we would not like the spiders to index with the search engines:
Disallow: /tutorials/meta/
Disallow: /tutorials/images/
Disallow: /tutorials/assets/
Disallow: /tutorials/404redirect/
Example: Disallow: /tutorials/
Results: All files and sub folders located within the folder tutorials which includes all the folders listed in the above example as well as any other sub folders of the tutorials directory will not be indexed by the robots if you use this statement.
This would mean that the /meta, /images, /assets, /404redirect, AND any other folders as well as all of the files in those foldes will not be seen by indexing robots.
You may also list specific files that you do not want indexed in a robots.txt file.
###############################
#
# sample robots.txt file for this website
#
# addresses all robots by using wild card *
#
User-agent: *
# list folders robots are not allowed to index
Disallow: /tutorials/meta/
Disallow: /tutorials/images/
Disallow: /tutorials/assets/
Disallow: /tutorials/404redirect/
#
# list specific files robots are not allowed to index
#
Disallow: /tutorials/meta_tags.html
Disallow: /tutorials/custom_error_page.html
#
# End of robots.txt file
#
###############################
Social Media Marketing
Have a look at this link:
Las Vegas Internet Consulting
Robots are merely for indexing your site – and having a robots.txt file is merely providing these robots on how to peruse your site.
But to boost rankings? No. Indexing is different from ranking. Robots.txt file can be used to tell the search engine robots which files to pick up and which to exclude (eg. if you have a paid subscription section on your site that you don’t want to be available freely on the Web). Many sites also use robots.txt file to tell robots when to go to the site to prevent overload
You can have a high ranking site without even using robots.txt file. It can help in making sure that your pages are in the search engines, as well as to prevent overload but just by having it cannot push you to #1. My site does not use robots.txt file and the site is in the 1-10 position for many important keywords
I suggest you read RobotsTxt.org to understand better what robots.txt file really is and what they do