Help me to write robot.txt

netweaver · #1 (**permalink**) 06-01-2004, 11:09 AM

Every time when i see the error messages, i see error "file not found robot.txt"

What is this robot.txt and how to write this.

jitender · #2 (**permalink**) 06-01-2004, 11:38 AM

Hi netweaver,

The robots.txt File
Search engines make use of a file called robots.txt to determine which web pages (and other files) are excluded from indexing when they visit your site. A sample robots.txt file has been included in your public_html directory.
Not all search robots honour this file. Never keep any sensitive information in your public_html directory as it could be viewed by anyone or indexed by search engine robots.

Some services are now indexing graphics on the web. If you don't want your images to be accessed in this manner, include them in your robots.txt file.

In case your robots.txt file is damaged or missing, here is a sample of the file. Simply cut and paste in-between the dashed lines.

Code:

---------
User-agent: *
Disallow: /images/
Disallow: /templates/
Disallow: /cgi-bin/
Disallow: *.jpg
Disallow: *.gif
---------

For more details try this link
http://www.robotstxt.org/wc/robots.html

Regards
jeet

netweaver · #3 (**permalink**) 06-01-2004, 11:42 AM

Thanks jeet for quick reply. I will try this.

flak · #4 (**permalink**) 04-21-2005, 09:55 AM

I would advise you to validate your robots.txt just for peace of mind.

Last thing you want to do is stop the spiders from indexing your content.

Here's a good tool: http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

spyrit · #5 (**permalink**) 11-21-2005, 10:43 PM

i never got this error, and i don't have robots.txt file, is this good or bad???

jugo · #6 (**permalink**) 04-28-2006, 01:50 AM

It's always good to have a robots.txt file. It help the search engine spiders to better find the info you want indexed and to keep them away from the pages you don't.

jitender · #7 (**permalink**) 04-28-2006, 07:10 AM

yes jugo,

if you don't have any robots.txt file on server, every search engine locate for it and you server log show you file not found error..

spyrit · #8 (**permalink**) 04-28-2006, 10:50 AM

it it's called robots.txt but is empty, is it ok?

jitender · #9 (**permalink**) 04-28-2006, 11:12 AM

yes..

if your robots.txt is empty it's ok

webhostingrebates · #10 (**permalink**) 05-04-2006, 06:15 PM

Will not having a robots.txt effect your rankings in the search engine?