what actually robots do?

[vineet jain]

Dear Friends,

I have my sites made by a friend.

can U pls help me on what does the robots do for our site or why do we need them?

can U pls also help me if we can make robots?

whats all the world talking abt?

I wish I know and am much more familiar wth them soon.


Thanks

Love and Light

Vineet Jain


I am proud to find this site !!!

[jitender]

Dear Vineet,

Robots.txt file used for search engine robots,

it's direct to robot where they go, access area or non access area,

if you want to disallow to robot to read a cgi-bin directory write below

 User-agent: * 
Disallow: /cgi-bin/ 

or you want to stop google robot to read you site's particular area use below

 User-agent: googlebot
Disallow: /cgi-bin/ 

And if you want to stop read a particular file

 Disallow: email.htm

I think it will help you

Regards
Jitender

[Thomas Schulz]

You can also use robots.txt for XML sitemaps autodiscovery