Dear Vineet,
Robots.txt file used for search engine robots,
it's direct to robot where they go, access area or non access area,
if you want to disallow to robot to read a cgi-bin directory write below
Code:
User-agent: *
Disallow: /cgi-bin/
or you want to stop google robot to read you site's particular area use below
Code:
User-agent: googlebot
Disallow: /cgi-bin/
And if you want to stop read a particular file
Code:
Disallow: email.htm
I think it will help you
Regards
Jitender
what actually robots do?
Linear Mode
