Robots.txt file used for search engine robots,
it's direct to robot where they go, access area or non access area,
if you want to disallow to robot to read a cgi-bin directory write below
Code:
User-agent: *
Disallow: /cgi-bin/
or you want to stop google robot to read you site's particular area use below
Code:
User-agent: googlebot
Disallow: /cgi-bin/
And if you want to stop read a particular file
Code:
Disallow: email.htm
I think it will help you
That's already in this forum
ethic
What it "Robots.txt"
Linear Mode
