All major Search Engine Bots search the robots.txt file ever time,
Code:
User-agent: *
Disallow:
The above code allows all robots to visit all files because the wildcard "*" specifies all robots.
if you want to disallow to robots to do not visit the site use below
Code:
User-agent: *
Disallow: /
The below code disallows all robots to visit cgi-bin and images directory
Code:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Enjoy
Jitender