Is robots.txt necessary ?

acca · #1 (**permalink**) 06-28-2005, 08:06 PM

Since I see that googlebot and other robots wisited my page and looking for robots.txt , I thought robots.txt is necessary. Or at least, good to have them. Now, if I want allowed all robots full access, what should I put in that file? Is alright only

Quote:

User-agent: *
Disallow:

And is coments, # , necessary ? I didn't see any robots.txt without #.

jitender · #2 (**permalink**) 07-04-2005, 06:32 AM

All major Search Engine Bots search the robots.txt file ever time,

Code:

 User-agent: *
Disallow:

The above code allows all robots to visit all files because the wildcard "*" specifies all robots.

if you want to disallow to robots to do not visit the site use below

Code:

User-agent: *
Disallow: /

The below code disallows all robots to visit cgi-bin and images directory

Code:

 
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

Enjoy

Jitender

acca · #3 (**permalink**) 07-04-2005, 11:01 AM

Thans for answer.
I have another question about robots :
I just checked out, some robot was looking for robotsxx.txt on my site server. Whats about? Some xxx search engine ?

ethic · #4 (**permalink**) 07-05-2005, 01:36 PM

Hi all

my site also showing the same it's looking for robotsxx.txt file

PlantyNet_WebRobot_V1.9 dhkang@plantynet.com from 211.108.90.37

but i didn't find any details about it

ethic

jitender · #5 (**permalink**) 07-08-2005, 10:07 AM

i also find some of bot looking for this type of files

regards
jitender

bluepoolarts · #6 (**permalink**) 08-01-2005, 08:57 AM

i suggest you mail te autorities regarding tis case! As it migt be dangerous (security) for your website!

Eclipse · #7 (**permalink**) 09-06-2005, 07:09 AM

yes i have the same thing i think that email address in the above topic is like some type of scammer

spyrit · #8 (**permalink**) 11-21-2005, 10:39 PM

i would like to know the answer to this too !

Thomas Schulz · #9 (**permalink**) 10-11-2007, 02:36 AM

If I were going to email the authorities every time some automated hacking attempt or strange HTTP request occurred I would... Well, you get the idea, write lots of emails

HostQuack · #10 (**permalink**) 08-01-2008, 05:06 AM

Its not necessary but recommended. It is the best way if you dont want certain pages indexed. Then you add them in that file.

--
Cameron