Baidu Spiders?

[corsair23]

Not sure why I noticed it, but I noticed that there were a LOT of "guests" with the IP address of 180.76.5.XXX

So, out of curiosity I did a Google search on 180.76.5 and one of the hits was:

https://www.vbulletin.com/forum/showthread.php/387062-Baidu-spiders


Not sure if they really affect our web site's performance but just now for instance, out of 75 "people" online, 36 of them were Baidu spiders - So almost 1/2. I know there are Google and Yahoo and other bots out there but typically you only see a couple instances of each at any one time.

Anyway, I'm not web guru but thought I'd bring it up...

 

[Hulk]

Looks like we just need to create a robot.txt file. I'll work on this.

 

[corsair23]

Looks like 180.76.6.xxx might be another one?

http://www.forumpostersunion.com/showthread.php?p=86441


The two together added up to 43 out of the 75 users online just a minute ago when I counted. Seems like they are the bain of many web folk

 

[60wag]

I know nothing about bots but have noticed the high number of users lately. How does the .txt file keep the bots out?

 

[corsair23]

I know nothing about bots but have noticed the high number of users lately. How does the .txt file keep the bots out?

Matt can probably explain it but this was a link within the first link I posted:

http://www.robotstxt.org/robotstxt.html

 

[simps80]

dang 8:56pm
Currently Active Users: 104 (12 members and 92 guests)
google spider's goin' nuts!!

 

[corsair23]

dang 8:56pm
Currently Active Users: 104 (12 members and 92 guests)
google spider's goin' nuts!!

no doubt...

12:26am
103 users (3 members and 100 guests)

I counted 22 Google spiders and at least 50 Baidu spiders...Oh, and 1 MSN bot

 

[corsair23]

Seems like the spiders are getting worse...

Just checked and out of the 61 users online a few minutes ago, 52 were spiders with IP addresses starting with 180.76.5 & 180.76.6

 

[wesintl]

so... they may suck but they are really only sucking bandwith which isn't an issue for us right now. :MEH: Not sure why it's got you so worried

Some don't respect the robot.txt file so you have to block them at htacces or other creative ways

 

[corsair23]

Spiders scares me

 

[Hulk]

Huh. I wonder if I can block them at the server level.

 

[Hulk]

Hmmm. Looks like I turned up the volume on the robots.txt file a little too high. I just fixed it.

Not sure if it ever did any good anyway.