Bogged Down?

Hey Dave, Things have been loading awfully slow the last day or two. Pages are taking 30 to 80 seconds to load. Is there something that's bogging down the system?

Same thing happening here Poppysue, everything seems to time out a lot.

Some search engines think they own the net... sheesh. We're being blasted simultaneously by Altavista, Fast.No, some kind of "fast-crawler", and something called "Pompos". All at once.

These robot developers have no consideration for other people. :( DG has over a million pages, and these search engines just release their robots to reek havoc on a large site like DG. One at a time is okay, but when 4 of them are hitting 5 pages per second, it causes severe slowdowns. :(

I put code in a few minutes ago that should slow them down. I am considering firewalling all search engines except for Google. That's where 99% of our searches come from anyway...

Dave

This is getting ridiculous. I just firewalled out Scooter (altavista) and Fast-Crawler (fast.no). Those two search engines are the worst of the whole bunch, and will no longer be able to crawl DG until I change my mind and let them back in (unlikely).

Dave

is that why im getting time-outs and i have to reload?

I just have to laugh. Another search engine just came into the mix, this one called inktomi, and the robot is called "Slurp".

You know, when you have a webmaster whose website is being bogged down by greedy robots, the last thing you want is to have a robot called "Slurp" show up in the logs at the moment that the webmaster is getting most irritated.

Why do they all pick one moment to all converge on a site all at once? Good grief!

...

I digress.

Dave

im not very computer smart, but where are the search engines doing?
a hostile takeover?

No, they are indexing the site. They pull down each and every page on DG, in order to add it to their inventory of information that their users then search through.

This is usually okay, because they are programmed to wait a second or two between pageloads. Bad robot design, however, just has the robot suck down as many pages as my server will allow. When several of the search engines gang up on a site, it results in sometimes 20-30 pages per second. Then, the paying subscribers to the site have to suffer severe slowdowns so these search engines can index a million pages. I don't really know a good solution, except to firewall out the "bad guys".

Dave

I am glad of the explanation because I had begun to think it might be my computer at fault. Thanks for informing us, Dave. Good luck ridding the place of all those hoggy search engines.

Thanks Dave! It's working much better now - I knew there was something going on.

thanks dave

Thanks a lot Dave, it's infinitely better now. Robots called Slurp? whatever next? LOL

Wondered what was happening. Certainly much better now thanks Dave. Just amazing to hear what's going on behind the scenes. Next after Slurp........... Burp??

LOL. 'scuse you :) (as my DD would say)

Slurp........and Burp.....all gone, gone!! Much better Dave!!

"eyes"

Was wondering what the heck was going on,thought it was my machine,lol' Curious Dave,could these "invaders" be gathering our e-mail info also here for their own personal "agendas",perhaps????? Sure hope '''TIA,Sis'

Sis,

Your E-mail address and other personal information are not visible to anyone, so the answer is no.

Dave

Popularity definitely has it price! LOL, Thankyou Dave, for being such a consciencious webmaster and all around good guy.

That's a relief,thanks Dave' Sis'

thanks will have time to get through the forms again

Was slow last nite.....

"eyes"

This was such a good thread to find! I thought I was going to have to reinstall something, just didn't know what. It's bad enough with the things my system does, I didn't realize the search engines worked like that. Or that you could tell when they were doing anything. Puting is hard, and running putering is just way hard.

Aimee-Ditto here. I just got a new hard drive loaded with Windows XP, and I've had my share of problems getting things to be compatible, so I was just blaming my computer, too!..Now, if I could get this new system to download photos on my new digital camera, I'd be a lot easier to get along with..WZ

I'm still finding things slow...
Someone said something about cleaning cookies????
How do I do that please?

It is slow for me also. I delete my cookies, Defrag, clean my cache and helps a little, I do this ones a wk.

Mummy, This is how mine works. Go to start,click on settings, click on control panel, click on internet options. You will see Temp. internet files. Click delete files. Done!

If you have IE version 6, there's a button under TOOLS - INTERNET OPTIONS - GENERAL for "delete cookies". It's a little different in version 6. I noticed it was really slow a few nights ago, too - this must be why!

here's a link that might help too? http://davesgarden.com/showthread/128554.html

I had no idea that's how the search engines worked. The more I'm on DG, the more I find out how much work it really is to be a webmaster of a site such as this. REALLY drives home the fact that I'm getting my fifteen bucks worth and THEN some. Really. I'm getting the better end of the deal here. Thanks, Dave.