Did we have a problem.....?

Dave,
Several of us had trouble getting pages to load around 1-2 pm today. I couldn't get anything, on DG, to load at all, I am on broadband. Did the site have a problem? Or was it at my end?

I had that also. I see that Dave made changes to the plantfiles around that time.

It's a hard story to believe, but someone in the data center in Dallas actually unplugged our entire rack, turning off all the servers. I couldn't believe it.

We were down for 26 minutes.

dave

I somehow have a mental picture of a janitor sweeping the floor, tripping over a plug and wondering -what/where-the plug was used for.

smiles,
Lisa

This message was edited Nov 3, 2005 7:51 AM

I guess they don't understand that it would have caused a world-wide panic if it hadn't been corrected as fast as it was.......did you have to call them Dave, or, did they stumble on it themselves??

I called them immediately, and told him that I was very panicked and that he needed to take care of this post-haste, which he did.

I hope it doesn't happen again.

When I go to Dallas around Thanksgiving time to install Lion and Lamb, then I am going to bring some plywood and will add extra protection to my rack.

dave

Was this one of those "The dog ate my homework" moments!? Who would have thought to look for a cord unplugged....panic time yes,but still funny! Sorry..strange sense of humor here. :-))

Hurricaine shutters for servers? Painted with the DG logo?

Tell them if it happens again, we will all show up and compost them!

I took it as a sign to do something else, so I took a much-needed (and highly unusual) nap!

With the time change, I'm getting an hour earlier, but I still go to bed at the daylight saving time, and an hour of midday snoozing did wonders for my disposition for the rest of the day ;o)

Terry,
Glad I'm not the only one having trouble adjusting to the new time - I just didn't get a nap since my boss might frown on such activities. I did notice DG was down when I tried to read it on my lunch break but just figured it was DH's computer - it is a little strange sometimes so I waited until last night after I got to the house - really slow on dial-up though

denise

Hey Denise!!! Chuckle. I used to tell my kids that *someday* they would wish with all their might for naptime, let alone have someone MAKE them take one every day, provide them a quiet, darkened, comfortable room with a snuggly blanket and soft pillow and let them sleep until they woke up on their own.

I think childhood is wasted on the youth - we certainly don't appreciate it until it's gone!

Hubby is in computers, says tripping over power cords happens a lot. The best one he had was the person who plugged the computer back in but it wouldn't power up. When he got to the desk, he found the person had plugged the powerstrip INTO ITSELF! We had a good laugh over that one!

Thanks for the info Dave, I'm glad it was a quick fix. Perhaps you should superglue the plugs in place when you get up there!!!

Sure less exciting than when construction workers cut right through a main cable!

What's Lion and Lamb?

There's a good DG Trivia question! I believe that Lion and Lamb are the original servers that were in Pennsylvania, and are being relocated to Dallas to join the newer ones there for an even bigger, better, faster website.

Well... assuming that Dave can secure all the power cables so they don't get unplugged again.

Did I win or lose? ;-)

Umm... did we just experience another hiccup? I lost the capability of connecting for about 30 minutes. :-(

Yes, it crashed again.

I found the problem with the server this time. The system was actually crashing when it ran out of memory (!!)

As it is, for some reason (that I don't yet know) my swap entry wasn't present in /etc/fstab so after the reboot yesterday, the swap was off.

I enabled swap and that should take care of it. I don't expect this problem again.

dave

Here's the error, in case any Linux nerds are curious out there, what happens when Linux runs out of memory. :-)

Nov 3 23:16:16 temple kernel: oom-killer: gfp_mask=0x80d2
Nov 3 23:16:16 temple kernel: Mem-info:
Nov 3 23:16:16 temple kernel: DMA per-cpu:
Nov 3 23:16:16 temple kernel: cpu 0 hot: low 2, high 6, batch 1
Nov 3 23:16:16 temple kernel: cpu 0 cold: low 0, high 2, batch 1
Nov 3 23:16:16 temple kernel: Normal per-cpu:
Nov 3 23:16:16 temple kernel: cpu 0 hot: low 62, high 186, batch 31
Nov 3 23:16:16 temple kernel: cpu 0 cold: low 0, high 62, batch 31
Nov 3 23:16:17 temple kernel: HighMem per-cpu: empty
Nov 3 23:16:17 temple kernel:
Nov 3 23:16:17 temple kernel: Free pages: 14980kB (0kB HighMem)
Nov 3 23:16:17 temple kernel: Active:486832 inactive:2014 dirty:0 writeback:0 unstable:0 free:3745 slab:6537 mapped:484544 pagetables:8083
Nov 3 23:16:17 temple kernel: DMA free:8168kB min:44kB low:52kB high:64kB active:3032kB inactive:72kB present:16384kB pages_scanned:6448 all_unreclaimable? yes
Nov 3 23:16:17 temple kernel: lowmem_reserve[]: 0 2031 2031
Nov 3 23:16:17 temple kernel: Normal free:6812kB min:5744kB low:7180kB high:8616kB active:1944296kB inactive:7984kB present:2080704kB pages_scanned:23694 all_unreclaimable? no
Nov 3 23:16:17 temple kernel: lowmem_reserve[]: 0 0 0
Nov 3 23:16:17 temple kernel: HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Nov 3 23:16:17 temple kernel: lowmem_reserve[]: 0 0 0
Nov 3 23:16:17 temple kernel: DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8168kB
Nov 3 23:16:17 temple kernel: Normal: 273*4kB 15*8kB 12*16kB 19*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 6812kB
Nov 3 23:16:17 temple kernel: HighMem: empty
Nov 3 23:16:17 temple kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov 3 23:16:17 temple kernel: Free swap = 0kB
Nov 3 23:16:17 temple kernel: Total swap = 0kB
Nov 3 23:16:17 temple kernel: Free swap: 0kB
Nov 3 23:16:17 temple kernel: 524272 pages of RAM
Nov 3 23:16:17 temple kernel: 10354 reserved pages
Nov 3 23:16:17 temple kernel: 61619 pages shared
Nov 3 23:16:17 temple kernel: 0 pages swap cached
Nov 3 23:16:17 temple kernel: Out of Memory: Killed process 1910 (mysqld).

(at that point the server stopped responded, and needed the reset button hit)

dave

Yikes! Glad you found the problem, though how odd that the swap portion was missing. (not a Linux nerd, just possess basic systems understanding)

Curious now, though. Can you reset from a distance or is a body required to perform that small feat? :-)

A body is required in my case (there is a device one can buy that remotely resets machines, but it costs $1,500 and typically I don't need that).

My data center is staffed 24/7 with people who can reset for me.

dave

BTW, Donna- You are correct about who Lion and Lamb are :)

Trish