Pip, images.fieroforum.com and myfiero.com will be down for a few hours incase anyones having problems viewing the images.
quote
Today at approximately 5:45 p.m (central time)., a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down. We are working with the fire department, with our facilities staff on site, to assess the situation.
IP: Logged
08:48 PM
PFF
System Bot
mrfiero Member
Posts: 9002 From: Colorful Colorado Registered: Mar 99
Cant access it to see what you mean.. are you referring to the googles badware thing? If so, the badware website tagged that site saying I had a script on there that was listed as badware.. their report showed it was ADSENSE!! lol
I sent a nice complaint letter to them and it was removed yesterday I believe..
Official word is that the first transformer exploded knocking down 3 walls around the electrical room and causing a second transformer to catch fire..
NO SERVERS WERE HARMED and testing of the electrical is scheduled for 5pm cst and this will take 4 hrs.. after this, the servers will be started up in blocks.
[This message has been edited by isthiswhereiputausername? (edited 06-01-2008).]
IP: Logged
04:12 PM
blakeinspace Member
Posts: 5923 From: Fort Worth, Texas Registered: Dec 2001
Yep.. origional statement was just a transformer that caught fire, then it was an exploded one that caught a second one on fire.. now the official statement is the explosion with the room damage..
It sucks when details are so sketchy as I need to provide answers to clients
IP: Logged
04:46 PM
Jun 2nd, 2008
Fierobsessed Member
Posts: 4782 From: Las Vegas, NV Registered: Dec 2001
Bump for those desperatly wanting the photos to come back.
trust me, I do too lol
(not much sleep and a steady migraine.. plus phone started ringing at 9am sunday morning)
Current status is power is being restored to the second floor, but the main floor where my server is with them, that floor was hit worse and will be running under generator power tonight. (hopefully) and then switched back to electrical next weekend when their electrical is all replaced there. I hope they can afford the cost of diesel LOL
[This message has been edited by isthiswhereiputausername? (edited 06-02-2008).]
The bad part is most on site generators were never meant to run that long, we had to run ours for 2 weeks and it was almost the end of it. Our UPS took a dump so we had no choice as the city power wasn't trustworthy.
The bad part is most on site generators were never meant to run that long, we had to run ours for 2 weeks and it was almost the end of it. Our UPS took a dump so we had no choice as the city power wasn't trustworthy.
Yep.. they plan on running them to power phase1 (first floor) of the houston datacenter until Saturday.
Server is back up and pip is running May still be a few issues with propergation for some users though.
IP: Logged
04:46 PM
Cliff Pennock Administrator
Posts: 11800 From: Zandvoort, The Netherlands Registered: Jan 99
Seems to be having problems with one of the generators powering the routers, they say the boxes "may" have power, but not accessable This is becoming very annoying!
Around 2:20 AM CDT, the backup generator being used to power H1 Phase | experienced an electrical issue resulting in service loss for Phase I; Phase II remains unaffected at this time. Our data center operations and facilities teams immediately began investigating the cause of the failure to restore power to the Computer Room Air Conditioner (CRAC) units and Power Distribution Units (PDUs) for Phase I.
The staff successfully tested the 2 megawatt generator without load, so they began powering up the CRAC units and PDUs to restore service to Phase I. While working through this power restoration, the generator's breakers were tripped by their internal electronics. The generator is rated to handle more than the load required to power the phase, and the generator itself is fully functional, but the breaker system must be replaced to guarantee stable power distribution.
We have attempted to locate a replacement generator and are evaluating the time necessary to repair the breakers on the current generator so we can restore power as quickly as possible. We do not have an ETA for power restoration, but we will be updating you hourly with our current status or sooner, as developments warrant.
Around 2:20 AM CDT, the backup generator being used to power H1 Phase | experienced an electrical issue resulting in service loss for Phase I; Phase II remains unaffected at this time. Our data center operations and facilities teams immediately began investigating the cause of the failure to restore power to the Computer Room Air Conditioner (CRAC) units and Power Distribution Units (PDUs) for Phase I.
The staff successfully tested the 2 megawatt generator without load, so they began powering up the CRAC units and PDUs to restore service to Phase I. While working through this power restoration, the generator's breakers were tripped by their internal electronics. The generator is rated to handle more than the load required to power the phase, and the generator itself is fully functional, but the breaker system must be replaced to guarantee stable power distribution.
We have attempted to locate a replacement generator and are evaluating the time necessary to repair the breakers on the current generator so we can restore power as quickly as possible. We do not have an ETA for power restoration, but we will be updating you hourly with our current status or sooner, as developments warrant.
Didn't mean to jinx you, we had the same issue where I work. The generator and UPS would hold all of our servers but the breaker wouldn't. They brought in a new UPS with breakers and it was way under rated for our data center.
I hope you have service soon, not just because of the pictures but because I can feel your pain.
I've redirected PIP to the main PFF server so you should be able to post images again. As soon as the "hostkansas" server is back up again, PIP will be redirected back to that server.
This means that previously posted images are still unavailable, but you are at least able to post new images.
They probably accidentally erased all their servers and are afraid to say it.
Thanks lol
Actually, they did end up telling me the other day that my box was ready, but I kept getting auths and know at the SSH login, it wasnt my box due to the different version.. took forever for them to understand this.. FINALLY they noticed it and swapped the boxes..
Shes up now, and propergating. I see some pff users on there now, so its starting to work for some..
IP: Logged
10:07 AM
Xanth Member
Posts: 6886 From: Massachusetts Registered: May 2006
Figured if I mentioned the worst case scenario things could only get better
trust me, it scared me bigtime when the OS was different..
lol
Happened to a guy I know last year, he upgraded his slave drive for backups and they ended up sending his both drives to the format dept.. after a day they were able to locate it since a tech forgot it inside the rack and was able to put it back in..
IP: Logged
10:25 AM
Monkeyman Member
Posts: 15829 From: N. Wilkesboro, NC, USA Registered: Nov 1999
I guess a second transformer blew as well. updtaes are slow to hear 
May still be a few issues with propergation for some users though.
Edit I quess it was fix 
my bad
