Apparantly, the BS never stops. Lately there has been a group of people that wanted nothing more than to see PFF die. I've been getting a lot of crap through email, people directed me to threads and threats on other forums which I all ignored - simply because I couldn't believe anyone was really serious about this. This is nothing more then an internet forum after all, right?
Well, today I found out my server has been comprimised. And not a "defacement" of my webpages, but a serious hack. So serious, the only way to get rid of it is to completely take my server(s) off line and to reinstall everything. Mind you, this is no easy task. This will take me days - if not weeks.
I am highly depending on my servers for my income. Something like this is going to cost me money. And probably clients too. To think I've been pumping cash into the forum for years at end, money earned with my "normal" work, and now that same forum will be the cause of even more income lost.
Since a year, I have an extra mouth to feed. I can not afford BS like this. So When I take my servers down, I will have to think long and hard if I'm going to bring the forum back up again. Because really, as much as I love the Fiero, and as much as I feel I have a responsibility towards the Fiero community, it isn't worth this kind of trouble.
I really have no idea where I am going to find the time to bring the server down and reinstall it. I really don't. I don't even know how my business will survive with my servers down. I'm highly depending on my servers, for just about everything. I was propably naive to think this wouldn't happen to me. Since again, this is just an internet forum. I don't have a website with radical political views. I don't have a website which propagates any radicals views. I'm just running a website that tries to help a lot of people that share the same interest: the Pontiac Fiero. And still, there are people out there that hate it and me for that.
So this time, the bad guys win. They can start celebrating on the other forums. PFF will go down soon - because like I said, the only way I can clean this mess is by taking the server down and reinstall everything. And will PFF be back? I don't know. I really don't. But right now my honest thought is that it isn't worth it. It's 4:15am local time now. I need some sleep. Maybe tomorrow I will feel otherwise.
It's been mostly fun the past 6 years. Too bad the last few years a handful of people felt it necessary to spoil the fun.
Update 7:45am I just had a few hours of sleep, so now I can tell you a bit more what's going on.
I still don't know how my server has been comprimized. I am subscribed to the security mailing list for my particular distro and I have applied every security patch relevant to my system as soon as they were released (so Cryptnix, I have not taken a "it works so leave it" stance - far from it. And I would appreciate it if you just kept your mouth shut if you don't know what the hell you are talking about). Or at least, I thought so. I've run a few "scanners" and apparantly, there were still two or three "vulnerable" spots. Now I receive the security alerts by email, but I also receive tons and tons of spam every day so it might very well be I accidently deleted a security bulletin or simply overlooked it. The server has been running for over six years without a hitch or without ever being comprimized before. That should give you a good indication of my dedication keeping the server clean. The fact people here wonder what happened when the server went down for only 10 minutes yesterday, is another. It means people got so used to it always being up.
Anyway, one of the checks I do on a daily base, is check for security breaches. Yesterday I noticed the server took longer than normal to log me in. It usually logs me in instantly, now it took a few seconds - something that has never happened before and immediately seemed suspicious to me. So I tried to view with "top" (a Linux program to show processes and their CPU/memory usage) if there was any program causing this. "Top" immediately threw an error which is usually a very bad sign. Because all exploits are very good at one thing: hiding themselves. And one of the ways they do that is by altering all programs that can bring their existence to light.
I immediately ran a few "scanners" (which is the reason I brought the webserver down for 10 minutes yesterday, because I needed the CPU cycles) and they all picked up that indeed a whole bunch of programs have been altered. There could be no doubt about it: the server has been comprimized.
The problem is that the recommended course of action action from this point is to backup as much data as possible, format the harddisk and reinstall everything. Because you really can't find out exactly what damage has been done. This is no easy task. For me it means I have to drive to another city to pickup my server, take it home and start the reinstallation process. This usually takes a few days because not only does it involve a reinstall of the OS, but all the security patches as well and testing the server. Taking the server down will make a few of my clients very unhappy, I know this for sure. But I don't mind that as much as that this is going to take time away which I usually spend with my 11-month old daughter, which BTW is something I'm unwilling to do (spend less time with my daughter). And that's the reason I'm considering to take down the forum. Not because I'm loosing money over this, or clients - but because this is beginning to affect my private life.
Do I know who did it? No. Do I think it's somebody from the PennocksSucks forum? I don't know. All I know is I've been receiving lots and lots of email lately from people who pointed me to threads there where they were discussing "some people" were up to "something". I never took that serious (so I never even visited that place) because I couldn't imagine people could actually get so worked up over an internet forum. But in the past few months, my log files did show an increase in hack attempts. Some ip-addresses closely matched those of banned members. Other ip-addresses were also found in the forum's access log, so I knew some of them frequently visited the forum. But since all of those attempts were really "script-kiddy" attempts, I wasn't worried and I didn't block those ip-addresses. Do I think Shaun did it (because he apparantly "said so" on the PennockSucks-forum). No. The guy has a bigger mouth than he has brains so he really isn't capable of doing such a thing. So don't direct your anger towards him. If you want to accuse him - or any other of the "playas" there - of anything, all you can accuse them of is that they are probably gloating over this. And in all honesty, if something like this happened to them I would be gloating a little bit too.
So who's to blame? I don't know yet. It could very well be a random attack. It could very well be a disgruntled ex-member. At this moment, I only blame myself for my own naiveness. Because that's going to cost me now. Dearly. I blame myself for not taking the "buzz" more seriously. If I had done so, I probably would have found out I had left a few parts of my server vulnerable.
I know Linux very well, but I have no experience with solving (as opposed to preventing) security related issues on it. Simply because it has not been necessary in the past 6 years. If you think you can help, you can contact me at CliffPennock@hotmail.com. Yes, I run my email through the same server, so I can't trust that anymore as well.
Update 11:50am I'm still nowhere closer to finding out how they did it. All I know now is when it happened. The server was compromized on friday at 6pm local time. I've been trying to make backups of the forum and get that over to my computer. Unfortunately, I can't get decent speeds out of the server anymore and trying to pump a few Gbytes to my computer would take days...
Update 4:30pm Well, I feel a little better now because there is a slight chance I can clean this up with little downtime. I might be able to at least secure the server so it can stay running while I prepare the new server. I figured out what it is they/he/she are trying to do so at least I now know what I'm battling. I might have caught this just in time, or at least, I hope so.
Also, I would like to express my sincere thanks to everyone who has shown their support. This is all very frustrating for me. Not because others are apparantly celebrating over this (because I couldn't care less really), but because this is taking so much time. And it feels like someone broke into my home. Your support really means a lot to me. It reminds me why I do all this - because for 99.9%, Fiero owners really are a great bunch of people. Whatever happens, I will make sure PFF stays alive - with or without me.
Update 6:10pm 14 hours of work later and I think I now have an ip-address of who did it. And if there's one thing I can do, is tie an ip-address to a name. So I do have a name right now. I also think this attack is unrelated to the forum (which is important [and a good thing] because I don't want people ripping each other a new one at coming car-shows ). My first priority now is to make sure the server is secure. I am on top of it, and I think I'm in control now.
Again, I would like to thank everyone for their help and support. I also would like to thank everyone who has made donations in the past day. However, when it turns out I don't need a new server just yet, I will refund it all. I know how tight funds are for some of you, and I don't want to take your hard-earned money when it's not needed. I'm pretty sure PayPal doesn't charge their fee if it's refunded within 30 days. This is non-negotiable. If after I refunded the money you still want to make a small donation, that's fine of course. But don't if you really can't.
Update Mar 14, 5:10pm As far as I can see, I have been able to completely isolate, delete and reinstall the infected parts of the server. And as I mentioned earlier, this hack was in no way related to the forum. Somebody was just out to setup a "0day" warez server, and he apparantly found a hole in my server's security. My luck was that I caught him in the act of installing a stealth ftp server. I was able to log his actions and get his ip-number. I went to the police today to press charges. They said they would forward it to the "digital crime" department but it probably won't be easy to proof he did it. I'll keep you updated.
If you need to shut down the servers, please do one thing before you take it down. Tell us who did the damage, so we can spend our PFF time meking life he77 for the scum bags.
If you need to shut down the servers, please do one thing before you take it down. Tell us who did the damage, so we can spend our PFF time meking life he77 for the scum bags.
Cliff, if worse comes to worse, could I suggest that you burn CD/s of the entire Forum History, and "Market" them as a complete set to offset the damage done to you.
Tom
[This message has been edited by California Kid (edited 03-12-2005).]
I am rather stunned at the moment. This forum has ment the difference between my 85 being the daily driver that it is now or being in a junk yard somewhere. I just can't fathom the mentality that would do such a thing to such a great group of people and you in paticular Cliff. Words just fail to describe how I feel right now.
Cliff, as a means to communicate during what will hopefully be a short PFF shut down & as a way to keep PFF believers informed as to what's going on, may I suggest people monitor Fiero-Tech-Net? I know the guy who owns Tech-Net & I'm sure he won't care.
Archie
[This message has been edited by Archie (edited 03-12-2005).]
exactly what paulcal said, with out this place many of us wouldn't be where we are at. I was on the mailing list before PFF started that fiero-list, but emails aren't to interactive like this and I refuse to go back cause you can't get feedback like here.
I just found this forum a few months ago. It has totally reinvigorated my intrest in the Fiero.
I sure hope that it doesn't go down.
Cliff, I am out of work at the moment, but I have some money in m Paypal account. If you decide to keep PFF online, I can send some your way. Hell, I think I'll just send some anyways since you said taking down the servers hurts your income.
I hope others can do the same to help out someone who has helped us so much.
If you do take it down, I hope the best for you. And I thank you. I also hope we all can find a place as cool as this to gather and talk about our Fieros and share info like we have here. Of course I hope it doesn't come to that.
Lotta hoping Rick
EDIT: I just donated to Cliff through my Paypal account. If the forum goes down at least my money will go to help him out for some of the money he lost through his business because of this. I too (as someone later in the thread stated) would buy an archive CD. Please, everyone donate to Cliff. If not to help keep the forum up and running, then for what he has already done by keeping it up this long.
[This message has been edited by slickrick2000 (edited 03-13-2005).]
Sorry to hear about what those a$$wipes have done to the forum and your livelihood. Please tell us whatever we can do to help. If the forum disappears forever I am going to need rehab.
Man..... this reminds me of these old forums I used to visit... Several disgruntled members were banned, and started their own forum devoted to the same topic. Both forums grew quite well... But the hatred between certain members of both forums grew, and then the attacks started. One forum would DDoS the other, and it was ugly. The "good" forum would be up for a day, and down for a week (or three). The "bad" forum was pretty much always down. Well, sad to say, now they both don't exist. The rocks they threw crippled each other so badly, there was nothing left of either to save. It's too bad too, they were both wealths of information.
I hope this isn't the start of one of these wars. I can't go through it again
Sorry to hear about this Cliff. Ive been a member now for what......5 years? And This is the only place I know of to get info on Fiero problems and Shows. With out this Forum we would not be able to find most of the shows out there. At least I wouldnt be able to. Meaning show dates and such.
Instead of complaining and stuff we should show them that we won't be defeated so easily...we should all try to help by donating to Cliff so he can keep PFF up easier. I'll try to donate but I'm low on cash atm.
Cliff, I'm sorry to hear what's happened to you, and even more sorry that what started out as a hobby, and has become quite an undertaking, is the cause of your problems. We all know that PFF is not a cheap, fly by night proposition, and that you DO throw a LOT of money at it to keep it going. I'm sure I speak for a lot of people here when I say I hope you DO decide to keep it going...but that if you don't, I understand, and respect your decision. Whatever you decide, I thank you, immensely, for all that PFF has provided me over the past years...technical help, parts location, a place to share my Fiero and non-Fiero information, and a place that has provided me with more than one good laugh from time to time, and many friendships that will last for a long time. You have provided us with a great service...and for that, I will always be thankful.
Sorry to hear of all the problems they have caused. This is the best place on the Internet for Fiero resources. Thanks for providing this to all of us over the years. I do hope it will continue, But if you decide not to, Take Care.
------------------ Happiness isn't around the corner... Happiness IS the corner. ZZ4 Powered !!
it's just a bunch of candy a** little punks who hide behind their computers. meet them anywhere else and they're little pansies. probably throw rocks at dogs on leashes too, cowards. this pisses me off. i come here for hours every night. throw some ideas out here as to what we can do to keep this alive. a few bucks from members may help out i'm sure. worse case, if this goes down, i need to get the CD's. i can't get it from eMule either for some reason. how can i buy it?
this blows Im stunned. I really feel like the PFF fiero community is my family. I just went though a hole change in my life and I feel more closer to the forum than ever. Im pissed that some one has to f with my family. I cant beleive some one would do that. Man I hope the forum comes back stronger than ever. Let us know how to help, I still cant beleive this.
laura (dookiecheese)
[This message has been edited by fierosis (edited 03-12-2005).]
Could have been anyone or just some random luck of the draw for some hacker, but please don't let this be the end. This forum is the best thing I've seen on the internet and I would hate to see it go. Times are tough, but we all can contribute something to save the forum.
------------------ Ron Freedom isn't Free, it's always earned. Pantero Creations
Cliff buddy.... i haven't been here very long but in the short time i have been ive learned so much and met so many new people. I understand you have to do what you have to do... but if theres a chance you can save this ship... try your damndest... i wish i could help in some way but im just one person with one pc and very little resources... but before you put the axe to it (if you do) remeber these words:
"The only thing necessary for the triumph of evil is for good men to do nothing..."
Could have been anyone or just some random luck of the draw for some hacker, but please don't let this be the end. This forum is the best thing I've seen on the internet and I would hate to see it go. Times are tough, but we all can contribute something to save the forum.
Wow, this sucks for sure. I dont want to see PFF die.
Cliff, why not host from an american server like HostRocket or IPowerWeb and let the attacks hit those servers so it is not so close to home on your other business? Also if someone attacks a company like that, I would think they have resources to get the FBI involved.
Without PFF, I will have nothing to stare at all night What would it take to keep it going? Let us know, theres enough of us, just tell us and well do what we can
I consider this forum as absolutely essential to my hobby, and I am sure thousands of others feel the same way. You already know that for every 1 agitator, there are thousands who support your efforts of providing the finest internet community there is. Most of us already know that behaving in a mature and civilized manner is in all of our best interests.
I would wire transfer a good sized contribution should you decide it is worth continuing. I hope others do the same. I've seen the "other" forum, and I find it lacks in quality standards and integrity.
I am sooo sooo sorry for all this, and I think that allot people will in fact donate some money to ya just to keep this forum alive........Heck, coming here everyday to check on things is a reason to get up in the morning for me...As I am sure it is for others as well....I do honestly think that people will contribute to the cause......SAVE PFF!! Come on everybody, this forum is one of the best outlets for many users to contribute, meet, discuss the car we love sooo much....This is our home, we can't let people force us from this....We have the rights to bear arms, only ? is WILL WE LET this place go quietly in this good night???????????
Originally posted by pavo_roddy: only ? is WILL WE LET this place go quietly in this good night???????????
NEVER!!!!!!!!!!!!!!!!!!!!
But people - resist the urge to do anything that will cause you to sink to their level. It is hard - this I know. But it must be done to keep anything from snowballing. No compu-lynch mobs, please. We gotta do this by the rules.
The first thing in the morning and last thing I do at night is cruise this forum. It means a lot to quite a few people. Just wanted to extend a thanks for all the years of good reading, relaxing, and keeping my Fieros on the road.
I understand you have to do what you have to do, but if there are other options, I'd really hope you choose to keep this place running
Cliff, Let us know what we can do to help. I check out this site everyday for info. I and many others would buy an archive cd if you have to close it down, but would rather contribute to keep it up.
Whatever happens, I will make sure PFF stays alive - with or without me.
). My first priority now is to make sure the server is secure. I am on top of it, and I think I'm in control now.
