Well, today I found out my server has been comprimised. And not a "defacement" of my webpages, but a serious hack. So serious, the only way to get rid of it is to completely take my server(s) off line and to reinstall everything. Mind you, this is no easy task. This will take me days - if not weeks.

I am highly depending on my servers for my income. Something like this is going to cost me money. And probably clients too. To think I've been pumping cash into the forum for years at end, money earned with my "normal" work, and now that same forum will be the cause of even more income lost.

Since a year, I have an extra mouth to feed. I can not afford BS like this. So When I take my servers down, I will have to think long and hard if I'm going to bring the forum back up again. Because really, as much as I love the Fiero, and as much as I feel I have a responsibility towards the Fiero community, it isn't worth this kind of trouble.

I really have no idea where I am going to find the time to bring the server down and reinstall it. I really don't. I don't even know how my business will survive with my servers down. I'm highly depending on my servers, for just about everything. I was propably naive to think this wouldn't happen to me. Since again, this is just an internet forum. I don't have a website with radical political views. I don't have a website which propagates any radicals views. I'm just running a website that tries to help a lot of people that share the same interest: the Pontiac Fiero. And still, there are people out there that hate it and me for that.

So this time, the bad guys win. They can start celebrating on the other forums. PFF will go down soon - because like I said, the only way I can clean this mess is by taking the server down and reinstall everything. And will PFF be back? I don't know. I really don't. But right now my honest thought is that it isn't worth it. It's 4:15am local time now. I need some sleep. Maybe tomorrow I will feel otherwise.

It's been mostly fun the past 6 years. Too bad the last few years a handful of people felt it necessary to spoil the fun.

Update 7:45am
I just had a few hours of sleep, so now I can tell you a bit more what's going on.

I still don't know how my server has been comprimized. I am subscribed to the security mailing list for my particular distro and I have applied every security patch relevant to my system as soon as they were released (so Cryptnix, I have not taken a "it works so leave it" stance - far from it. And I would appreciate it if you just kept your mouth shut if you don't know what the hell you are talking about). Or at least, I thought so. I've run a few "scanners" and apparantly, there were still two or three "vulnerable" spots. Now I receive the security alerts by email, but I also receive tons and tons of spam every day so it might very well be I accidently deleted a security bulletin or simply overlooked it. The server has been running for over six years without a hitch or without ever being comprimized before. That should give you a good indication of my dedication keeping the server clean. The fact people here wonder what happened when the server went down for only 10 minutes yesterday, is another. It means people got so used to it always being up.

Anyway, one of the checks I do on a daily base, is check for security breaches. Yesterday I noticed the server took longer than normal to log me in. It usually logs me in instantly, now it took a few seconds - something that has never happened before and immediately seemed suspicious to me. So I tried to view with "top" (a Linux program to show processes and their CPU/memory usage) if there was any program causing this. "Top" immediately threw an error which is usually a very bad sign. Because all exploits are very good at one thing: hiding themselves. And one of the ways they do that is by altering all programs that can bring their existence to light.

I immediately ran a few "scanners" (which is the reason I brought the webserver down for 10 minutes yesterday, because I needed the CPU cycles) and they all picked up that indeed a whole bunch of programs have been altered. There could be no doubt about it: the server has been comprimized.

The problem is that the recommended course of action action from this point is to backup as much data as possible, format the harddisk and reinstall everything. Because you really can't find out exactly what damage has been done. This is no easy task. For me it means I have to drive to another city to pickup my server, take it home and start the reinstallation process. This usually takes a few days because not only does it involve a reinstall of the OS, but all the security patches as well and testing the server. Taking the server down will make a few of my clients very unhappy, I know this for sure. But I don't mind that as much as that this is going to take time away which I usually spend with my 11-month old daughter, which BTW is something I'm unwilling to do (spend less time with my daughter). And that's the reason I'm considering to take down the forum. Not because I'm loosing money over this, or clients - but because this is beginning to affect my private life.

Do I know who did it? No. Do I think it's somebody from the PennocksSucks forum? I don't know. All I know is I've been receiving lots and lots of email lately from people who pointed me to threads there where they were discussing "some people" were up to "something". I never took that serious (so I never even visited that place) because I couldn't imagine people could actually get so worked up over an internet forum. But in the past few months, my log files did show an increase in hack attempts. Some ip-addresses closely matched those of banned members. Other ip-addresses were also found in the forum's access log, so I knew some of them frequently visited the forum. But since all of those attempts were really "script-kiddy" attempts, I wasn't worried and I didn't block those ip-addresses. Do I think Shaun did it (because he apparantly "said so" on the PennockSucks-forum). No. The guy has a bigger mouth than he has brains so he really isn't capable of doing such a thing. So don't direct your anger towards him. If you want to accuse him - or any other of the "playas" there - of anything, all you can accuse them of is that they are probably gloating over this. And in all honesty, if something like this happened to them I would be gloating a little bit too.

So who's to blame? I don't know yet. It could very well be a random attack. It could very well be a disgruntled ex-member. At this moment, I only blame myself for my own naiveness. Because that's going to cost me now. Dearly. I blame myself for not taking the "buzz" more seriously. If I had done so, I probably would have found out I had left a few parts of my server vulnerable.

I know Linux very well, but I have no experience with solving (as opposed to preventing) security related issues on it. Simply because it has not been necessary in the past 6 years. If you think you can help, you can contact me at CliffPennock@hotmail.com. Yes, I run my email through the same server, so I can't trust that anymore as well.

Update 11:50am
I'm still nowhere closer to finding out how they did it. All I know now is when it happened. The server was compromized on friday at 6pm local time. I've been trying to make backups of the forum and get that over to my computer. Unfortunately, I can't get decent speeds out of the server anymore and trying to pump a few Gbytes to my computer would take days...

Update 4:30pm
Well, I feel a little better now because there is a slight chance I can clean this up with little downtime. I might be able to at least secure the server so it can stay running while I prepare the new server. I figured out what it is they/he/she are trying to do so at least I now know what I'm battling. I might have caught this just in time, or at least, I hope so.

Also, I would like to express my sincere thanks to everyone who has shown their support. This is all very frustrating for me. Not because others are apparantly celebrating over this (because I couldn't care less really), but because this is taking so much time. And it feels like someone broke into my home. Your support really means a lot to me. It reminds me why I do all this - because for 99.9%, Fiero owners really are a great bunch of people. Whatever happens, I will make sure PFF stays alive - with or without me.

Update 6:10pm
14 hours of work later and I think I now have an ip-address of who did it. And if there's one thing I can do, is tie an ip-address to a name. So I do have a name right now. I also think this attack is unrelated to the forum (which is important [and a good thing] because I don't want people ripping each other a new one at coming car-shows ). My first priority now is to make sure the server is secure. I am on top of it, and I think I'm in control now.

Again, I would like to thank everyone for their help and support. I also would like to thank everyone who has made donations in the past day. However, when it turns out I don't need a new server just yet, I will refund it all. I know how tight funds are for some of you, and I don't want to take your hard-earned money when it's not needed. I'm pretty sure PayPal doesn't charge their fee if it's refunded within 30 days. This is non-negotiable. If after I refunded the money you still want to make a small donation, that's fine of course. But don't if you really can't.

If you need to shut down the servers, please do one thing before you take it down. Tell us who did the damage, so we can spend our PFF time meking life he77 for the scum bags.

I second that

Cliff, if worse comes to worse, could I suggest that you burn CD/s of the entire Forum History, and "Market" them as a complete set to offset the damage done to you.

Tom

Looks like your back stabbing has paid off.

After all they won't have anything to say sucks.

Cliff, as a means to communicate during what will hopefully be a short PFF shut down & as a way to keep PFF believers informed as to what's going on, may I suggest people monitor Fiero-Tech-Net? I know the guy who owns Tech-Net & I'm sure he won't care.

Archie