A funny thing happened on the way to reviewing my logs

At work we're in the process of implementing a SIEM (Security Information Event Management) system. I'll leave the vendor nameless for the moment but they have a reputation of making most everything harder than it needs to be. Until that time all logs have to be reviewed manually and obviously that means that they are not reviewed in real time. I have others that monitor most of the logs but I monitor our IPS logs from the UTM device. Usually I review them each morning when I come in  but last week I didn't get a change to so yesterday I was playing catchup.

As I reviewed them I noticed something new. There were lots of  entries where XSS had been stopped. At first I was really worried thinking that if there were that many attempts then that must mean that there was a vulnerability that someone found and now they were trying to exploit it. Of course I had no real way of knowing if they had been successful or not. At least not w/o more research. As I looked into it more I noticed that it only happened on one day and that it was only for a short period of time. Then I started to do a whois lookup on the IP address and discovered that it was me. I had been doing some testing on a new feature of our web site and part of that was for XSS vulnerabilities.

It's good to know that the IPS caught this and stopped it before it got the the server itself. That makes me feel a little better but I sure will be glad when the SIEM implementation is complete so I can see these things in real time and have a better grasp on what is going on.

Staying Fresh

Rebecca Herold has a good post on her blog about keeping your security, privacy, or compliance program fresh. She makes a good analogy between how your program can slowly become ineffective over time due to lack of attention and how running shoes can slowly become less effective over time. I can't relate to that because I bought my new running shoes in April and they haven't had as many miles put on them as hers gets in one day. Try as I may I just can't get into a running frame of mind.

I've seen fist hand how programs start strong and slowly erode or die over time. They don't get the TLC that they need to stay alive. They are put in place to satisfy a audit or a new boss and then they end up in a closet or on a shelf only to be given a passing glance from time to time.

I recently did a review and update of security policies for a company. What they had was between 4 and 8 years old. They had been created (mostly just changed the company name on a template) and filed away. As I looked over them I started asking questions about them. Is this really what is done? Where is the ??? to back this policy up? Where are the ??? that this policy states is happening? Blank Stares and hidden smiles met me. They weren't being followed. They were just there to satisfy a whim.

These documents and programs are living. They are meant to be reviewed regularly, followed consistently and changed as needed. They are not static documents that are just to satisfy an audit. When I create a policy program or a security plan I make sure to write it in such a way that those who are entrusted with it know that it is a living document. I include regular review schedules and then I encourage those who are entrusted with them to go ahead and put reminders in their calendars to review them. I can't make them do it unless they report directly to me, but I can try to make it easy for them to do.

Another area that gets ignored is log review. Most people hate to review logs. Especially if they don't have a SIM, SEM or some other method for automating it. I've done it before. I've had to sift through thousands of entries to try and find the "bad" stuff. It's no fun. Unfortunately it has to be done and you need to be able to prove that you are doing it. If your policy says that you are doing it then the auditors are going to want to see proof. How many times have you or someone you know spent a day or two prior to an audit "falsifying" log reports. Going through and checking off that they were checked when they haven't been looked at in days, weeks or months.

It's important to remember that these things are crucial to the success of your information security program. If you let them get sick or die then your program will do the same. Security Professionals need to follow the policies and those in management need to ensure that they are being followed. Those who are tasked with keeping the policies or program alive need to be proactive in doing so. Don't wait until the last minute and try emergency CPR. If you will schedule a little time weekly or monthly to check on them then they will stay healthy and your program will be more successful.

The Importance of Logs (and looking at them)

This post was prompted by Dr. Anton Chuvakin and his post on ignoring logs. I've mentioned this story before briefly here but felt that more detail would be beneficial to those debating the merit of reviewing log files. There may not be anything more boring in Security than reviewing log files, but there also may not be much that is more important.

A few years ago I did a stint as a Consultant for a small Kentucky company. Shortly after I started a customer called with an emergency. The guy who worked this account was on vacation so I went to investigate the problem. They were having problems authenticating users to the domain and therefore many resources were unreachable. I asked the standard questions about what had changed recently or had anything new been added to the network. They assured me that nothing had changed or been added. After having them show me exactly what they were doing and seeing what was happening I started looking at the DC to see what I could find. In reviewing the Security logs I noticed that a new administrator privileged account had been created 2 weeks earlier. After waiting 2 weeks to ensure that the account had not been discovered the hacker then proceeded to load file sharing software on the server and copies of 4 of the latest movies (2 of them weren't even in theaters yet). Every time the P2P application ran it disrupted AD on the server and caused users to lose their credentials.

How did this happen? There were at least 2 MAJOR mistakes made here. First, the server, which was the Global Catalog and Primary Active Directory server, was dual homed and one NIC was on the internal network and the other NIC was on the Internet so partners could get to it for FTP transfers. I won't even comment on that. The second problem was that they were not monitoring logs. They did a lot of network performance monitoring and WAN connectivity monitoring. Things that look cool on graphs and have a little sexiness to them, but they ignored the mundane, boring task of log monitoring. Had they been doing so they would have noticed the new administrator account and deleted it. Then they could have investigated how it happened and closed up the hole that the truck drove through.

Luckily this turned out to be just a big nuisance. I was able to repair the damage, remove the P2P app, restore everything and get them back up and running in about 4 hours. Nothing else seemed to have gone awry during this. My investigation didn't turn up any other mischief. Needless to say the first order of business after that was to build them a new FTP server that sat on the DMZ all by itself. Then we implemented a log monitoring program to ensure that this didn't happen again. I stayed with the consulting firm for a year after that and no other issues were reported so either they were successful in keeping the bad guys out or too embarrassed to let it be known that it happened again.