The Importance of Logs (and looking at them)

This post was prompted by Dr. Anton Chuvakin and his post on ignoring logs. I've mentioned this story before briefly here but felt that more detail would be beneficial to those debating the merit of reviewing log files. There may not be anything more boring in Security than reviewing log files, but there also may not be much that is more important.

A few years ago I did a stint as a Consultant for a small Kentucky company. Shortly after I started a customer called with an emergency. The guy who worked this account was on vacation so I went to investigate the problem. They were having problems authenticating users to the domain and therefore many resources were unreachable. I asked the standard questions about what had changed recently or had anything new been added to the network. They assured me that nothing had changed or been added. After having them show me exactly what they were doing and seeing what was happening I started looking at the DC to see what I could find. In reviewing the Security logs I noticed that a new administrator privileged account had been created 2 weeks earlier. After waiting 2 weeks to ensure that the account had not been discovered the hacker then proceeded to load file sharing software on the server and copies of 4 of the latest movies (2 of them weren't even in theaters yet). Every time the P2P application ran it disrupted AD on the server and caused users to lose their credentials.

How did this happen? There were at least 2 MAJOR mistakes made here. First, the server, which was the Global Catalog and Primary Active Directory server, was dual homed and one NIC was on the internal network and the other NIC was on the Internet so partners could get to it for FTP transfers. I won't even comment on that. The second problem was that they were not monitoring logs. They did a lot of network performance monitoring and WAN connectivity monitoring. Things that look cool on graphs and have a little sexiness to them, but they ignored the mundane, boring task of log monitoring. Had they been doing so they would have noticed the new administrator account and deleted it. Then they could have investigated how it happened and closed up the hole that the truck drove through.

Luckily this turned out to be just a big nuisance. I was able to repair the damage, remove the P2P app, restore everything and get them back up and running in about 4 hours. Nothing else seemed to have gone awry during this. My investigation didn't turn up any other mischief. Needless to say the first order of business after that was to build them a new FTP server that sat on the DMZ all by itself. Then we implemented a log monitoring program to ensure that this didn't happen again. I stayed with the consulting firm for a year after that and no other issues were reported so either they were successful in keeping the bad guys out or too embarrassed to let it be known that it happened again.