Pushing without testing

My first week at work was pretty exciting. Several things happened that allowed me to jump right in and start putting my training to work. I'm not going to go into any details obviously, but there is one incident in particular that I want to talk about.

Our network is quiet extensive. It seems to have been well thought out in it's design and although security wasn't always a top priority they have done a pretty good job of implementing policies technologies to mitigate threats and to "shore things up". We have several partner networks that connect back to various segments of our network and one of them went awry this week. It wasn't exactly a security issue but easily could have been.

The partner, which maintains a important aspect of our business, pushed out an upgrade and it caused all sorts of problems. Fortunately this segment is completely separated from our core network and it is not accessible from the Internet in anyway. What if it wasn't though? What if we had an Internet facing interface that was affected by this. What if we didn't have an air gap between this network and our core?

The potential for a breach would have been very great. Either from the Internet or from the partner network. This just goes to show that diligence pays off in designing security for your network. I know many small and medium sized companies that would not have been so diligent in ensuring that the design of this was secure and that the proper controls were in place. Why? Lack of staff, knowledge and money.

How could this have been averted in our case? Obviously the vendor needed to do more testing before pushing out the upgrade. The biggest thing is that they pushed it all at once. Every location was upgraded at the same time so the problem affected all locations. If they had pushed it to one or two locations and then let it run for a day they would have discovered the problem and rolled back, fixed it and averted a big problem.

Other than that it was a quiet week. The other issues mentioned earlier were nothing compared to this. They just required some changes in the way a couple of things were configured and in how a couple of things were done. It does feel good to make a difference on your first week. Especially when it doesn't require me to be up all night working on something that broke. I think I'm gonna like this. :)