How high up in the list of priorities should your patching be? Tales of woe and gnashing of teeth stopping this endeavour? Or heroes on the front line?

We all know about “Patch Tuesday”, Microsoft release date for patches to it’s operating system, .net and other software. We all have plans around this activity and many of us have processes set up to facilitate these updates. Well, ok, most of us do.

However your ICT estate does not comprise of MS alone. You will have other kit that has software running it. You might even have linux servers as well as switches and routers available from a variety of vendors.

Do you know if these are up to date?

I recall a conversation regarding patching with a group of data centre techies and managers. We addressed the MS side quite easily and with little headache for the most part. However, after an audit, it was apparent that the linux servers had not been patched in years, and by years I mean more than six or seven. Mainly because it was felt that they didn’t need patching when they were rolled out, linux being considered more secure.

However patching is not only about killing exploits. It is also about increasing the efficiency of the software as well as hardware. So these servers were running flavours of linux that were very much out of date, that in some cases had vulnerabilities. All were impacted in terms of running the latest revision of firmware, drivers, and OS.

Then we have the other parts of the estate – switches, routers, firewalls, load balancers, storage devices. Certainly all these can be compromised and result in embarrassing the IT department. As IT professionals we certainly don’t want that, right?

So how do you weigh the risks? There are costs involved, obviously, as well as possible impact on production. I recall one organisation running a update on their back up software that resulted in a pretty major catastrophe by taking the storage array off line.

Keeping up to date with your vendor’s updates is vital as is understanding what the update will do both during the up date and after. Will it break applications for example (with storage), will the device reboot and leave your network unprotected (firewall)? Hopefully the answer is no. However if there is one thing that you should do, if at all possible is test the updates first before releasing. Unless it is a hugely critical security update (and even then I seriously suggest testing before release, and have a back up plan to hand when it goes wrong) you can take time to do this.

If you have to wait for a suitable window in which to do an update then that is ideal to also do testing. Testing tells you if the update is stable, it gives you an opportunity to learn about any changes and certainly should tell you if you can expect any major issues.

Sure patching is a pain and no one really likes to do it because the pain it can cause but as indicated above there are steps that do help reduce or limit that pain. However you need to weigh up the pro’s and con’s as a business.

Ultimately you are protecting the business.

Advertisements

Messy! Why you should never have automated updates on production systems

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2060190

A pre-release version of vSphere Replication has been made available and if you have Automatic Check and Install updates selected the vSphere Replication appliance will automatically upgrade to version 5.5.
vCenter Update Manager may also have this version downloaded. VMware recommends not installing the vSphere 5.5 Replication version in your 5.1 environment.

I’m not blaming VM Ware…just dispensing some well learned experience. What I am saying is that when looking at new software (including updates) to put on your systems you need to be aware of unintended consequences. Here its an upgrade to a new version of vSphere Replication. Which will, by all accounts, leave you in a world of pain. This VM Ware technology is used to provide business continuity by replicating virtual machines.

vSphere Replication 5.5 is not compatible with vSphere 5.1, which results in an inability to manage vSphere Replication and initiate failover.

Using proper controls when releasing software and updates does tend to mitigate these kinds of issues. You should be looking to test the new software or update on a non-production environment to understand how it installs and to familiarise yourself with any interface and options or, in the case of updates, how the software has changed. You should then look to stage it in an environment that replicates your systems as close as possible to look for any issues with applications, operating systems, networks and storage. You also look to see how you roll back in the event of any serious issues, just in case.

Once you’ve done these things you should look to write up your findings and roll back techniques as part of your release management strategy (if you don’t have one…perhaps you should). If all is good and approval given in a change management meeting (aka Change Advisory Board) you go ahead and release into the production environment.

It really is not rocket science and saves you from major headaches.