I’m subscribed to the CentOS-announce mailing list and I’ve read the definitions of the various severity tags such as Critical, Important, Moderate, Low, BugFix, etc. but I’m still unclear as to what really needs to go out and what can wait (as I need to justify an interruption to our regular work to get the updates out there).
So far I’ve tried to get critical updates applied in our production environments as quickly as possible.
What is the normal schedule of applying these updates? They happen so frequently clearly we can’t apply them all quickly because they need to be tested to ensure they haven’t broken our systems.
For context, we mainly run a website and there’s no extra users on the system so I’m typically not worried about ‘local exploits’ as much as I’m worried about remote exploits. But I feel I’m still in the dark about how best to apply these updates.
I manage a number of different web sites for various clients, and this is generally what I do.
First, the top priority for security updates should be the web application itself. The vast majority of attacks will be targeting your web site and the code that runs it, and this needs to be kept secure. If you use off-the-shelf web applications, this is one place where I would even consider automatic updates, but in no case would I let a security update for something like WordPress or Drupal age more than 24 hours before testing it and most likely rolling it out.
If your web app is custom built, ensure that your developers are keeping on top of security issues, to whatever extent that you can do so. This is a scenario in which your organization should be doing DevOps to ensure that the web developers and IT are working well together and that issues are resolved timely.
After that, the next critical updates I consider are those that “go viral” and that you hear about on national news, such as Heartbleed, POODLE, etc., as well as updates to anything in the critical path, like nginx and PHP for Drupal and WordPress sites. I apply updates for these as soon as they’re available. I also am on the mailing lists for the upstream packages (e.g. I am subscribed to openssl-announce) so that I get notification for really important things as soon as possible.
Next, I apply updates to every public facing server (web front ends, load balancers, etc.), and every supporting server (databases, etc.) once a month. This includes any leftover security and bug fix updates. In many organizations this is done quarterly across the board, but it’s my opinion that public web sites, especially those doing e-commerce, should not be left to rot for that long. I almost always do this the weekend after Microsoft’s patch Tuesday, which is almost always enough time to hear about bad Microsoft updates (though I mostly run Linux systems, it’s easier to have one weekend to update everything).
Finally, I sit back, relax and wait to see what breaks. Despite your best efforts at testing updates, something will eventually go wrong. Watch your monitoring system. If something breaks that isn’t being monitored, fix it and then start monitoring it. Be prepared to rollback (
yum history undo is helpful).
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.