One of the ongoing services that Palante offers is regular updates of virtual private servers (VPS). VPS run a lot of the critical infrastructure on which our client’s websites and databases depend. Updates to these servers provide critical security patches and are really important to the overall security and uptime of an operation.
We thought it’d be a good idea to provide a brief overview of our maintenance process from a technical standpoint, as well as some context for why we think it’s important. Some folks find VPS updates tedious and risky, given that there’s always the risk of downtime after an update. We, however, prioritize the security of our servers over their uptimes considering the importance of our clients’ work.
Technical Overview
We maintain about 20 servers for clients, all of which are Debian or Ubuntu Linux. We subscribe to Debian and Ubuntu security updates through our ticketing system, so notifications for new updates are automatically assigned to one person (Jamila!) on the team. Jamila runs our update process as these notifications come in, so about 1-2 times a week!
Because all of the Content Management Systems (CMS) and Content Records Management (CRM) we and our clients use are based on PHP, we have decided to use the PHP repositories managed by Ondřej Surý at deb.sury.org in order to have all the latest versions of PHP available on our Debian and Ubuntu servers. CMS and CRMs are designed to work best with the newest PHP versions that those projects list as compatible, so sourcing the newest PHP code can result in a faster system with fewer issues. More information about deb.sury.org packages is available at the project F.A.Q.
We use Ansible to update all of our servers at once. You can read more about how Ansible works here, but in short it is an open source tool that provides enterprise-level automation for a range of IT tasks. Instead of running update commands on each of our servers, we can enter one command through Ansible, which then runs it on multiple servers simultaneously. It connects to these servers over SSH by default.
For VPS updates, we run this command:
ansible VPS -m apt -a "upgrade=yes update_cache=yes" -b -K
VPS in this command is the group of servers that we want to update. We then review the Ansible output to make sure there’s no ERROR messages. We then typically leave the servers to reboot if needed after the update. Rather than waiting for each of them to come back up, we rely on Icinga, our monitoring system, to tell us if something doesn’t come back online after the update.
We like to reboot servers after hours for Linux kernel updates, which are more likely to result in downtime than other updates.
Why do we do it like this?
Regular VPS updates are an easy way to maintain the security of a system.
Security vulnerabilities are found in all different kinds of software frequently used on servers. Some of the fundamental underlying programs such as OpenSSH, Curl, and PHP have all had major vulnerabilities in the last few years that required updates as soon as possible.
In our experience, part of the recommended fix for Debian and Ubuntu security issues is almost always to update the software. The operating system is the most foundational piece of software on a server, so it’s important to keep that up to date.
The Debian Linux project is well trusted for its security and stability, so Debian updates to Linux are even more important to run in case of vulnerabilities. In the rare case that there’s a problem with a Debian update, the developers are able to quickly fix it and we can apply the fix as soon as possible.