Ensure business continuity with change management

Changes to systems and networks happen every day. When implemented, changes usually come with some risk of system failure. They can also inadvertently weaken security. A documented, policy-driven change management process helps reduce risks associated with change.

When we make a change to a system or network, we face the possibility that security may be weakened or that the risk of business process interruption increases. This includes increasing risk to data unexpectedly crossing trust boundaries.

A trust boundary exists between two network segments or two systems with different trust levels. A trust level is determined by how well the infrastructure and software is hardened and monitored. For example, a system handling payment card information (PCI) might possess a higher trust level than a file server. In that case, if data passes from the PCI system to the file server, they cross a trust boundary.

And there is always the risk that the changes to a system or network device can cause infrastructure or software failure. Infrastructure failure not only affects the changed business process, but it can also affect the organization’s ability to execute downstream business processes: processes relying on input from the failed process.

An example

Let’s walk through a simple example of how adding a new system can increase risk. In Figure A, we see a segmented network. Security has done a good job of ensuring the backup VLAN 20 (red) is separated from the general business VLAN 10 (yellow). An access control list helps keep the unauthorized from backed up information.

A new project is about to implement a retail sales network, as shown in Figure B, including accepting customer payment cards. This new network is also segmented as VLAN 30 (green), presumably preventing payment card information access by anyone or anything on VLAN 10.

However, one step in the implementation process is to ensure the retail network could print to a shared printer: a cost management decision. This breaks security by allowing PCI to cross a trust boundary, between VLAN 10 and VLAN 30, that should be insurmountable.

If this change runs through a change management process, the printing risk issue would likely be identified and an adjustment made. If no change management process exists, the risk to payment card data would likely be higher than security or management expected.

The Solution

Setting up a change management process begins with a policy. The policy should clearly state that no change may be made to production infrastructure or systems without passing through the change management process. As we discuss later, this means the process must include how to make changes during business continuity events.

Oversight: The oversight body of change management is often known as the change advisory board (CAB). The board is responsible for developing change procedures and making decisions regarding high risk changes.

Some believe all changes should go through the CAB. However, this can unnecessarily slow the change process. So many security professionals believe the CAB should only review changes that have a higher than normal probability to interrupt a critical business process: either via unavailability or by data compromise. Other changes are reviewed by representatives of key stakeholder teams.

In addition to the the CAB, the day-to-day change management process must be assigned to a responsible manager. In my case, I was responsible for the change process as the director of information security. My team received change requests and ensured the correct sign offs were obtained.

The process: The change process should always include three phases: submission, change approvals, and a decision point at which the change management team decides whether or not the change should go before the CAB.

Finally, the organization must identify who must sign off on changes to ensure the proper reviews are completed. Reviewers normally include server engineering, network engineering, software development, technical operations, and security. The important takeaway is to include every team necessary to ensure any availability or security risk is addressed before implementing the change.

The change process begins with submission of a change request to the team responsible for managing the change management process. Change request documents include:

• A description of the change
• A list of all systems and network devices affected, including relevant network and data flow diagrams
• A detailed implementation plan
• A detailed back out plan for use if things do not go well during implementation
• A description of the potential risk associated with the change

The change team ensures copies of the change request go to all signatories. In many cases, the approval process is automated. We used Microsoft SharePoint and a proprietary workflow process.

Expedited changes: The standard change process should not stand in the way of recovery from a business continuity event. Such changes should be made quickly, yet subject to review after business process recovery. “Quickly” does not mean the response team fails to document the change enough for later review and possible removal.

The Final Word

Change management is not an option. It is an important piece of business interruption prevention and helps ensure security risk does not drift up during projects and day-to-day activities.