Cybersecurity Patch Management Best Practices

Much of the critical (and dangerous) industrial infrastructure in the developed world is controlled and protected by computer-based automation systems. A malicious compromise of those systems gives the potential for serious harm to plant workers, the public, and the environment.

Safety is one of the most important differences between traditional IT and plant operations. Rebooting OT systems due to software updates can cause system crashes and create a potential for worker injury and loss of production.

• IT systems tend to be replaced every few years and generally keep pace with technology. Plant automation systems operate for ten or more years and often incorporate old hardware and software.
• Patching and updating software is done almost daily on IT systems. On the OT side, plant systems often never get patched or updated due to the risk of introducing a problem (“if it isn’t broken don’t fix it!”) and generally there is no test system to validate the updates.
• To a certain degree DCS and SCADA systems look like conventional IT systems except for some unusual peripheral devices and application programs. In fact, some SCADA vendors now mainly sell software and allow customers to purchase and stage their own servers, PCs, LANs (and commercial software, including OS and networking.)

For these reasons some organizations have put their IT departments in charge of supporting their computer-based industrial automation systems. This is not always the best approach due to differences in philosophy between traditional IT best practices and the operational requirements of an industrial plant or facility.

Moreover, patching in the OT environment can be an expensive method. If the risk can be reduced to an acceptable level by applying alternative controls – meaning if the attackers can be prevented from reaching the vulnerable assets – then the cost or effort of patching and applying alternative controls need to be compared to decide which approach is best.

It is not feasible to patch all the OT assets, thus, it is recommended to patch smartly.

STEP  1:

A comprehensive inventory of all software, firmware, and hardware within the OT environment, including all the assets from the industrial demilitarized zone (level 3.5) to the cell/area zone (level 2-0) in the ISA/IEC-62443 Purdue model, is a critical piece of any OT patch management process. Once there is a clear picture of what is present, it will be easier to compare the known vulnerabilities to the inventory to quickly discover which patches matter to the OT environment.

STEP  2:

To assign criticality to the OT asset, a system for assigning criticality scores needs to be established. This may already exist due to the regulation of the safety system. The criticality needs to be assigned considering business Impact i.e., the impact of lost accessibility, reliability, integrity, etc. to the business safety, profitability, etc. 

STEP  3:

It is not possible to deploy all the patches in all the OT assets at the same time. Also, it is not possible to patch one by one. It is recommended to prioritize patch deployment that is specially designed for the OT environment. 

STEP  4:

Run a diagnostics test to confirm the asset(s) has the required resources available for installing updates.

STEP  5:

It is recommended to have:

1. Baseline patches: Getting a baseline from all the OT systems in the OT environment will provide a starting point for comparing any changes in the future. Considering the patch management approach, the baseline should provide the patches that should be installed when a new system is commissioned. It is understood that the baseline needs to be updated regularly as new patches are released.
2. Record installed patches: OT asset owners need to ensure that the baseline and changes are recorded. It is crucial to record the current patch status; in other words, the list of all the patches currently installed in each asset. As the new patches are deployed, they need to be recorded as well. if any patch is not installed, then the alternative applied controls for exempted patches need to be recorded as well. This would help to audit if any patches that are not vendor-approved or not tested are pushed to the OT assets.
3. Review installed patches: The changes in the OT environment need to be tested and reviewed regularly before and after the changes. With that said, the patches need to be tested and reviewed before and after patch installations.
4. Document the patch process: It is recommended to have a documented change management process. This process should not be done as the last thing but should in parallel as the changes are made to refer to it for future troubleshooting and forensics if needed.
5. Have a rollback plan: Sometimes, the vendor-approved/supplied and end-user tested patch may crash the critical OT asset because the testing environment can be different from production environment. It is always recommended to have a rollback plan with complete tested backups before patching critical OT assets.

STEP  6:

The patch management policy should at a minimum include:

1. Policy for defining a scanning schedule
2. Policy for categorizing and separating patches
3. Policy for prioritizing and deploying patches
4. Policy for exempted patches

Polytron applies and adjusts these steps to meet the needs of each client and project. Our approach to security applies a design methodology for your network requirements to develop a physical framework to align to support your manufacturing strategy. Learn more about patch management and reducing risk with cybersecurity with the on-demand workshop here.