Healthcare System Tests Major UC Upgrade to De-risk Downtime
In this case study
Challenge
Failed communication upgrades with their resulting downtime and system unavailability can be devastating in a 24/7 environment. When it was time for a large healthcare system to upgrade their Cisco Unified Communications (UC) system, they knew they needed to validate their upgrade; they'd experienced failed upgrades in the past, and this upgrade needed to be a success.
The challenge they faced was a distributed system with enough latency at various points to adversely affect a normal upgrade process. They had 16 servers spread across 10 WAN environments, and wanted to be sure that any problems or issues were found out well before trying to perform the upgrade.
Solution
To ensure that their upgrade plan would be able to account for all the unique quirks of their systems, we leveraged the capabilities of our Advanced Technology Center (ATC) to create a fully functional simulation of the healthcare system's environment.
Diagram of the solution.
We were able to quickly use existing infrastructure and toolsets within the ATC to simulate their servers, connections and WAN latency, all of which would have been prohibitively expensive to try and set up from scratch at their own facilitates. Once we had an environment to represent their systems, we got to work reviewing their current UC setup to ensure that they had a designated primary and backup subscriber and that none of the backup subscribers had registrations.
Once we understood the current-state of their system, we got to work proving out various upgrade paths:
- Upgrade publisher and reboot when complete
- Upgrade secondary subscribers and reboot when complete
- Upgrade primary subscribers with special consideration to service impacting actions
- Reboot a test group of phones and observe TFTP update and registration to new cluster
- Stop CM service on primary subscriber and allow all phones and gateways to perform TFTP updates and register to new cluster version
- Perform upgrade on primary subscribers
During these tests, we identified several bugs that would have greatly impacted the system in the course of an upgrade. Since we caught them up front, we could provide a detailed upgrade plan so they could avoid those bugs during the live run of the upgrade.
Conclusion
The WWT proof of concept helped the healthcare provider identify and solve any potential issues caused by the size and geographical distribution of their network during the upgrade. We helped them build an exact replication of their environment, run through the upgrade process and document all the steps to ensure success when they performed the upgrade themselves.
Equipped with a detailed upgrade plan that included steps, timelines and tips, the healthcare provider was able to successfully execute the upgrade, achieving their goal of minimal downtime and service interruptions.
Business outcomes resulting from the POC
- Risk Reduction: By thoroughly vetting the upgrade process and creating a detailed upgrade plan, the healthcare organization dramatically reduced risks associated with the upgrade.
- Confidence in Decisions: Seeing the tests of the upgrade in a simulation of their environment gave the organization greater confidence that the upgrade would be successful.
- Process Improvements: Taking the time to test, retest and test again created a number of improvements to the organization's upgrade process, ultimately resulting in the successful UC upgrade.
Check out the CUCM and Collaboration workshop for additional information about how to upgrade your system.