Parallel Testing Environment Compresses Development Cycles
In this case study
A Fortune 50 Financial Institution needed to accelerate migration of database, messaging and logging platforms to a new private cloud architecture. The Institution had aggressive transformational goals for the private cloud infrastructure:
- Develop and deploy multicloud infrastructure to four new North American data centers, consisting of private cloud offerings interconnected with public cloud offerings.
- Develop a private cloud infrastructure that is competitive in features and cost.
- Create an API-driven private cloud that would remove dependence on any single public cloud service provider.
- Migrate platforms application and systems to the new multicloud architecture.
- Refactor existing applications to the new private stack.
- Use AIOps to dynamically control the placement of workloads between private and public cloud infrastructures.
- Transform teams to use modern engineering practices.
The client was struggling with trying to upskill staff and prepare the platforms for migration while at the same time architect the private cloud solution.
The client engaged WWT Application Services to help. The engagement had several objectives, all of which needed to be accomplished simultaneously. WWT was asked to upskill the client's customer engineering teams; create build automation using Infrastructure as Code (IaC); and use the build automation to repetitively build/destroy the environment to conduct performance testing.
Initially, WWT engineers created a prototype of the client's production private cloud in the our Advanced Technology Center (ATC). The prototype was dubbed the "Migration Factory." This environment mirrored the "to be" environment of the new data centers. It contained:
- x86 hardware to mimic production.
- NetApp Solidfire storage arrays.
- VMware VCF.
- HashiCorp Terraform.
- Gitlab CI/CD toolset.
Within the Migration Factory, the delivery of build automation, IaC, performance testing and upskilling of the client's staff was organized into weekly sprints. Sprints are short, time-boxed periods when a development team attempts to complete an agreed set of priorities.
Each sprint was executed by a co-development team of WWT and client engineers. The team was led by a WWT DevOps engineer who co-developed build automation, IaC and executed the database and storage testing along with client engineering staff.
Work was performed based on priorities established by the client during a weekly live demonstration meeting. The co-development team used WWT Agile and DevOps best practices to execute weekly Agile Sprints to achieve the Institution's objectives as described below:
- Perform functionality and performance testing of database, messaging and logging platforms.
- Perform functionality and performance testing of storage options: Local, VSAN and Solidfire.
- Maintain and expand current Tool Chain as required to support testing activities.
- Maintain testing of existing hardware and software configuration in ATC as required to support testing activities.
- Provide mentorship for client teams involved in agile/infrastructure development activities.
Agile and DevOps best practices used
- Use of DevOps Tool chains and automation.
- Use of an integrated toolchain that aligns with the Institution's toolchain.
- Elements are virtualized and implemented as disposable ephemeral infrastructure (IaC).
Weekly live demonstration meeting activities
- Review Kanban Board to determine status of features.
- Discuss, define and agree to add new features to the Kanban Board.
- Prioritize backlog of features to determine priorities for the next sprint.
- Conduct demonstration of features developed during the previous week's sprint.
- During the demonstration, test results may be presented with visualization tools such as Splunk or other reporting tools.
- At the end of each demonstration, the backlog of features is re-prioritized, and the Kanban Board is updated for the next sprint.
- Daily Standup Meetings: Short meetings utilized by engineers to identify potential challenges as well as to coordinate efforts to resolve difficult and/or time-consuming issues. These meetings promote close working relationships, which results in a higher rate of knowledge transfer.
- Daily Remote Pairing sessions: Pair programming is an agile software development technique in which two engineers work together at one workstation. One, the driver, writes code while the other, the observer or navigator, reviews each line of code as it is typed in. Webex and other collaboration tools are used to form a remote pairing team.
All database, messaging and logging platform teams developed build automation in the Migration Factory and used the build automation to test the performance of the respective platforms. The client was able to use this solution to create a parallel testing environment and increase capacity for testing that resulted in shaving many months from the development cycle.
At the end of the engagement, the client was able to upskill its teams and move its testing activities from the ATC environment into its internal streamlined development pipeline.
The client is currently on track to migrate all database, messaging and logging platforms to its new private cloud architecture.