Continuous Deployment Process

Contact centers need extra features and new channels of interaction, but application development can be both resource-intensive and disruptive to a business. We designed our latest cloud solution, PureCloud℠, so customers gain enhancements and new platform features without expending resources or adding risk to business operations. With continuous deployment, changes roll out weekly with minimal user disruption. To get access to roadmap features, customers need not wait for large quarterly releases.

Since PureCloud went GA last July, Interactive has completed over 700 code pushes to production, addressing everything from bug fixes, to feature enhancements, to all new functionality (Release Notes). That is an average of 27 code pushes a week, while supporting live customers, adding new customers, and maintaining an availability of over 99.99% globally. See for yourself here!

How do we do it?

deployment-process Rigorous testing requiring multiple levels and types of testing are baked into release cycles. PureCloud architecture is composed of small independent microservices running on individual Amazon Machine Instances (AMI) in Amazon Web Services (AWS) data centers worldwide. Each microservice communicates with each other using stateless REST APIs.

 

Continuous delivery of features starts with the development of any new code introduced to the PureCloud platform. Our deployment process is based on an immutable architecture, which deploys code on new AMIs instead of updating old ones then having both run in parallel until the new AMIs are validated. Every update or feature to the platform follows a predefined path through our development, testing, staging, and production environments. Prior to production, we add new groups of people and testing requirements to each environment.

Application development
Each microservice has its own code repository built on PureCloud’s base AMI approved by our Development Operations and Security teams. It’s important to note that the base AMI is updated regularly with the latest security patches and operationally updated every 30 days or less. When a development team is ready to commit a new code, the Jenkins build tool executes an automated deployment job. We use Asgard to deploy new builds to the development environment. For more information about security, see Security and compliance. For more details on the process of code promotion to production:

Development environment
Jenkins deploys new microservices running the new code in parallel with old microservices (A + B mode). Automated tests begin immediately on the new microservices. The following types of test cases are required depending on build requirements: new test cases, Base Functionality Tests (BFT), security scans, and integration tests. Criteria for success at this stage is the responsibility of our development staff. Successful test results result in the replacement of new microservices in place of the old ones and an automatic push to our testing environment. Any failed automation test results in the newly deployed microservices being destroyed and development addressing issues in a new build candidate.

Platform testing
The development environment is also where we test the PureCloud platform as a whole. PureCloud is designed with the idea of embracing failure, meaning we have designed the platform with the understanding that things break. In our development environment, we test our ability to handle the unexpected with the deployment of Chaos Monkey. Chaos Monkey randomly terminates whole groups of microservices and tests our platform’s ability to provide service despite failure. This tool runs independent of any specific code deployments. Issues attributed to the inability to withstand Chaos Monkey are flagged and remedied by the development of new updates to the platform.

Testing environment
A replication of A + B services running in parallel occurs in the testing environment. More testing occurs, such as functional testing, regression suites, performance and scalability tests, along with overall integration testing with other related features. Criteria for success at this stage is owned by our testing staff. New microservices that pass tests are automatically pushed to our staging environment and replace existing microservices. Newly deployed microservices that fail an automation test are destroyed so that the development team can address issues in a new build candidate.

Staging environment
The staging environment is where the human element of testing begins. Interactive Intelligence refers to this commonly as “eating our own dog food.” The global Interactive Intelligence staff use this environment in production. Staff report issues directly to development via the PureCloud user interfaces (browser client or desktop application) by clicking Help > Report an issue. Time in production depends on the nature of the feature, update, or bug fix. Criteria for success at this stage are owned by our product management staff. Code ready for use by customers follows the change management control process led by our Release Management team.

Production environment
New microservices run in parallel with old services until testing is completed successfully. We deploy services approved for production using the same method that the development, testing, and staging environments use. But testing does not end when production begins. We perform penetration tests annually, and further apply automation test suites. PureCloud’s operational teams (Development Operations, Security, and Support) work together to ensure that customers have success using the service. In the event of a problem, customers report issues directly to Support and Development staff by means of a support ticket or the ‘Report an issue’ option.

Rollback procedure
In cases where code must be reverted for any reason, a programmatic deployment of the most recent version of the microservice triggers to restore functionality. It’s important to note that PureCloud is designed so that no single microservice can destabilize the overall platform.

Zachary Hinkle

Zachary Hinkle

Zach Hinkle is a Solutions Program Manager in the Product Management organization in PureCloud. He is a 10-year veteran with Genesys Interactive Intelligence with roles in the Support organization, Strategic Initiatives Group, and most recently in the Solutions group for PureConnect (CaaS).