We're proud to provide DevOps services.

We enjoy working as a team to deliver on our full service commitments.

We deliver a range of expertise to provide solutions for things like integrating with SalesForce to pull in membership data, integrating for set automated set up of group based content access on their subscriptions driven web service. Another example, moving them to a highly available, scalable cloud based infrastructure with Apache Solr and high performance caching technologies.

The main StoryCorps Archive access point was built on a robust Drupal platform consisting of over 60,000 interview records and approximately 27TB of associated metadata, WAVs, MP3s, JPGs and PDFs. The StoryCorps Archive platform connected with several critical business systems and performed around-­the-­clock ingests from their on­site storage arrays to the Drupal system, via rsync. StoryCorps was looking for a trusted and capable firm to migrate their entire Archive— including the website, connected services, and media— from their single-­server host to a combination of Amazon Web Services (AWS), EC2, S3 and Glacier.

Last Call Media performed a thorough analysis and audit of all StoryCorps’ source data prior to and following the massive migration. We worked closely with StoryCorps’ internal Digital Team and engineering consultants to design, test, implement, and ultimately maintain the new AWS server infrastructure.

In the discovery phase, we reviewed the marketplace to find the most appropriate CDN for the State’s use case, balancing security, performance, and cost considerations. Ultimately, Cloudflare was selected as the best fit because of its extensive firewall and DDOS protections, and granular cache control using “Cache Tags,” which have the potential to boost performance for the constituents and reduce the risk of site instability.  

The first, and perhaps most critical concern we addressed in the course of this project was that the CDN needed to be resilient, serving pages even if the site itself was not functioning properly. For example, the development team does code releases periodically that take the backend of the site completely offline, but constituents still need to be able to access content during this time. To meet this requirement, we adjusted the site’s caching headers to include directives to serve cached responses in the event of an error response received from the origin.  As a result, constituents are able to access the majority of Mass.gov, even if a catastrophic event takes the web servers completely offline.

As a government website, Mass.gov is always at risk of attack from malicious actors. To mitigate this risk, Last Call Media undertook extensive configuration and testing of Cloudflare’s various security features, including the Web Application Firewall (WAF), DDOS protections, and custom firewall rules. We had a few hiccups along the way with configuring the security features (at one point, content authors were receiving CAPTCHA verifications when submitting their changes), but were ultimately able to work through these issues to dial in the right balance of security and ease-of-use.

Next, we implemented Cloudflare’s brand new “Workers” feature, which gives granular control over CDN functionality using a javascript “service worker.” The Worker we wrote for this project handles more than 6 million requests a day, and gives the Commonwealth the ability to test and deploy CDN level changes to development, staging, and production environments independently, making it much faster and safer to verify and release changes. The worker implementation benefits the Commonwealth giving them flexibility for the future, while also reducing cost over the previous CDN.

These workers were also integral to the success of this migration beyond what we had initially imagined. During the testing and release phases of the project, they gave us a mechanism for fixing changes that was reviewable and testable. Having a well-defined review and deployment process improved the team’s visibility into what changes were being made, and let us avoid silly mistakes. Overall, we felt the development team’s velocity was greatly improved by using this workflow.

After studying Pega’s workflow to identify areas of improvement and discuss pain points, we found that the team’s main challenges were around deployment. Developers did not have a strong sense of confidence in the deployability of the code and team members still relied heavily on manual testing during the development cycle. This caused delays that frustrated the team and impacted the business agility.

We all agreed that the development team should be able to write automated tests to rely less on tedious manual testing. With this goal in mind, we looked into removing blockers for the Pega team.

The development team should be able to write automated tests to rely less on tedious manual testing.

Pega’s hosting provider at that time was making it prohibitively expensive to spin up the desired number of environments to test on, meaning, we had to find a cost-effective solution to spin up and spin down environments to enable automated testing. Providing those initial environments would also unlock the ability to demo and review work more easily before deploying to production.

Pega is a highly technical group and we wanted to fully empower them to do their best work confidently. After reviewing and discussing internally and with Pega all of the various options on the market and the costs associated, we chose to develop a custom solution built on Kubernetes in AWS. We made sure that all configuration of the entire system was built with Terraform and in a Github repository, meaning anybody on Pega’s team could dig into the codebase to understand it and make changes in the future. Fully documenting the new system and how to use it has always been a priority for any client engagement.

Fully documenting the new system and how to use it has always been a priority for any client engagement.

With that foundation set up, we were quickly able to load Pega’s application into Docker images, write sample automated tests and give their development team the ability to deploy an unlimited number of environments, one per Github branch, at a reasonable cost.

On a daily basis, dozens of developers work simultaneously on Finder.com, adding new functionality, fixing bugs, and creating new ways to provide value to users. As the team scales, it is faced with several challenges around managing the deployment pipeline. 

Until recently, developers were often limited and blocked by a complicated build process, where any code change would take about 12-16min to go live. Making that change often required modifying multiple repositories since all the themes and plugins were split and could not be fully decoupled. Deploying to staging environments was a manual process, and code reviews required lengthy instructions and thus were error-prone. The developer that authored a change, needed to remember to merge all repositories for a successful deployment or risk bringing down the site. 

Speeding up the deployment process 

An ineffective deployment pipeline can easily add up to hundreds of hours of time a month waiting for a build to complete that could have been spent on delivering other functionality instead. The development workflow should be smooth with stumbling blocks removed. Since Finder’s engineering team is growing rapidly, this problem needed to be addressed first. 

To speed up time to market, we overhauled the build process at Finder. We implemented Buildkite to enable continuous integration in a more efficient way. As a result, build time decreased by more than 50%, to only 8min. 

This increased efficiency sped up the process for Finder’s developers to get their work onto staging environments and out to production, and in the hands of customers.

Offering stability to the development team

Another challenge for Finder was the stability and time it took to deploy a hotfix to production. If multiple developers were trying to deploy to the product at the same time, they would often frustratingly block each other, further increasing everyone’s time to production. 

To release this functionality, we identified and rearranged certain key jobs so they could be run in parallel. We also identified build steps that could be made more efficient. For example, there were many jobs downloading the same libraries, and when the Docker image was built in the end, these libraries were again downloaded, synchronously. By limiting this, speed was improved significantly.

In the end, speeding up the build process, meant increased stability, and a decreased time to deploy a hotfix. 

Building comparison tables

In addition to platform work, Finder brought Last Call Media on board to help improve their comparison tables. Finder’s business model requires them to accurately connect users with relevant products. If a user with a “poor” credit score tries to apply to a credit card that only accepts “very good” scores, they’ll waste their time, and Finder will not be succeeding in its mission to connect users to relevant products. The tables were written in legacy JavaScript, HTML, and CSS and we worked to rebuild the tables in modern React, adding modals on top of the tables to surface the most important data filters to the user first.

Now, after users make an initial selection, they can see the most relevant results for their case. We added advanced interactive features such as calculators to the top of tables so users can input data relevant to their personal circumstances and see automatically calculated savings for each product.