Overview
The Massachusetts Department of Family and Medical Leave (DFML) partnered with Last Call Media to address integration challenges hindering the consistent rollout of features and enhancements for the Paid Family and Medical Leave (PFML) program. This partnership aimed to meet the technical demands of a complex system while ensuring a seamless experience for Massachusetts residents.
About the Client
The Department of Family and Medical Leave oversees the Commonwealth’s PFML program. This program provides temporary income replacement to eligible workers who are welcoming a new child into their family, are struck by a serious illness or injury, need to take care of an ill or ailing relative, and for certain military considerations.
Goals
DFML formed a multi-vendor team to create a cohesive experience for residents, employers, and its claim reviewers. Their goal was to develop a PFML application consisting of three key components: a user-facing Portal, an API integration layer, and a claims processing system. These would be integrated to offer a comprehensive end-to-end experience.
Challenges
This project was inherently complex due to the multi-vendor team working on distinct components that needed to gracefully integrate into a cohesive system. It quickly became clear that a thorough testing strategy was needed. Last Call Media stepped in to design and lead the implementation of the testing strategy and navigate the complexities of ensuring an error-free integration between each component.
Through testing, LCM’s objective was to ensure smooth interaction among the various teams within the system so that DFML could release with the certainty that:
- Massachusetts residents would be able to apply for PFML’s benefits.
- Massachusetts employers would be able to preserve the integrity of the PFML program by reviewing and providing details about their employees’ applications for PFML benefits.
- Adjudicators would be able to review the claims, process them, and notify the applicants and their respective employers about their application status.
Solutions
To overcome these challenges, Last Call Media focused on early and continuous testing and high observability, applying DevOps and CI/CD best practices. Building upon these foundational concepts, LCM’s testing solution encompassed three types of testing:
- Automated End-to-End Testing continuously brings certainty that value can technically be delivered.
- Load and Stress Testing brings certainty that value can technically be delivered under load.
- Business Simulation Testing, including Manual Testing, brings certainty that value can operationally be delivered.
LCM coupled this testing solution with high observability. We created a customized New Relic Dashboard that displayed test results in greater detail—providing DFML staff and other teams with insights into regressions, test failures, environments, versions, and other key aspects of the application’s test results and performance. This level of visibility and transparency empowered DFML and its teams to address issues promptly and make informed decisions throughout the development lifecycle.
The LCM team is a pleasure to work with. The team leans in, collaborates, fills white space, etc. It’s noticed and appreciated.
Results
By leveraging tried-and-true DevOps strategies, teams not only accelerated the release of features but also significantly improved key metrics that help monitor the system’s overall health. The results revolved around 3 key areas:
Reduction in defects slipping into production
A robust suite of end-to-end tests was implemented, meticulously covering both the main and high-risk features of the application. This approach played a pivotal role in reducing defects slipping into production by improving our capacity to detect issues early in the development cycle.
Once identified, issues were promptly triaged and assigned to the respective feature teams for resolution. Next, fixes were deployed to lower environments, where automated tests were rerun to ensure there were no f regressions. This systematic process not only increased the stability of the application but also nurtured a culture of continuous improvement and quality assurance within the development cycle.
Program Wins!
- Enhanced residents’ and employers’ experience: Users were less likely to encounter disruptions or errors during the application process, leading to a smoother and more efficient experience when going through the PFML application process.
- Confidence and residents’ trust in Government: A consistently smoother experience when using the application boosted confidence when using other government solutions.
- Reliable performance: DFML delivered releases with certainty that MA residents, their employers, and stakeholders, could rely on the application to perform as expected, knowing that there were rigorous testing measures in place to ensure its functionality and stability.
- Continuous improvement: Proactive testing led to swift identification and resolution of issues, resulting in quicker fixes and enhancements that minimize disruptions for all users.
High observability led to faster releases and early regression detection
Last Call Media created a customized New Relic dashboard to address challenges stemming from limited visibility of releases, component versions, and regression detection.
Program Wins!
- Improved transparency and collaboration: The centralized dashboard fostered collaboration and accountability among teams by providing a shared platform for monitoring and tracking deployment progress. This promoted a culture of transparency and teamwork within DFML, leading to smoother collaboration.
- Enhanced operational efficiency for DFML and teams: Streamlined monitoring and improved visibility reduced time and effort in issue resolution, saving costs and resources.
- Faster and consistent releases for MA residents: Deployment visibility accelerated release cycles, enabling DFML to deliver value more quickly to their residents.
- Data-driven decision-making: DFML stakeholders were able to make informed decisions by analyzing this data.
- Increase in application reliability: Early detection of regressions after the test runs in a particular environment, with a specific version, led to increased stability. Additionally, the collection of historical data allowed for the analysis of feature vulnerabilities over time.
DFML’s increased confidence in the performance and scalability of the application under load.
DFML’s increased confidence in the performance and scalability of the application under load was a result of Last Call Media’s rigorous load stress testing protocols.
LCM ran a load stress test before major releases and before the introduction of features or changes that impose an increased load on the API endpoints. Traffic spikes were set up to significantly exceed typical production traffic levels. The goal was to monitor the system’s performance and maintain error rates below a defined acceptable level.
The analysis of these results—such as claim submission error rates, and request/response times on a particular component version—provided valuable insights into the application’s performance under various conditions, ensuring its resilience and scalability over time.
Program Wins!
- Performance improvement from 2021 to 2023: Versus 2021 and 2022, 2023 showed a substantial reduction in the error rate under load.
- Historical load stress testing data helped pinpoint common API endpoint failures: These errors were surfaced and resolved, leading to fewer incidents, downtime, and slowdowns in production, resulting in a better resident and employer experience.
- Load stress testing enabled DFML to release with confidence that the application would withstand the spikes in claim submission and post-processing claims.
Learnings
Governments and their partners must collaborate closely to craft digital solutions that guarantee a seamless and reliable experience for residents. This involves prioritizing end-to-end value delivery to effectively develop large-scale systems. By instilling a culture of continuous testing and observation throughout the application development cycle, governments can ensure smooth software development and consistent value delivery for their residents.
On this Page
Focusing on end-to-end value delivery is the only way to ensure the effective completion of a large-scale system
Services
- DevOps
- Continuous Delivery
- Data Visualization
- Agile Coaching