We're proud to provide Continuous Delivery services.

Expand
Massive Performance Wins on Mass.gov

Processes
  • Continuous Delivery
Team Leadership
  • Senior Architect
    Rob Bayliss

Last Call Media team members were invited to join the Digital Services team at Mass.gov, to help them operationalize their Drupal 8 platform following the public launch.

Mass.gov is the website of the Commonwealth of Massachusetts. The primary stakeholders are the constituents who visit the website, and the state organizations who publish content on and visit the website for aspects of their job. It receives upward of 15 million page views a month and changes to the site are released twice weekly by a team of developers. The traffic profile for the site is interesting, yet very predictable. The vast majority of the site traffic occurs between 8:00 am and 8:00 pm during the business week. Site editors are working during the business day (9:00 am - 5:00 pm), and releases happen after the work day is over (after 8:00 pm). On analytics graphs, there are always five traffic spikes corresponding with work days, except when there is a holiday—and then there are four.

LCM assisted in making some pretty dramatic changes on both the front and back end of the site; every action we took was in service of either site stabilization or improving content “freshness.” State employees need to be able to publish content quickly, and constituents need fast access to the information that’s being published, without disruptions while they’re visiting the site. These two needs can be considered opposing forces, since site speed and stability suffers as content freshness (the length of time an editor waits to see the effect of their changes) increases. Our challenge was to find a way to balance those two needs, and we can break down our progress across an eight-month timeline:

September, 2017

The new Mass.gov site launched after roughly a month in pilot mode, and we saw an increase in traffic which corresponded with a small response time bump. The site initially launched with a cache lifetime of over an hour for both the CDN and the Varnish cache. This meant that the site was stable (well insulated from traffic spikes), but that editors had to wait a relatively long time to see the content they were publishing.

November, 2017

We rolled out the Purge module, which we used to clear Varnish whenever content was updated. Editors now knew that it would take less than an hour for their content changes to go out, but at this point, we still weren’t clearing the CDN, which also had an hour lifetime. Site response time spiked up to about two and a quarter seconds as a result of this work; introducing “freshness” was slowing things down on the back end.

December, 2017

We realized that we had a cache-tagging problem. Authors were updating content and not seeing their changes reflected everywhere they expected. This was fixed by “linking up” all the site cache tags so that they were propagating to the pages that they should be. We continued to push in the direction of content freshness, at the expense of backend performance.

To address the growing performance problem, we increased the Drupal cache lifetime to three hours, meaning Varnish would hold onto things for up to three hours, so long as the content didn’t get purged out. As a result of our Purge work, any content updates would be pushed up to Varnish, so if a page was built and then immediately updated, Varnish would show that update right away. However, we saw very little performance improvement as a result of this.

January, 2018

Early in the month, we experienced a backend disruption due to some JavaScript changes that were deployed for a new emergency alert system. In development, we added a cache-busting query parameter to the end of our JSON API URL to get the emergency alerts. However, in the production environment, we were adding one additional, completely uncached request for every person that hit the site. As a result of this relatively minor change, the backend was struggling to keep up (although constituents saw almost no impact because of the layered caching). This illustrated the importance of considering the performance impact of every single PR.

Careful study of the cache data revealed that each time an editor touched a piece of content, the majority of the site’s pages were being cleared from Varnish. This explained the large spike in the response time when the Purge work was rolled out, and why raising the Drupal cache lifetime really didn’t affect our overall response time. We found the culprit to be the node_list cache tag, and so we replaced it with a system that does what we called “relationship clearing.” Relationship clearing means that when any piece of content on the site is updated, we reach out to any “related” content, and clear the “cache tag” for that content as well. This let us replace the overly-broad node_list cache tag with a more targeted and efficient system, while retaining the ability to show fresh content on “related” pages right away. The system was backed by a test suite that ensured that we did not have node_list usages creep back in the future. This earned us a massive performance boost, cutting our page load time in half.  

We found that the metatag module was generating tokens for the metatags on each page twice. The token generation on this site was very heavy, so we patched that issue and submitted the patch back to Drupal.org.

February, 2018

We had another backend disruption due to some heavy editor traffic hitting on admin view; our backend response time spiked up suddenly by about 12 seconds. A pre-existing admin view had been modified to add a highly desired new search feature. While the search feature didn’t actually change the performance of the view, it did make it much more usable for editors, and as a result, editors were using it much more heavily than before. This was a small change, but it took what we already knew was a performance bottleneck, and forced more traffic through it. It demonstrates the value of being proactive about fixing bottlenecks, even if they aren’t causing immediate stability issues. It also taught us a valuable lesson—that traffic profile changes (for example, as a result of a highly desired new feature) can have a large impact on overall performance.

We got a free performance win just by upgrading to PHP 7.1, bringing our backend response time from about 500 milliseconds down to around 300.

We used New Relic for monitoring, but the transaction table it gave us presented information in a relatively obtuse way. We renamed the transactions so that they made more sense to us, and had them broken down by the specific buckets that we wanted them in, which just required a little bit of custom PHP code on the backend. This gave us the ability to get more granular about what was costing us on the performance side, and changed how we started thinking about performance overall.

We added additional metadata to our New Relic transactions so we could begin answering questions like “What percentage of our anonymous page views are coming from the dynamic page cache?” This also gave us granular insight on the performance effects of changes to particular types of content.

We performed a deep analysis of the cache data in order to figure out how we could improve the site’s efficiency. We broke down all the cache bins that we had by the number of reads, the number of writes, and the size. We looked for ways to make the dynamic page cache table, cache entity table, and the render cache bin a little bit more efficient.

We replaced usages of the url.path “cache context” with “route” to make sure that we were generating data based on the Drupal route, not the URL path.

On the feedback form at the bottom of each page on the site, the form takes a node ID parameter, and that’s the only thing that changes when it’s generated from page to page. We were able to use “the lazy builder” to inject that node ID after it was already cached, and we were able to generate this once, cache it everywhere, and just inject the node ID in right as it was used.

We took a long hard look at the difference between the dynamic page cache and the static page cache. Without using the Drupal page caching, our average response time was 477 milliseconds. When we flipped on the dynamic page cache, we ended up with a 161 millisecond response, and with the addition of the static page cache, we had a 48 millisecond response. Closer analysis showed that since Varnish already handled the same use case as the Drupal page cache (caching per exact URL), the dynamic page cache was the most performant option.

We automated a nightly deployment and subsequent crawl of site pages in a “Continuous Delivery” environment. While this was originally intended as a check for fatal errors, it gave us a very consistent snapshot of the site’s performance, since we were hitting the same pages every night. This allowed us to predict the performance impact of upcoming changes, which is critical to catching performance-killers before they go to production.

As a result of all the work done over the previous 5 months, we were able improve our content freshness (cache lifetime) from 60 minutes to 30 minutes.

April, 2018

We enabled HTTP2, an addition to the HTTP protocol that allows you to stream multiple static assets over the same pipeline.

We discovered that the HTML response was coming across the wire with, in some cases, up to a megabyte of data. That entire chunk of data had to be downloaded first before the page could proceed onto the static assets. We traced this back to the embedded SVG icons. Any time an icon appeared, the XML was being embedded in the page. In some cases, we were ending up with the exact same XML SVG content embedded in the page over 100 times. Our solution for this was to replace the embedded icon with an SVG “use” statement pointing to the icon’s SVG element. Each “used” icon was embedded in the page once. This brought pages that were previously over a megabyte down to under 80 kilobytes, and cut page load time for the worst offenders from more than 30 seconds to less than three seconds.

We reformulated the URL of the emergency alerts we’d added previously to specify exactly the fields that we wanted to receive in that response, and we were able to cut it down from 781 kilobytes to 16 kilobytes for the exact same data, with no change for the end users.

We switched from WOFF web fonts to WOFF2 for any browsers that would support it.

We used preloading to make those fonts requested immediately after the HTML response was received, shortening the amount of time it took for the first render of the page pretty significantly.

We added the ImageMagick toolkit contrib module, and enabled the “Optimize Images” option. This reduced the weight of our content images, with some of the hero images being cut by over 100 kilobytes.

We analyzed the “Did you find what you were looking for?” form at the bottom of every page, and realized that the JavaScript embed method we were using was costing our users a lot in terms of bandwidth and load time. Switching this to a static HTML embed with a snippet of JavaScript to submit the form had a dramatic improvement in load time.

The Mass.gov logo was costing the site over 100 kilobytes, because it existed as one large image. We broke it up so that the image of the seal would be able to be reused between the header and the footer, and then utilized the site web font as live text to create the text to the right of the seal.

Additional efforts throughout this time included:

  • Cleaning up the JavaScript. We found an extra copy of Modernizer, one of Handlebars, and a lot of deprecated JS that we were able to get rid of.

  • We removed the Google Maps load script from every page and only added it back on pages that actually had a map.

  • We lazy-loaded Google search so that the auto-complete only loads when you click on the search box and start typing.

Our work across these eight months resulted in huge improvements in both the front and back end performance of the Mass.gov site. We achieved a 50% overall improvement in the back end performance, and a 30% overall improvement in the front end performance. We continue to work alongside the Digital Services team on these and other efforts, striving for the best possible experience for every single user and constituent.

See the BADCamp presentation about this work here.

Expand
Delivering a high stakes MVP.

Processes
  • Agile/Kanban
  • Agile/Scrum
  • Continuous Delivery
Team Leadership
  • Senior Producer
    Sean Eddings

Following a branding initiative, Designow worked with Last Call Media to build and launch the initial release of a complex crowdfunding platform on Drupal.

When we began, initial prototyping and feature development had already been started. As is often the case, much of the prototyping had made its way into substantial portions of what was now the project’s foundational work. It had become difficult to determine exactly what was completely done, what was partially done, and what requirements had been left out along the way.

We used our expertise in Agile methodologies to bring order to complexity and the product to launch. We began by building and prioritizing a backlog of tasks, and getting high-priority tasks into a ready state by defining appropriate definitions of done. We focused on implementing strategies for limiting the work in progress present in the project when we started. We forecasted a few sprint goals, informed by our backlog development and resulting strategies, and quickly went into heavy development.

Immediate visibility for stakeholders into development progress became of major importance for their own ability to make quick business decisions concerning their investment. In order to maximize the value of the work, we adapted our initial Scrum iteration approach into shorter, one-week sprints of continuous delivery to get small groups of completed tasks out faster. This soon evolved into more of an ideal Kanban flow, allowing a continuous awareness and continuous delivery of the right value.

Since the initial release, LCM and Designow have continuously measured and inspected valuable feedback. Our work is ongoing, as the product is currently receiving regular releases of new features and refinements.

Expand
Consortium Assault Services app.

Processes
  • Continuous Delivery
Team Leadership
  • Senior Producer
    Kelly Albrecht
  • Senior Architect
    Rob Bayliss
  • Senior Development
    Rob Bayliss

In response to growing concerns and attention around Sexual Harassment and Assault Nationwide, Amherst College needed a tool to serve students of the Five Colleges with rapid access to Title IX office information and emergency services.

LCM and Amherst College worked together with student advocates, Title IX, LGBTQ, and other campus offices and organizations to design and develop an iOS App that puts valuable information, from a Drupal site Amherst can administer, into the hands of students. The major feature of the app was to direct assault survivors to emergency contact information, help services, and other advocacy groups, anonymously and quickly.

The app was announced to all incoming and returning students during new school year orientation. Information about the app has been circulated through the Five Colleges on promotional materials and “get help” brochures and posters.

Expand
Custom tailoring Haverford.edu.

Processes
  • Agile/Scrum
  • Continuous Delivery
Team Leadership
  • Senior Producer
    Kelly Albrecht
  • Senior Architect
    Jeff Landfried
  • Senior Development
    Jeff Landfried

When LCM got the call from Haverford about redoing their site in Drupal, we thought it was going to be a redesign. After an initial conversation we agreed to do a Discovery and Strategy engagement first, to determine what the true needs were and then develop a strategy for solutions. We conducted stakeholder interviews and developed the User Personas of who our work was to be for. Interestingly, none of these personas ended up being an anonymous site visitor, but instead were different types of content editors and administrators.

Haverford didn’t need a redesign; their site looked great already. The biggest issue they needed to solve was not having enough time to do new feature development themselves. They were a smart and capable team and over many years had developed and maintained a large custom PHP implementation for college’s website needs. They were finding, however, that too much of their time was being spent updating pages for college members and groups. Their hope was for us to build them a site that looked and functioned like their current site, but built on a modern CMS with exceptional user management and publishing workflows.

Once we had alignment on their needs and a strategy for solutions, we built out a Content Model, Product Backlog, and Information Architecture. The project was completed with Drupal in steady collaboration with the Haverford Communications team, in 5 development Sprints, and launched on the Acquia Infrastructure.

The Haverford Team really jumped in and took off with it. It was really great!

You can read more about this project as LCM’s Acquia Certified Grand Master Developer, Jeff Landfried shares details of his experience here.

Since then, Last Call Media has continued to work with Haverford on an ongoing basis as part of our dedicated Continuous Delivery relationship, where our dedicated team of developers helps to keep their site secure, up-to-date, and assists as needed with anything from strategy to design to development.

The migration of Haverford to Drupal on Acquia remains one of our favorite projects to date, and it’s a great source of professional pride for all of us. We look forward to a long partnership with Haverford as we continuously evaluate how their online experience is meeting the needs of the College.  

Expand
Forklift to Drupal 7 in 8 weeks

Processes
  • Agile/Kanban
  • Agile/Scrum
  • Continuous Delivery
  • XP
Team Leadership
  • Senior Producer
    Sean Eddings

Leveraging a small multidisciplinary production team and agile methodologies, LCM migrated SUNY Potsdam from their self-hosted legacy CommonSpot CMS to Drupal 7 on Acquia in just 8 weeks.

The small team at the SUNY Potsdam Office of Public Affairs had been managing the proprietary CommonSpot CMS since its implementation in 2008. It was inflexible and the team struggled with reliability issues, so SUNY Potsdam was looking for a more useable, stable, efficient, and scalable solution. They chose Drupal 7, Acquia, and Last Call Media.  

SUNY Potsdam partnered with Last Call Media and Acquia to migrate their site to Drupal 7. Leveraging the scrum methodology, LCM broke down and organized the major site features into a prioritized backlog, groomed for two-week iterations. While planning and backlog refinement was ongoing several times a week, the core development team at LCM met with Potsdam every other Friday to review work completed, provide training on the new CMS, and to facilitate the feedback-gathering process.

Since SUNY Potsdam had recently gone through a redesign, the project required the new site to maintain the existing look and feel. We began with an in-depth audit of all the different page types and page elements. Instead of doing this manually page by page, we first looked for a programmatic solution. Since CommonSpot did not provide a way to generate this information within the CMS, LCM used its HTML Crawler tool to programmatically crawl the existing production site and analyze the various HTML tags to determine page elements (such as slideshows, feeds, etc), including where and how often they appear. This provided tremendous visibility into the site’s underlying structure, which was critical in planning our approach to the migration to Drupal.

Potsdam Art Page

 

After reviewing this data with SUNY Potsdam, we began the process of consolidation– instead of building one-off page elements, we consolidated similar elements into single widgets that behaved differently based on where on the page it was placed. This helped reduce the vast number of options a content author has to choose from, making it easier for them to do what they need to do: focus on the content. To achieve the desired platform flexibility, LCM built a repository of flexible and adaptable widgets to allow the marketing team at Potsdam to build custom pages. 

The migration included several different page templates and tens of thousands of pieces of content, which required writing and testing a series of migration scripts to get all the content from one CMS to another without downtime or a lengthy content freeze. Since the CommonSpot installation did not have a concept of structured content, LCM used it’s HTML crawler tool again to programmatically identify page content and then map it to its new location in Drupal. Once the custom scripts were written and tested, the migration took only 15 minutes for tens of thousands of pieces of unique content and the associated metadata, such as date published, authoring information, and URL

Potsdam Events Page

 

The new site also pulls in events automatically from their event management system, SOGo, and tags the event to the relevant department or office in the CMS so that it appears on that organization’s page.  

Lastly, in order to make it as easy as possible for content authors to login to the site, we leveraged the identity management service at Potsdam, Active Directory, to allow users to use their domain credentials to authenticate with Drupal.

This project addressed several internal pain points with the SUNY Potsdam main website, allowing the marketing team to move from maintenance and support tasks to other organizational priorities. The site loads blazingly fast on Acquia, and Potsdam continues to work with LCM in an ongoing support relationship.   

Expand
Strategic pivot, design and development.

Processes
  • Agile/Kanban
  • Agile/Scrum
  • Continuous Delivery
Team Leadership
  • Senior Producer
    Kelly Albrecht
  • Senior Architect
    Jeff Landfried
  • Senior Development
    Jeff Landfried

Last Call Media helped the CIO Council team rethink their strategic approach. The CEC Exchange needed to be rebranded and transformed from a place where very few people logged in, to one that showcased the key benefits of membership in the Council: media placements, personalized leadership development, and exclusive access to unique content, events, and peer-matching services.

The CIO Executive Council is a community of Chief Information Officers and other high-level IT professionals from corporate, nonprofit, education, and government backgrounds, gathered together by the International Data Group, a giant technology media, data, and marketing firm responsible for such well-known brands as PCWorld, MacWorld, and CIO Magazine.  

The CIO Executive Council created the CEC Exchange, which had originally been envisioned as a community space for their CIO members to share things of mutual interest. Two years in, the log-in only service was largely unused by the incredibly busy C-Level executives who were members of the Council.  

CEC Exchange uses Salesforce to manage information about each user in their membership pipeline. On the Drupal side, the site contains several main sections that each have their own exclusive resources and content. These sections require a subscription to access. Using a combination of the Salesforce Suite and the Organic Groups module, the site regularly retrieves updated information from Salesforce and syncs it with the Drupal user list, tracking their subscriptions by assigning them to the appropriate groups. The site’s dashboard displays specialized tiles that administrators have full control over, allowing them to highlight featured content and the most sought-after resources within each section for subscribed users. Site visitors and users without a subscription can still see the content that is available from each subscription package, but receive an admin-customizable call to action specific to the content they are attempting to access when they click on a tile from the dashboard.

nav hihglight

phones

 

icons

tablet

Last Call Media designed a much simpler and more attractive interface built around the most compelling products and services that the CIO Exchange has to offer. A CIO visitor to the site can now see how the resources there can help them further their own career, stay on top of the ever-changing trends in technology, reach out to peers facing similar challenges, and access the CIO Exchange’s team of expert concierge staff to answer any questions they may have.

Through a mix of branding and design, Last Call Media re-made the site, which was recently relaunched to incredibly positive initial feedback.

Expand
Asia Society's International Centers.

Processes
  • Continuous Delivery
Team Leadership
  • Senior Producer
    Kelly Albrecht
  • Senior Architect
    Kelly Albrecht
  • Senior Development
    Jeff Landfried

Asia Society wanted to expand their current Drupal website with a new capability. Asia Society Centers were to be information portal microsites for their international member branches. The new functionality had to integrate with their existing site design, and provide editing capabilities for branch content authors.

The project required an aggressive timeline, and was delivered in two weeks. Organic Groups (OG) and Nodequeues were used to provide separate branch-specific sections, or “centers,” of the website, and to provide branch editors control over their own center’s content. The new sections were integrated with the existing site design through customized theming.

Thanks so much for the smooth migration on such a short deadline. I know there were (more than) a few balls in the air at all times and we really appreciate your attention to detail and organized process.

Megan MacMurray, Associate Director of Technology

The ability to provide rapid development through a flexible team of developers ensured the project was delivered on time, fully tested and polished. The “Centers” have provided Asia Society the means to expand their content reach and focus.

Expand
Rainforest Alliance's Global Impact Map.

Processes
  • Agile/Kanban
  • XP
Team Leadership
  • Senior Producer
    Kelly Albrecht
  • Senior Architect
    Rob Bayliss

Rainforest Alliance needed an effective way to illustrate their global efforts to preserve one of the earth’s most valuable resources: the environment.

Most recently impressive is the mapping project that we unexpectedly threw on Rob and Tom’s lap, and which they handled quickly and expertly. 

Melissa Normann, Senior Manager, Web Strategy and Development

Knowing that a simple page with paragraphs wouldn’t suffice, we built a map that allows the Rainforest Alliance to display data from a specific region. The user can then zoom in on those regions or look at specific data points to learn more. The map takes a large spreadsheet of geodata, created based on information from the Rainforest Alliance’s CRM, and creates an interactive map powered by CartoDB that allows users to see RA-certified organizations, and what they do.

The project took less than a month to complete, using a combination of Kanban and XP management processes.

Expand
Massive nightly sync.

Processes
  • Continuous Delivery
Team Leadership
  • Senior Producer
    Sean Eddings
  • Senior Architect
    Rob Bayliss
  • Senior Development
    Rob Bayliss

Yale University Press has a massive collection of over 15,000 unique publications they’ve published over the past 100 years. The Press desired the ability to allow their users to browse, check inventory and purchase items directly from their Yale University Press Drupal site, which required relaunching their site on Drupal 7, integrating their collection management system and an e-commerce and fulfillment solution. After working with another vendor for over three years to get the critical nightly sync from their Microsoft SQL Server Management Studio database to Drupal sync running, Yale University Press was seeking a second opinion. 

Working with Yale ITS and the Press, we successfully implemented a nightly sync that queries their SQL Server for changes made in the last 24-hrs and updates the records in Drupal in under 15 minutes every night at midnight.

Expand
Leveraging our Scaffolding and Drupal 8.

Processes
  • Agile/Scrum
  • Continuous Delivery
Team Leadership
  • Senior Producer
    Sean Eddings
  • Senior Architect
    Rob Bayliss
  • Senior Development
    Jeff Landfried

Since early 2014, LCM has continued a productive, ongoing partnership with Chicken Soup for the Soul, and supports their web properties and the associated infrastructure. Recently, Chicken Soup asked LCM to launch two new and completely different Drupal 8 sites within a month. LCM worked off of prototypes from Chicken Soup for the Soul and was trusted to move quickly. By deploying two separate teams of 2 developers, LCM was able to take each site from prototype to launch on D8 and Pantheon within two weeks, while another team maintained the ongoing feature release schedule on Chicken Soup for the Soul’s massive Drupal 6 site.

In June of 2016, Chicken Soup needed a simple site for their rapidly-growing television and online programming production and distribution business. The site needed to handle a collection of content pages and videos, and was intended to be another microsite that would follow some standard templating and functionality as laid out for previous Chicken Soup sites LCM had worked on, and new sites that were still to come.

Chicken Soup was looking for an alternative approach.

Building new features to support growing business lines inside their massive aging Drupal 6 site was becoming unsustainable. Over time, the site had accumulated so much functionality that each deployment ran a high risk of breaking something, which led to lengthy deployments. Recognizing that issue, a plan was developed in partnership with Chicken Soup for the Soul to spin out a series of smaller, more focused sites sharing a similar architecture. Drupal’s modular architecture, and particularly Drupal 8’s approach to dependency management, made it a great fit for this task. Additionally, while the core CMS functionality of Drupal 6 worked well, the UI was becoming dated and cumbersome to work with. Drupal 8 featured a lot of usability enhancements, such as the built in WYSIWYG, that would make the site much more usable overall. Finally, the feature set of the site was tightly focused, and after consideration, we were able to implement it with a small handful of contributed modules, and very little technical debt. 

Following on the success of the Chicken Soup for the Soul Pet Foods site, Last Call Media used a similar formula: leverage Drupal 8 core wherever possible, and avoid contributed modules. This was a great strategy in terms of avoiding the turmoil of early Drupal 8 contrib churn, and had the side benefit of keeping the site very lean and performant. After experiencing some past pain points in using the bare “Configuration Management” system in Drupal 8, we chose to use the Features module on this project. Features makes it easy to bundle configuration into modules, and makes it easier to share configuration (in the form of Drupal modules) between the brand’s sites should the need arise in the future. 

The site uses Last Call Media’s boilerplate Drupal 8 scaffolding build, which helped jumpstart the development process by providing a suite of best practices and quality assurance tools with no extra effort.

The goal of this project was to build a flexible marketing site capable of showcasing Chicken Soup for the Soul’s entertainment offerings; primarily their TV shows and online videos. The biggest obstacles the project faced were the looming deadline, the relative instability of Drupal 8 immediately following the initial release, and the lack of contributed modules that were available to us. For example, the Media-related modules we would normally use for the online video section were not stable yet. Instead of using a media/file entity as we normally would to store an online video, we leveraged Drupal core’s new URL field to store the URL of the Rumble video, and used a field template to output an embed link. It was a simple and elegant solution to a difficult problem. 

Thanks to excellent communication with Chicken Soup for the Soul’s Digital Strategy team, and Last Call’s experience in working with Drupal 8, we were able to turn the project around in just two weeks. This met the deadline set by the marketing team, and achieved all of the goals that were set out.