A Reimagined Rainforest Alliance on Drupal 8 in 6 Sprints

In the fall of 2016, the Rainforest Alliance and Last Call Media launched an exciting redesign of www.rainforest-alliance.org, built on Drupal 8, employing seasoned agile software development methodologies.  Our productive partnership with the Rainforest Alliance resulted in a technically groundbreaking site that allowed users unprecedented access to the riches of their content after just four months of development.  The tool is now primed to drive the Rainforest Alliance’s critical end-of-year development activities. 

Our relationship with the Rainforest Alliance began in August of 2013 when LCM undertook a massive Drupal 6 to Drupal 7 upgrade.  We enjoy a strong relationship with the Rainforest Alliance team, working together to continuously deliver strategic value in their digital properties, and were proud to be chosen for a full site redesign and upgrade.

Over the years, RA has cultivated a repository of structured content to support their mission.  While the content is primarily displayed as long form text, there is a wide variety of metadata and assets associated with each piece of content.  One of the primary goals of the new site was to enable discovery of new content on the site through automatic selection of related content driven by the metadata of the content the user was viewing.  Additionally, RA had a future requirement for advanced permissioning and publishing workflows to enable stakeholders outside of the web team to play a role in the content lifecycle.  

After some initial consideration, the Rainforest Alliance and Last Call Media decided to build a responsive Drupal 8 website, which included both building out new content types and migrating existing content from their then-current Drupal 7 site. It needed to launch on a 4 month timeline, by the end of September 2016.

Why Drupal 8

Drupal 8 was selected for this project based on several factors.  First, its focus on structured data fit well with Rainforest Alliance’s need for portable and searchable content.  Second, the deep integrations with Apache Solr allowed for a nuanced content relation engine.  Solr was also used to power the various search interfaces.  Third, Drupal has historically had powerful workflow tools for managing content.  While these tools weren’t quite ready for Drupal 8 when we built it, we knew they would be simple to integrate when they were ready.  In short, Drupal was a perfect fit for the immediate needs, and Drupal 8 met the organization’s longer term goals.

Why Agile/Scrum

To meet the 4 month timeline, the project followed a highly collaborative, agile project management style (Scrum) such that RA could provide wireframes, design and UX direction, technical specifications, user testing and QA for all content types, and LCM would carry out all Drupal Development, including theming, and project management, providing guidance based on our expertise and best practices. 

Furthermore not all requirements were known at the outset and many things were known to potentially need to go a different direction depending on some of the outcomes along the way. Agile methodologies avoid the “big reveal” better than other styles of project management. Issues requiring a change in direction are raised in near realtime, resulting in saved time, in the long run, by better avoiding going in the wrong directions.

Backlog Development, Grooming, and Refinement

Leading up to Sprint 1, for a period of 4 weeks, I worked as a Product Owner (PO) and Agile Coach, with the Rainforest Alliance (RA) webteam in a Business Owner (BO) role, and Rob Bayliss, as a Subject Matter Expert (SME) from Last Call Media (LCM), to build and groom the initial backlog. During this time, we coached RA on Agile/Scrum and being effective in the Business Owner role. 

Together, meeting twice per week, we groomed and refined the project backlog to the point where SME guesstimated sizing seemed reasonable for forecasting purposes. We did this sizing together too with each epic guesstimated for size by Rob, on note cards. We went through each of these note cards together with RA in planning poker style, where everyone guessed at how big they thought the epic was and then, revealing Rob’s guesstimate, we compared the group’s sizing differences. It didn’t take everyone too long to pick up on Rob’s style of sizing, but many misconceptions and misunderstandings were revealed when someone’s sizing guess was wildly off from someone else’s. This exercise fine tuned our alignment on the backlog items and allowed for more accurate forecast sizing. 

Sprint Forecasting

Based on the sizing exercise, we set to strategically sorting our epic note cards into sprint piles. The strategy was to group cards into earlier sprints piles based not only on the importance of the epic but also on how helpful it would be to have that epic in place for later sprints. A grouping was considered full when it reached resource capacity, which was determined by adding the sizing from each note card up to an agreed upon threshold. It turned out we were able to forecast 7 sprints worth of backlog items, with approximately 2 more sprints worth of “nice-to-haves” as well as some epic cards determined to be no longer needed.

The result was each sprint’s goal being forecasted to better enable following sprint goals and concurrent releases for user testing, feedback, and iteration. Additionally, as the project budget and timeline only allowed for 6 sprints, our forecast safely assumed for a backlog of certain items being left undone at the final public release after the 6th sprint. This concept of moving items to a backlog for after sprint 6 would become a critical one later in the project as complexities were uncovered, new directions emerged, and priorities changed.

The above spreadsheet shows a forecast of 6 Sprints worth of guesstimated epics, The concept of a forecast was an important one. Just like with the weather forecast, the further out it goes, the less it is to be expected to be accurate. The sprint forecast further became a living document for maintaining an evolving project vision across all of its iterations.

Agile Planning

With each sprint, we made three planning ceremonies available to the project. Pre-sprint grooming was a group exercise for the development team to go over the upcoming sprint’s wish list for the purposes of optimizing the official Sprint Planning meeting. Sprint Planning was held on the first day of a sprint and followed traditional Scrum guidelines. In Re-Forecasting, the team gave feedback from the trenches on the original forecast of SME guesstimates. This enabled the opportunity to adjust the forecast to be more realistic, evolving the project vision as it was adapted to these reports. Additionally, in later sprints, we began doing mid-sprint releases of completed work to be previewed. We did this to enable better planning and re-prioritization from the RA team.

For openness, I kept a Product Owner journal shared with the RA team. In it I tracked the daily standups, their sticking points and resolutions, as well as each day’s User Stories that were completed and the work-unit points that I awarded. This last piece of daily info was used to keep a realtime project build up chart.

A plan requires an awareness of things to consider for planning. Agility is the concept that we need to be ready to adapt our plans over time as we gain additional awareness. To that end, all of Scrum, from its Values to its Ceremonies, is designed to increase awareness, enabling better adaptation to change. The following build up chart is a Scrum artifact from this project that was used for the purpose of increasing awareness and better planning.

The Build Up Chart additionally serves well for telling this project’s story across its six sprints. For example, one can tell things about the project just from this chart. The sharp upticks toward the end of each sprint are indicative of a new build, where substantially complex functionality is being attempted each sprint resulting in shippable increments on most stories not being realized until the very end or their sprint. One can also tell that this project was re-forecasted at least three times, resulting in the adjustments to the estimated project size over time.

Sprints 1 and 2

Sprints 1 and 2 stayed on track nicely, getting most of the critical core functionality in place. Some key Drupal 8 modules implemented during these sprints included Page Manager, Layout Plugin, Panels, as well as Search API/Search API Solr, Media Entity (and related modules), Entity Embed, Entity Browser, Inline Entity form, and Features. Core project functionalities are described by Jeff Landfried, lead project developer, below in the context of each of these Drupal modules.

Page Manager/Layout Plugin/Panels:
Page manager is a great tool for making it easy to create specialized landing pages, and when combined with Layout Plugin and Panels it provides the ability to use different layouts when viewing different node types (or other entity/bundle combination). Specialized landing pages were built as specialized page manager pages, many with their own layouts. All of the different full node displays were handled by Page Manager, using different variants for each node type.

Search API/Search API Solr:
Most content types have a “related content” section at the bottom of the page. Tagging content is one great way to handle something like this, but for our requirements we needed to have logic that was more robust than only showing other content with the same taxonomy terms. We went with Solr for this, specifically for the “more like this” (MLT) functionality that it provides. Search API Solr provided the interface for managing our servers and indexes, then with a custom block we were able to leverage MLT with our own boost criteria to help control how the related content lists were generated.

Media Entity (and related modules):
Drupal core provides file fields, which allow us to upload files to different entities, but this project had a requirement that we must be able to reuse the uploaded files, and have the ability to add additional related information for each file or image that is uploaded. Things like caption, image source, etc. On top of that, we needed to be able to display these files in different ways - in some places an image may display the caption as a tool tip, while in others it should display below the image. The Media suite of modules is perfect for this type of thing. We were able to use different modules from within the media ecosystem to handle images, embedded videos, and PDF documents, and add appropriate fields to each media entity bundle, and using Drupal core’s view mode system we were able to set up multiple displays for each media type.

Entity Embed / Entity Browser/Inline Entity form:
It hasn’t always been easy to empower content teams to easily add images and other entities to WYSIWYG fields, especially when those items need to be themed in a special way. Entity Embed allowed us to add new CKEditor buttons to the WYSIWYG that provide a dialog where the user can choose an entity that they want to appear in content, the view mode that they want it to display with, and then position it on either the left, center, or right side. One great thing about this is that the module uses a text format filter, so different text formats can display the embedded entities, while the others don’t. Entity Embed is primarily used for embedding images, but we also used it to give content editors the ability to embed blocks in their wysiwyg content as well.

Inline Entity Form allowed us to create entity reference fields, but gave content editors the ability to create the referenced entities right from the node edit form, something that can be a big time saver for content editors.

Entity Browser ties in with both Entity Embed and IEF by adding a button that opens a dialog displaying a view that allows users to select the entity that they want to use from a list, rather than having to remember media names, taxonomy names, or node titles and enter them into an autocomplete field.

These modules combined help make for a great editorial experience.

Features:
Drupal 8’s CMI initiative solved a lot of issues around managing configuration. That being said, we’ve found that bulk exporting/importing an entire site’s configuration isn’t a great workflow for our team that involves multiple environments and developers, each potentially having a few of their own special configuration options that need to be be set. Manually seeking out and overriding those configuration options in settings.php isn’t something that we decided was sustainable, and has its own drawbacks. The features module in D8 allows us to package and ship the configuration that we need to be consistent, while allowing us to leave out what may be different across environments (such as development only modules, css/js aggregation and page caching).

Sprint 3

Even though things were on track nicely across the first two sprints,  we could see work in progress and technical debt catching up with us during sprint 2. The first two sprints laid a tremendous amount of ground work, but also left as much technical debt. This resulted in a pivotal moment in the build chart, shown up close below, with progress dipping below target.

 

Notes from Sprint 3’s Planning Meeting:

This was forecasted as a really full sprint, but the team decided to only accept the stories above to make sure that there would be time to completely wrap up. The team is trying to be mindful of the amount of WIP tasks that already exist, and avoid creating more of those loose ends.

 

Fewer Stories were committed to, meaning fewer work-unit points earned, resulting in a below target buildup, but the silver lining was that the results were beyond excellent with far less technical debt left over to be carried into future sprints.

Sprint 4

 

For Sprint 4, we reconfigured project priorities with RA and reassessed forecasted story sizing. This reconfigured sprint forecasting resulted in batches of stories different than originally forecasted. It also resulted in the Estimated project size increasing. By the end of Sprint 4, we worked with RA to move tasks out of the project. This resulted in the Estimated Project size decreasing, putting the project on target with the build team’s velocity.

The team’s continued focus on careful commitment paid off big in this delivery in terms of comprehensive task completion, as well as delivering on a stretch goal, which nudged them just above target, leaving them in a great place for the next sprint. In addition to further iterating on the groundwork from the previous sprints, Sprint 4, by way of Drupal’s Panelizer,  brought with it Landing Pages and Content Hubs.

 

 

Panelizer:
When viewing taxonomy terms for several vocabularies we needed the ability to have a consistent layout, but to place different Custom Block entities on each term, which is essentially what Panelizer’s made for. The module doesn’t have full taxonomy term support yet, but the community is working to get it added, and the patches provided in this issue were far enough along that we were able to make it work without issue.

 

 

We were technically crushing it, but Sprint 3 was the beginning of a turning point that comes in every project worth discussing. We didn’t know it at the time but Sprint 3 was the transition point from Certainty to Doubt in this project’s Emotional Cycle.

 

The idea is that first there is a honeymoon period of Optimism and Certainty, but inevitably things don’t always work out as expected and Doubt creeps in, pulling the team into a Pessimistic state. Good teams identify and adapt to this shift, moving from Doubt to Hope quickly with minimal emotional damage. With every project, I’ve tended to look to preempt the slip from Certainty to Doubt, somehow looking to skip to Hope or even somehow to Satisfaction. Reality, however, can often be too elusive until it smacks you right in the face. I’ve come to accept that traveling the path through Doubt, into the pessimism, is simply the sign that a project is attempting to do at least as much as it should. Doubt comes from the realization that not everything imaginable is realistic to expect. 

Pessimism is a part of the process of grieving the loss of things hoped for that now seem unrealistic. The uptick toward Hope comes from acceptance and adapting expectations to reality. 

Sprints 5 and 6

The following overlaying of the two graphs, Build Up and Emotional Cycle, show their relation visually.
 

 

The ascension to Hope, on this project, can be attributed to an understanding of and a dedication to the five Scrum Values (Openness, Focus, Commitment, Respect, and Courage), followed by a more rapid iteration strategy, with frequent mini-releases during Sprint 6. Since most core functionality was solidly in place by this time, it became possible to squash larger numbers of site-wide bugs by relating them to the current sprint’s stories. This resulted in those stories shipping as more highly polished than was possible in previous sprints, while at the same time further iterating on stories from past efforts. Also, to increase awareness, and thereby project agility, during the final sprint, all Accepted Backlog Items were released and reviewed, as they were completed. The result was a highly collaborative finishing of the final shippable increment. At the close of Sprint 6, there were zero critical and only 3 moderate issues. The final Sprint/Project review had only 3 support questions.

The project used its remaining time until launch day running extensive QA with the LCM Continuous Delivery team making adjustments, finally launching as arguably the most impressive Drupal 8 site launched within a year of the initial release of the latest major version of the Open Source CMS, and most importantly in time for Rainforest Alliance’s major end-of-year donation campaign. The site delivers on its promise to showcase the Rainforest Alliance’s exciting and informative messages and beautiful imagery, and stands as testimony for the efficacy of the agile approach.

Last Call Media is a full-service creative agency developing solutions for partners online and off through innovative strategy, branding, print, and digital design. Last Call Media enjoys work with purpose– building engaging solutions that assist and support organizations working to improve their communities.

*This post was written with the assistance of Rob Bayliss, Jeff Landfried, and Alan Wolf.