Founder and CEO of Inflectra , with 25 years of experience working in the technology industry.
In the software industry, in which I have worked for the past 25 years, we often use the adage, “What’s old is new again.” Today’s messaging tools like Slack and Teams, for example, owe a lot to the previous generations of messaging like AIM, ICQ and YTALK.
Likewise, the latest trend in the software lifecycle methodology world is “scaling Agile.” (I will use this terminology “scaling agile” vs. “Scaled Agile,” since the latter implies a specific framework or methodology). However, in the rush to address some of the (valid) limitations of current Agile methodologies, there is a danger that we simply reinvent Waterfall. In this article, I’ll look at why that could cause problems.
Why was Waterfall replaced?
Waterfall, the original methodology for software development , was based on having a series of linear stages where you take a customer’s requirements, capture them in a written form, then design, develop and test them in sequence. This sounded good in theory, but each stage took several months. In addition, because the design was usually incomplete and the requirements were unclear, testing was rushed and the final product was buggy and late. Also, by the time the product was finished, the customer realized that it wasn’t what they wanted.
The ” Agile Manifesto ,” announced in 2000, set out to change how software was created. With its emphasis of real-time communications and working software over formal documentation, it was a sea change in how projects were managed.
One of the major practices that developed in the various Agile methodologies (for example, Scrum and eXtreme Programming) was to have short iterations or sprints (two to three weeks) where you would deliver working software in an incremental fashion. That way the customer could provide timely feedback, significantly reducing the conceptual, technical and programmatic risks to a successful outcome. In addition, you had working software at all times, so you could make changes in direction without lots of “sunk cost.”
Why are we looking to scale Agile?
One of the assumptions behind the “Agile Manifesto” and much of the early literature (e.g. Extreme Programming Explained: Embrace Change by Kent Beck ) was that Agile worked best when:
• You had a small team of about 10 cross-functional people.
• The team worked in a co-located fashion and could collaborate face to face.
• Customers were on-site and could give real-time feedback.
• The individual requirements (called “user stories”) were completely independent of each other.
• The project team could work independent of other teams.
However, as IT has become critical to the functioning of the entire world (software is eating everything), the complexity of projects has increased. In a given automobile, there may be thousands of software products that have to be integrated and coordinated. The rise of remote-working also means that teams are often not co-located or even in the same time zone. Increased vigilance around security means that we have to manage the compliance aspects of our software products, making sure that all of the software code has been adequately tested and reviewed.
As a result, companies using methods like Scrum, XP and Kanban are struggling to deliver value at scale. Hence the need to come up with a way to be agile at a larger scale.
What are the pitfalls of scaling Agile?
Consider a retail bank. They have many teams working on various systems and products. They might have teams working on their T24 core banking system, customer website, online account management portal, in-branch teller platform and consumer mobile banking apps for iOS and Android phones. Each one of these products would comprise multiple separate Agile teams with their own backlogs, sprint plans and release schedules.
Now, suppose the company has a strategic goal to “redefine banking” to be the most customer-friendly bank in the industry. How does the organization prioritize all of the work going on in the bank and set a top-down strategic direction? How does it do this without creating a complex hierarchical top-down project plan that resembles the “bad old days” of waterfall?
A major pitfall would be to create a single giant company-wide backlog of requirements, with all the requirements, features and needs managed in a single, monolithic structure. Similarly, having the CIO/CTO manage the entire effort as a single tightly-coupled effort will lead to micromanagement.
Sadly, I have seen this in action, where a developer updated a task and immediately got a call from the CTO asking why he was putting the whole project timeline in jeopardy.
How do you succeed with scaling Agile?
The key is to understand the different levels involved and to ensure that the information and teams are organized accordingly. When scaling agile, there are really three key levels:
• Portfolio (or Department): This level is responsible for determining the key strategic outcomes that the business cares about and allocating them to high-level milestones. For example, “We want to launch our new consumer-friendly concept for retail banking in Q3 2024.”
• Program (or Solution): This level takes the strategic objectives and determines what are the cross-cutting capabilities that need to be in place to realize the vision. These capabilities cut across multiple products and will be mapped to their own high-level timeline. For example, a capability might be “the ability to see account balances across all channels by Q1 2024.”
• Product (or Project): Each product will be responsible for delivering discrete functionality that provides part of the overarching capability. For example, the web portal, iOS app and Android app product teams will each have to have specific requirements (“user stories”) in their backlog that need to be in place for the omnichannel experience to be available.
With this model, instead of having a single monolithic project plan, you have each level working autonomously to deliver the needed functionality, with the organization able to measure the value and prioritize resources to deliver on the strategic objective. That way the teams are agile, but working in support of the greater vision.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?