Written by Raj Fowler
DevOps is transforming businesses of all shapes and sizes globally, and aerospace giant BAE Systems is one of them. During my time at the company, I spent two years implementing the philosophies and practices of Agile and DevOps to help accelerate organisational growth and performance.
To me, DevOps isn’t just getting to a stage where you can release 100s of deployments a day. It’s a revolutionary cultural mindset, underpinned by Agile practices and philosophies, and combined with those of Scientific Management, Deming Cycle, Lean, Six Sigma, Scrum, ITIL, high-performance teams, and human psychology.
At the end of the day, to create valuable products that resonate with customers and increase profits, organisations must foster an environment capable of responding and adapting to market changes. By rolling out DevOps, Agile, and Lean Six Sigma across BAE Systems – encompassing practices including Kanban and Scrum – my team and I were able to create such an environment.
Taking these steps simplified internal operations; improved the delivery of back office systems such as SuccessFactors, ServiceNow, Salesforce, Cognos, and bespoke applications; and modernised internal business processes. But that’s not to say everything was plain sailing. Challenges included factoring in unplanned work, adhering to customer needs, and creating a mindset of shared responsibility. In this article, I’ll take you through the DevOps journey of BAE Systems.
In the Phoenix Project, Gene Kim highlights the issues posed by traditional delivery mechanisms. Three years ago, it felt like the BAE Systems business was stuck in the early chapters of this book. We were delivering between 50 and 100 changes per annum across 80 systems, all of which were managed through standard structures and practices. The result? Change was not only difficult, but project-to-service transitions were painful.
My role saw me overseeing more than 100 business systems, but we categorised eight of them as products and put product teams in place to manage them. We also delivered a staggering 2,500 changes per annum across these products, while improving service (by 35%), customer satisfaction, and employee morale. What’s more, change is no longer a big event but something that happens every day, and teams are delivering new features while cleaning up their code and removing technical debt per release. We were also able to do these things with little automation.
Leading this initiative was a surreal experience. Having worked in a blue-chip company for 20 years, I’ve never felt so vulnerable and isolated. However, at the same time, I feel enormously proud and connected. Driving much needed change for an organisation with over a century’s worth of heritage and being exposed to a diverse culture was a unique opportunity, but extremely difficult too. For most organisations, transitioning to DevOps will be completely new, and that was certainly the case for BAE Systems. We were embarking on a major journey with uncertain outcomes, limited organisational understanding, and no comprehensive plan.
Another challenge is the fact that, unlike factories, most of what we do in IT is invisible. But of course, a big part of DevOps and Agile working is breaking down silos and boosting collaboration between teams. When you try to make the invisible visible, you have to deal with what you see. To do this, there are a number of foundations you must build – these include:
1. Executive sponsorship
Before enabling change, we had sponsorship at senior levels of the organisation. It was essential to see this as a cultural transformation and work out the path we needed to take. We understood that this journey wasn’t something we could buy or replicate. There was one sponsor who championed this movement from the top, and that was only made possible because of belief and trust.
Initially, we developed a business plan detailing the structure and approach we’d take – including education, working with the right coaches, and measurement tools. When challenged about the things we needed to enable this transformation, I responded with education and time. Every transformation is different, and it can take up to three years before mature transition occurs.
3. Product teams
We set up product-centric teams consisting of new roles such as product owner, product technical lead, product delivery lead, product engineer, product analyst, and product admin. What sets these roles apart is that they not only build but support the product. For example, the product owner owns the requirements, service, and associate performance of the product.
4. Visualising the work
To eradicate silos and ensure everyone is on the same page, visualising work is crucial. We did this by creating a Kanban board with backlog, define, execute, release, and done columns, along with a definition of done. Soon, everyone could see the things they were working on, focus their time, and raise any blockers.
5. Practices and ceremonies
As we learnt about the ceremonies of scrum, we arranged stand-ups, sprint reviews, sprint planning sessions, retrospectives, and backlog grooming as part of two-week sprints. From the beginning, we capped our stand-ups to 15 minutes and worked backwards to finish work we had started. Our key question was: Is there anything blocking progress? Holding these ceremonies improves communication and visibility within teams.
6. Improving the flow
Kanban boards are great for improving visibility, but you can’t rely on them alone. To help improve flow within our teams, we applied Work-In-Progress Limits and visualised the movement of work using a cumulative flow diagram. You can then identify and monitor blockers.
We also used Swarming to improve team working, sprint, and release burndown charts to identify constraints, as well as the Yesterday’s weather technique to better manage unplanned work. The focus shifted to “Service is King, Change is Queen” – meaning that when incidents come in, Service Level performance was a higher priority than change.
7. Customer engagement
To generate constant value for customers, we had to bring them on this transformation journey. We needed to prioritise requirements, batch them into releases to deliver value incrementally, agree and deliver on Minimum Viable Product, and implement a healthy backlog of customer requirements.
8. Estimation and velocity
To help improve the way teams estimate the projects they’re working on, we introduced relative sizing and story points. First, you define a reference point, which is a story or feature that’s the least complex thing to do. And for more complex tasks, we began using the Fibonacci sequence (1, 2, 3, 5, 8, 13, 21, 34, 55, etc). Now, teams are using collective estimation to size work at the story, epic, and theme level.
9. Engineering excellence
A big part of this transformation initiative was removing debt by cleaning up code and configuration, upgrading our platforms, and simplifying workflows and other internal processes. We not only became more adaptable to change but reduced incoming incidents by 35%.
We also shifted the focus to product-centred thinking and worked hard on transforming our engineering architectures and methods – considering what would deliver value. Our engineering teams were challenged to look at methodology – how pipelines are built, how requirements are captured, what good looks like, and what “done” means – and the way products are monitored and managed.
Consequently, we combined Product Engineering Plans and Service Management plans into a Product Engineering Plan. Standards and structures are now applied in workflow management tools such as Kanban Board, Jira, and ServiceNow.
10. Automation and orchestration
By monitoring key performance measures, progress against projects, and alignment of work, teams have access to a lot of actionable data and can see where the constraints are.
Automation is a great way to get everyone focusing on more important tasks and innovating, but reaching this stage isn’t a quick process. First, you’ve got to improve architecture, improve working practices and engineering practices, and eradicate technical debt.
We experimented here by choosing a team and automating the regression test pack, with most product deployments already automated. In the future, BAE Systems will need to think more about test-driven development and build its governance processes into the automation.
At the start of the year, our product teams held a retrospective to come up with themes of improvement for 2018. Using visible data, they chose to focus on customer engagement, Agile and DevOps contracts, technical development, automation and orchestration, and engineering excellence.
I’m really proud of the things that the teams at BAE Systems achieved within a small space of time. Three years ago, we were implementing 50 to 100 changes annually. Now, after embarking on a DevOps transformation journey, BAE is delivering more than 200 changes per month. But, more importantly, change is the norm.
However, despite these great things, DevOps is not the goal. And Agile is not the goal. Yes, we optimised the delivery of IT products, but this isn’t the “whole system”. Goldratt says optimising anything other than the constraint is a waste. The goal of a business is to make money, and just making IT better does not lead to better business performance.
It’s important to deliver great services and deliver faster, high-quality change. But if the things you’re delivering are wrong in the first place, what’s the point? We need to consider the whole system and make sure that we are changing our systems to deliver business value.