We need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.
I bet you were wondering when I was going to bring Star Wars to this series – well, I hate to disappoint so here goes!
We have done a lot of groundwork but we haven’t ‘done’ anything yet. Many of you may ask why I have barely talked about DevOps and Agile. What’s all this got to do with delivering software?
Good question – so far it’s been all about the groundwork – creating the foundations to build a high performance teams. If the foundations are shaky, you cannot build skyscrapers. Foundations take time – you are not going to undergo a DevOps transformation in a couple of months – more like a couple of years. Check out Amazon’s letter to their Shareholders – greatness takes time.
I would recommend that you read A Story of DevOps, Origins, One Ring, Freedom, Assemble, Shield and Kryptonian before you read this.
“Everyday we push performance to its limit. Our performance. Our car’s performance. We innovate, we engineer, we design. We master rules and then break them. Only to push further. Past the limits of convention. This is where performance becomes art.”
Yoda is training Skywalker to become a Jedi (I am assuming you have all watched the Star Wars series). Yoda has started to explain what the Force is and how to use it. Skywalker starts to get to grips with using the Force and then Yoda asks him to raise the crashed aircraft from the swamp.
“Try not. Do. Or do not. There is no try” – What a legendary line. Luke struggles to apply the same Force to a larger object – his answer is to ‘try’ and then he fails. Skywalker thinks that the ship was too big to move and Yoda replies with “Judge me by my size do you?”. When Yoda raises the ship from the swamp Skywalker says “I can’t believe it” to which Yoda responds “That is why you fail”.
A little while earlier, Skywalker had to face his fear and I would recommend that you read this article by Brene Brown.
Like Skywalker, we must understand the Force, overcome our fears and insecurities and practice. Once we have mastered the Force, we become a Jedi.
Are you ready Padwan?
In previous articles, we have talked about the Force which consists of great leaders, vision, product teams, tools and environment, let’s now discuss the principles behind the practices.
Core to the principles is the Agile Manifesto:
“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
That is, while there is value in the items on the right, we value the items on the left more.”
This is supported by the following Agile principles:
“We follow these 12 principles:
The principles of Scrum (even if you do not perform full Scrum practices) are also essential. The following are the takeaways from Scrum, Doing Twice the Work in Half the Time, Sutherland.
If I have not mentioned this enough times already, you need to read the Phoenix Project, Gene Kim. In his book, Gene describes the Three Ways.
“The First Way emphasises the performance of the entire system, as opposed to the performance of a specific silo of work or department — this as can be as large a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g. a developer, system administrator).
The focus is on all business value streams that are enabled by IT. In other words, it begins when requirements are identified (e.g., by the business or IT), are built in Development, and then transitioned into IT Operations, where the value is then delivered to the customer as a form of a service.
The outcomes of putting the First Way into practice include never passing a known defect to downstream work centres, never allowing local optimisation to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system (as per Deming).
The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.
The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.
The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure; and understanding that repetition and practice is the prerequisite to mastery.
We need both of these equally. Experimentation and taking risks are what ensures that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone. And we need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.
The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.”
There are many ways to take the above and put them into practice and even with my Product Teams, each team has a slight variants in their application of the principles. However, there are a number of basics that they have all started with.
Warning: The following is not a step-by-step guide for implementing DevOps practices – it’s just how we started. You will need to take the principles and make this work for you in your environment:
This is the starting point. Build a Kanban Board. Don’t do this on paper or in a system – get yourself a big open wall space or a whiteboard. Something you can write on and keep in place – at least until you master the basics. Also, don’t do this alone – do this as a Product Team.
Now draw some columns – To Do, Doing, Done. Does this reflect what you do? Is this your actual process of delivering value? (This is not just for software by the way, I have a Kanban for household chores, DIY, etc.). Modify the Kanban lanes until they reflect what you actually do e.g. Define, Design, Build, Test, Release, Deploy, Done (You know that paper-based business management system that you have locked in some filing cabinet somewhere? That may help…).
Ok – Columns done? Now draw 4 rows. In DevOps, there are four types of work;
Finally, agree as a team on how the work comes into the team – try and eliminate emails, phone calls, taps on the shoulder and have an agreed mechanism of how new work enters the backlog. In our case this happens via the Product Owner for Projects and Change or via the IT Service Management processes and toolset (ServiceNow) for Incident, Problem, Request, Event and Alert Management.
Now, everyone gets the infamous post-its and a marker pen. Create post-its for all the work that everyone is doing and mark your post-its with something like this image.
Ideally, write the task in the form of a Story (As a <user>, I want <feature> so that <reason>). Keep a track of the Task ID as you will need this to help with measurements later on.
Everyone now populates the Kanban with all the work. In the unplanned work, you may have hundreds of incidents – perhaps categorise them and note the volume (e.g. P1 – 5, P2 – 12, P3, 2,500, etc) and place these in the relevant columns.
Finally, near the ‘Done’ column, write an agreed ‘Definition of Done’ so it’s all visible. This may include; Feature deployed to production, user acceptance, customer sign-off, documentation up-versioned, peer review complete, knowledge articles updated, etc.
When you have done this, reflect on what you have just done and what observations you all have as a team. Perhaps, you feel there is a lot of work on the board and in all columns. Perhaps you notice some people have more work than others. Maybe there is a whole load of work that you all were not aware about.
Take a bit of time to re-arrange these post-its so that the ones nearer the top of each row is the highest priority – This is where the Product Owner comes in. You should also remove any tasks that are not needed or not important. ‘Groom’ the board.
If you have time at this stage, use space around the Kanban to put up additional information:
You now need to create ‘flow’. For every column on the Kanban (apart from ‘To Do’ and ‘Done’) set a Work In Progress Limit (WIP Limit). Don’t make this too large and set it to between 5 and 10 items. That is to say that there should not be more that 5 to 10 items in each column. You will find that after completing the above exercise that you will have exceeded the WIP limit in all active columns.
You may now want to look at each of the Tasks and apply estimates to each one. Traditionally, we estimate in time and if you have read Scrum (Sutherland) you will know that this is not the most effective measure of a task. Instead, I would recommend you use relative sizing, the Fibonacci sequence and a methodology called Planning Poker to estimate all the work. You could apply this to both the complexity of the work (Story Points) and to the value of the work (Value Points). Any task that cannot be completed in an agreed time box (Sprint) which could be 2 to 4 weeks, should be broken down into smaller ‘value-delivering’ segments.
Assuming you are not also recording the items in an Agile toolset such as Jira or ServiceNow, put together a spreadsheet and keep a note of each task/story:
If you have a toolset, make sure you can capture the above in the tool and can extract the data or create reports. You should also be able to retrieve service performance metrics such as volume of incidents, SLA performance, availability, mean time to repair, incidents age, etc. from your service management toolset.
From this information you should now be able to create Cumulative Flow diagrams, Burn Up and Burn Down charts and understand your service and change delivery performance through service level metrics, throughput of change, story points and their lead and cycle time.
The Cumulative Flow Diagram shows all the work done as it passes through the stages. It provides a very visible view of the work and good ‘flow’ is represented by smooth upward curves. Spikes and flat lines show poor ‘flow’ and high variations between the various states show potential areas of blockage.
Burn Up Charts are extremely useful for a release. If you know a project is estimated at 1000 story points and your team velocity is 100 story points per sprint you can now estimate that the project will take 10 sprints to complete. You can also make decisions against scenarios where time needs to be fixed or scope needs to be increased. Furthermore, estimates can be made regarding how much value can be delivered at specific points in the project. Actual progress can be tracked against the Burn Up and this provides a very purposeful means of understanding the performance of a team against the estimated work.
A Sprint Burn Down is used as part of a sprint. The team’s velocity is known e.g. 100 Story Points per Sprint and the sprint backlog is set to 100 Story Points and at the end of each day, progress against the burn down is noted. This gives a day by day progress update to the whole team on how they are achieving against the Sprint goals.
If the Product Team supports the product, then unplanned work capacity needs to be factored in. This can be done using the ‘yesterday’s weather’ model which basically means that you take previous sprint’s capacity used for unplanned work, apply seasonal impact where appropriate, and plan this capacity into the Sprint.
Review the service measures such as SLA performance, availability and request fulfilment and review volumes of incidents and request. At your retrospective reviews, understand the root causes and submit improvement initiatives (Kaizen) into the backlog (under Internal Projects) – Do this every Sprint.
Also, review the number of changes delivered in the sprint and the corresponding Story Points. Take a look at the average time taken in each stage of the Kanban and, as a team, look at what improvements can be made. Focus on what the blockers were and optimise the constraint. The Theory of Constraints states that optimising anything other than the constraint is a waste.
The above charts and data points should give you the information needed to ask questions. As mentioned in a previous article, ask ‘why?’ 5 times on all the charts and data points. Take time getting to the root of the performance blocker. Usually, the ‘5th Why’ is a knowledge or understanding ‘thing’ so build this into the next sprint. Take at least one improvement (or Kaizen) into the next sprint. This is how you improve – get it done!
Once you have visualised the workflow and the workload, defined your release plan which is visualised by a Burn Up Chart and you have all the necessary artefacts to measure and visualise performance, the team need to start the ceremonies. This needs to be done immediately after creating the Kanban board. These ceremonies include:
Sprint after Sprint, Release after Release, Project after Project – a united stable team will improve as collaboration, team spirit and trust improves. They will understand the product and customer better and they will become more productive as they both build and support product with the right environment and support around them.
In time, they will remove technical debt thereby reduce incoming unplanned work which will improve user satisfaction as the service improves and they will create more capacity of value delivery and they will streamline the process, remove waste and accelerate the delivery of value. They will innovate and experiment (if allowed to do so) and this will lead to new ideas making their way to production, automation and orchestration of delivery and hackathons with customers.
Over time, the product will evolve beyond what you could imagine or have predicted – some thing may not be great, some ideas may have got thrown away but if you build the right teams, give them the right environment and principles to build good practices upon with time to innovate the improve, they will create a great product but more importantly they will form into an awesome team.
Mastery takes practice.
“Try not. Do. Or do not, there is no try”
We have seen amazing results which include improved service performance, increased throughput, faster and more predictable change, happier employees and more satisfied customers.
Read more of what we achieved in the Story of DevOps in the link below.Read More