Help your team understand how to measure DevOps outcomes and agree on possible strategies to achieve them.
The DevOps Metrics Canvas is broken into 4 boxes, the aim is to complete these boxes with your team during a short (70-120 minute workshop), as a result of this you will have:
This can be a challenging workshop and it will benefit from a strong facilitate and good preparation. It can be hugely beneficial for teams and works well when aligned to goal setting frameworks such as OKRs.
See our instructions for how to run this workshop below.
Download the DevOps Metrics Canvas for use with teams in your organisations and explore new ways to collaborate.Download Canvas
Step 1: Understanding your outcome (5-10 mins)
The first section of the workshop is for the facilitator to present to the team the outcome that will form the core of this workshop. This can then be written in the “Outcome” box.
Hint: make sure you are focussing on an outcome and not on outputs (see below) and make sure you spend some time ensuring everyone has a clear understanding of the outcome, this will make the remainder of the workshop much clearer.
Step 2: Measuring the outcome (15-20 mins)
The second section of the canvas to be completed are the measures for the outcome (bottom right). This box contains the data points or proxies for the outcome we have established.
The team now works to identify measures that ensure they are achieving the outcome. These should be written on post-its and stuck in the outcome/data box. Once the team has identified all of the options they should be prioritised and ensure the selected measures (3-7) contain counter-measures (see below).
Hint: a valid data/proxy must be something that you can measure but it doesn’t need to be easy to measure! Just because something is difficult to measure doesn’t make it the wrong thing to measure. Validate your proxy by ensuring that the following makes sense – If we see an increase/decrease in [data/proxy] we might infer that we are achieving our outcome. (Because outcomes are typically complex one proxy in isolation is unlikely to prove it is being achieved).
You should look to have approx 5 data proxies that have been agreed by the team, prioritised and confirmed for balance when you finish this section.
but just because something is hard to measure.
Step 3: Inputs (20-30 mins)
In this section of the workshop the team identify their ‘Inputs’. Inputs are the actions/activities that the team believe will help achieve the outcome as measured by the values determined in step 2. The team should write down as many ideas as they can within the allotted time before grouping and ranking them to end up with approx 5 activities. It sometimes helps to think of these are the levers that we can pull to help us achieve our outcome.
Hint: validate that your input is valid by asking yourself the following – “We believe that doing [action] will drive/impact/ correlate to our [outcome]. It is common for people to try and put input measures in this box not the actions. In this case ask the following: “what would you do to move that number?”.
You should look to have approx 5 data actions that have been agreed by the team and prioritised.
Step 4: Input Data/Proxies (20-30 mins)
For each of the actions prioritised in Step 3, the team now have to ask themselves ‘how will we measure that we have performed that activity’. This should result in a list of measures aligned to your actions.
Hint: This sometimes works well by dividing the team into smaller groups each working on one activity. This is especially useful if you are running short of time.
You should look to have at least two measures for each action by the time you finish this section.
Step 5: Concluding the session (10 mins)
To wrapping up the session you should summarise the canvas and agree on next steps with your team. The canvas can be summarised as follows:
As a team we are committed to [outcome]. We will know we have achieved this outcome by monitoring the following measures [outcome data/proxies]. To achieve our outcome we believe focus on [actions] will deliver the outcome. We will track these actions by monitoring [input data/proxies].
The Metrics Canvas is Copyright DevOps Research and Assessment LLC.
Running this workshop successfully requires that you go into it with a clear outcome for your team and appreciate the difference between an outcome and an output (actions in the above model). It is the outcomes that the business wants and needs and typically multiple outputs might align to the same outcome.
An output is likely to focus narrowly on the things your team or organisation do an outcome will articulate the reason they are doing it.
A simple analogy for this would be a highway agency (as aligned to our four boxes above):
While this example is vastly over-simplified it shows how outputs and outcomes come together in this model.
When you are identifying metrics it is important to make sure that any metric has a counter-balance. This is to prevent us from driving undesirable behaviours/outcomes within the organisation (a phenomenon sometimes referred to as the cobra effect). A good example of this would be attempting to increase the number of people visiting a company website. While ‘number of visitors’ is a great metric it should also be combined with another metric (session duration or pages per session perhaps) to ensure these are the right users.
Examples of getting this wrong can be seen in high profile case of the Wells Fargo account fraud scandal where poorly balanced measures (number of accounts per customer) drove unwanted (and in this case illegal) behaviour.