DevOpsGroup has joined Sourced, an Amdocs company. Read more about this partnership and what this means for the future in our blog.

DevOpsGroup Blog DevOps does not equal

DevOps does not equal “Developers managing Production”

I’ve had a few conversations lately, mainly with smaller start-ups or development houses, who tell me “yes, we work in a DevOps model”. What they really mean is “We pretty much have no Operations capability at all, and we rely on the Developers to build, deploy and manage all of the environments from Development to Test to Production. Mostly by hand. Badly”.

As someone from a predominately Operations background I find this quite frustrating!

Operations is a discipline, with its own patterns & practices, methodologies, models, tools, technology etc. Just because modern cloud hosting makes it easier to deploy servers without having to know one end of a SCSI cable from another doesn’t mean you “know” how to do Operations (just like my knowledge of SQL is enough to find out the information I need to know monitor and manage the environment but a long way from what’s required to develop a complex, high-performing website).

DevOps means Development and Operations working together collaboratively to put the Operations requirements about stability, reliability, performance into the Development practices, whilst at the same time bringing Development in the management of the Production environment (e.g. by putting them on-call, or by leveraging their development skills to help automate key processes).

It doesn’t mean a return to the laissez-faire “anything goes” model where developers have unfettered access to the Production environment 24x7x365 and can change things as and when they like.

Change control was invented for a reason, and whilst change control has becomes its own “cottage industry” involving ever more bureaucratic layers of “management by form filling” the basic discipline remains sound – think about what you want to change, automate it if you can, test it, understand what to do if it screws up (roll back plan), document the change, make sure everyone knows when, where and how you are making the change, and make sure the business owner approves.

When I took over the Operations of a high-volume UK website about 8 years ago I spend the first 3 weekend working fighting fires and troubleshooting Production issues.

My first action after that baptism of fire was to revoke access to production for all developers (over howls of protests). Availability and stability immediately went up. Deafening silence from Development – Invitations to beers from the Business owners.

Next step was to hire a build manager to take over the Build and Deployment automation, and a Release Manager to coordinate with the Business what was going into each release, when etc. End result – 99.98% availability, with more releases, being deployed within business hours without impacting the users, and a lower TCO. The Business was much happier, and so was the Development Manager, as he was losing far fewer developer-hours to fire-fighting Production issues, and hence the overall development velocity improved considerably. Win-Win.

Was that a DevOps anti-pattern? Did I create more silos? Probably… but in a fire-fight a battlefield commander doesn’t sit everyone down for a sharing circle on how they are going to address the mutual challenge of killing the other guy before he kills you. Sometimes a command & control model is the right one for the challenge you face (like getting some supressing fire on the target while you radio in for some air support or artillery!).

That said, once we had developed a measure of stability we did move partway to a more DevOps pattern – we had developers on-call 24×7 as 3rd line support, we virtualised our environment(s) and gave Developers more control over them, and we increased our use of automation.
Organisationally we remained siloed however – we were incentivised in different ways (Operations emphasising availability, Development emphasising feature delivery), we remained in essentially a waterfall delivery model and Ops VS Dev was a constant struggle for manpower & resources. All the usual problems that the DevOps movement is trying to address.

In summary, what I am trying to get at is please don’t devalue the “DevOps” concept by saying you do DevOps when you don’t.

Unless you currently do both Development AND Operations separately, and do them well, AND you’re now trying to synthesise a better, more agile, more cloud-oriented way of working that takes the best part of BOTH disciplines… you aren’t doing DevOps!

— TheOpsMgr

Looking for some help on your DevOps transformation? Have a look at the services we offer.

14 thoughts on “DevOps does not equal “Developers managing Production”

  1. Heya! I know this is kind of off-topic but I needed to ask.
    Does operating a well-established blog such as yours require a massive amount work?
    I am completely new to writing a blog but I do
    write in my journal everyday. I’d like to start a blog so I can easily share my personal experience and thoughts online.
    Please let me know if you have any kind of ideas or tips for brand new aspiring bloggers.

  2. Great article!
    But in reality the mindset that the executives of less than 2000 employee company is that with devops, operations simply totally goes away because the developers takes over the operations. It is easier for developers to take over operations than operations to get some coding practice done (developers will not show their code & code design to operations) even if a ops person can code. In the whole devops execution, ops guys (including good ones) get nothing & get fired. The whole devops thing is all about developers gaining more roles & responsibilities & job satisfaction, and is about total elimination of ops team in entirety. That is the truth.

    1. You know, developers have similar frustrations. Suddenly, they’re supposed to provide support, to be on call and to keep things running in production, with traditional ops people still blocking their access every way they can.

      I have both hats on now. If I absolutely need to, I can restart a container with an image customized for debugging, open a port in it, connect to that image from my locally running IDE, see what’s happening and revert to the old image in less than an hour, possibly a lot less than an hour. In fact, I do it all the time, occasionally in production. So far, this has never caused a problem – because, as me with my dev hat on understand how things work, me with my ops hat on has built a resilient system, which allows a great deal of tinkering without being in danger of catastrophic failure. (Plus, being an automation freak, forcefully healing the system is always just a click away.)

      A year and a half ago, on a different project, with only the dev hat on, I absolutely needed to debug a memory leak in production, because it was the only place where it was happening (due to some very subtle differences in infrastructure setup, it turned out). The same process took a week.

      The product owner of the project where ops were dictating eventually got sacked, and the project went on for a long while eating up funding and producing nothing. The product owner on this new project, where anybody can do anything, is constantly amazed by the speed at which problems get solved.

  3. DevOps and the Cloud is killing the standard I.T. person. Certainly, just as there were plenty of horrible programmers there were plenty of bad I.T. people. For sure, developers make bad operations people and operations people make bad developers. Go ahead and ask your Java or C# developer about configuring routing (BGP for example) or backups!!
    IMHO, there is nothing wrong with a silo. Just communicate. Just integrate. Just stop throwing stuff over the wall like little children. That’s all it really takes.

Leave a Reply

Your email address will not be published.