DevOps Essentials

I came across my notes from a course I took on DevOps from “A Cloud Guru” called Dev Ops Essentials. DevOps is one of those concepts that came to be after I finished my degree, and honestly was nothing I have encountered formally in a tech job. When I did do development, and that is being generous with the term, the Senior Developer on the team took my code and deployed it manually. I’m not sure if Dev Ops is still popular, if the role itself has morphed into something else, or its been automated away into workflow, but this course helped me understand it a lot more. Here are my notes.

When people first hear the term DevOps, they often think about tools like Jenkins, Ansible, Docker, or Kubernetes. Those tools are certainly part of the conversation, but DevOps itself is not really a product or a specific technology. It is more of a culture built around developers and operations people working together instead of working in separate silos. That sounds simple, but historically those groups have not always had the same priorities. Developers generally want to move fast. They want to release new features, fix bugs, and make improvements as quickly as possible. Operations teams are usually focused on keeping systems stable, available, and predictable. Those goals can easily come into conflict. DevOps is meant to close that gap. The idea is that an organization should be able to move quickly without constantly breaking production. Ideally, the result is faster time to market, fewer failures, and faster recovery when something does go wrong.

This is also why DevOps and Agile tend to go hand in hand. Agile encourages smaller, more frequent changes rather than huge releases after months of development. DevOps takes a similar idea and extends it across the rest of the software delivery process. One phrase that comes up a lot in DevOps is “throwing code over the wall.” In a traditional environment, a developer might finish writing code, hand it over to another group, and basically consider the job done. If the code fails in production, it becomes somebody else’s problem. DevOps tries to get rid of that mindset. The same goes for the classic phrase, “It works on my machine.” Usually that means the developer’s local environment is different from production, or the developer does not have much visibility into how the application actually runs once it gets deployed. DevOps tries to make those environments more consistent and gives developers, QA, and operations teams more shared ownership of the outcome. Automation is a huge part of making that work.

In a DevOps environment, a lot of the pipeline can be automated. Code can be integrated, built, tested, deployed, monitored, and, if necessary, rolled back without someone manually handling every step. Build automation is one example. Instead of relying entirely on a developer’s IDE and whatever settings happen to exist on that person’s machine, builds can happen centrally using a consistent process. That makes the results more reliable and reduces the chance of a bad build caused by differences between environments. Continuous Integration, or CI, fits into this as well. Rather than developers making large changes and merging them all at once, CI encourages smaller and more frequent commits. Every time code is committed, a CI server can automatically build the application and run tests. Jenkins is one of the best-known tools used for this. A developer commits code, Jenkins kicks off a build, and if something fails, the developer can find out almost immediately. That matters because problems are usually easier to fix when they are caught quickly.

If a test fails after one small code change, there are only so many things that could have caused it. The developer can fix the problem or roll the change back without having to dig through weeks of changes from multiple people. The same idea applies to deployments. DevOps does not assume that failures will never happen. That would be unrealistic. Instead, the goal is to make failures less painful. If a deployment causes an issue, an automated pipeline may make it possible to roll back quickly, fix the problem, and deploy again. The real advantage is not just preventing failures. It is being able to recover from them quickly. Monitoring becomes extremely important for the same reason.

Monitoring tools can track things like CPU usage, disk activity, network traffic, response times, and application errors. Application Performance Monitoring, or APM, goes further by looking at logs, transactions, and application behavior. The point is to identify problems before users are the ones telling you that something is broken. Monitoring can also work together with orchestration. If an application instance becomes unhealthy, an orchestration platform may be able to remove it automatically and replace it with a healthy one. This is where technologies like Kubernetes come into the picture. Kubernetes manages containerized applications across multiple systems and can help with scaling, availability, and recovery. If demand increases, it can add more application instances. If an instance fails, it can replace it.

Docker is another major piece of the DevOps world. Containers package an application together with what it needs to run, which makes it easier to move that application between environments. Containers can help solve some of the same problems behind “it works on my machine” because the application is packaged in a more consistent way. Before containers became as common as they are today, virtualization was one of the major ways companies made infrastructure more flexible. Hypervisors such as VMware products or Citrix XenServer allowed multiple virtual machines to run on the same physical hardware. Containers take a lighter-weight approach. Instead of creating a complete virtual machine for every application, containers share more of the underlying operating system while still keeping applications isolated. Microservices fit well into this environment too. Instead of building one giant monolithic application, a microservices architecture breaks an application into smaller services that communicate through APIs. Those services can often be developed and deployed independently. That gives teams more flexibility. One service could use one programming language while another uses something completely different. A change to one service may not require the entire application to be rebuilt and redeployed.

Configuration management is another major part of DevOps. Tools such as Ansible, Puppet, and Chef help automate the configuration of systems instead of relying on administrators to manually set up every server. Ansible is especially interesting because it uses a declarative approach. You describe the state you want the system to be in, usually using YAML, and Ansible figures out the steps needed to get there. It also does not require agents to be installed on every machine. Puppet also uses a declarative model, while Chef has traditionally used more of a procedural approach. The larger idea is that infrastructure should be repeatable. If one server is configured correctly, the goal should not be to manually recreate those same settings on the next ten servers. That process should be automated. Infrastructure as Code takes that idea even further. Tools such as Terraform allow infrastructure itself to be described in code. Instead of clicking around a cloud console to manually create servers, networks, and other resources, those resources can be defined in configuration files. This means infrastructure can be versioned, reviewed, reused, and recreated much like software code.

Terraform, Ansible, and Kubernetes are sometimes mentioned together because they can complement each other. Terraform can provision infrastructure, Ansible can configure systems, and Kubernetes can manage containerized workloads running on top of that infrastructure. Cloud computing has made all of this much easier. Services such as Amazon EC2 and Google Compute Engine provide Infrastructure as a Service, or IaaS. Platforms such as Google App Engine or AWS Elastic Beanstalk move further toward Platform as a Service, or PaaS. Serverless computing moves even further away from managing infrastructure. With services such as AWS Lambda or Google Cloud Functions, developers can deploy relatively small functions and let the cloud provider manage most of the underlying environment. All of the major cloud providers now offer their own collection of DevOps tools. AWS has services such as CodeBuild, CodeDeploy, CodePipeline, CloudFormation, Lambda, and CloudWatch. Google Cloud has Compute Engine, App Engine, Cloud Functions, Google Kubernetes Engine, and tools for interacting with its APIs and services. Microsoft Azure has its own container services, application hosting, source control and CI tools, and monitoring through products such as Application Insights.

It is easy to get overwhelmed by the number of tools available, and that can sometimes make DevOps seem more complicated than it really is. The tools matter, but they are not the point. A company can install Jenkins, Kubernetes, Terraform, and every other popular DevOps product and still not really have a DevOps culture. If developers, QA, and operations still work in isolation and blame one another when something breaks, the technology has not solved the real problem. To me, the most interesting part of DevOps is that it challenges the idea that speed and stability have to be opposites. With smaller changes, automated testing, consistent builds, monitoring, containers, orchestration, and better collaboration, it is possible to move faster while also reducing the risk of each individual change. That is really what DevOps is trying to accomplish. It is not about Jenkins, Kubernetes, Docker, or any single tool. It is about building a process where the people responsible for creating software and the people responsible for keeping it running are working toward the same goal.