Automate everything?
I’m always looking for ways to automate myself out of a task.
There’s likely a laziness factor to that urge of not wanting to do something over and over again. And automating something is typically far more interesting than grinding it out. But in a professional setting, there has always been voice in the back of my head, asking: should I really automate this manual task?
Often, the answer is yes. If the task is:
repetitive
error-prone
time-consuming
…it’s likely a good candidate for automating.
Sounds like simple criteria! However, there have been countless times when I’ve found myself pounding away at the keyboard for hours wondering if it’s all really worth it. Because honestly, there is typically no shortage of work to be done on a DevOps team and maybe our time could be better spent.
As an engineer, it’s not just the duty of our manager to keep us on the right path: it’s important for us to understand how our work fits into the goals of our team and into the goals of the company. It’s our duty to prioritize accordingly before finding ourselves a week into a previously estimated day-long task that ultimately isn’t worth the time compared to other, higher impact items to do.
So then, what should we consider outside of that criteria before we find ourselves patching together inadequate automation that took embarrassingly too long to write?
Know your audience
An important first question to ask: who or what will use this automation?
If it’s for yourself for the purpose of getting things done faster: do what you want within reason. For example, if my task requires gathering some specific information off several Kubernetes clusters, I might timebox building an incoherent, one-off Bash pipeline or Python script pieced together from various Stack Overflow answers. But if the internet isn’t on my side, maybe it’s best to keep it simple and do it what takes to get the job done even if more manual effort is involved.
If it’s automation that is expected to be used and understood by someone else, I have some further considerations to make:
How and where will this code run?
How I can make it easier for someone else or some centralized entity to run my code?
Can someone other than myself understand and modify this code?
If I don’t/can’t provide tests, what additional context is needed to ensure someone runs my code safely and as intended?
If I can’t provide these considerations in a timely manner, I have to factor that into my own cost-benefit analysis of providing the automation or reassessing whether it’s worth the time investment. It’s preferable to provide automation that’s packaged with an execution environment like a container; that way the versioning of the languages and libraries are preserved and less likely to rot compared to a script. The last thing you want is to write a script that gets outdated really fast, and only you can understand, maintain, and execute on your machine only.
The maintenance cost
If you’re writing a custom script for a repetitive task, there’s a good chance that script will outlive your tenure at your company. In my experience, I’ve been the proud inheritor of scripts of all kinds: Bash, Ruby, Ansible, Chef, Go, Python…
Understanding and maintaining all of these have a cost and it can really build up on a team. Some implementations can go years without anything going wrong - or being run at all - and typically the original author is no longer there with the tribal knowledge to save the day.
Instead of writing custom scripts, it’s well worth it to explore what’s available through open source first. I’ve been down the road of writing my own Python library or modules for IaC tools like Ansible/Terraform, and if it can be avoided, it’s usually not worth the headache you’ll impose on someone else down the road.
This is why I favor pulling from existing modules, libraries, or frequently/recently contributed open-source projects when possible. There’s a time and a place to write your own scripts, but often someone has solved your problem and you don’t have to reinvent the wheel.
(Vet accordingly as open-source can introduce its own set of problems and unintentionally expose you to exploits through transitive dependencies)
Do it right the first time
If our automation is simply a stop-gap solution, is it worth automating? If the automation is incomplete or error-prone in itself, is the maintenance burden worth putting on the team?
It’s easy to jump the gun and optimize what doesn’t need optimizing in that exact moment. Sometimes it’s better to write a playbook and experience the toil of the task over repeated instances to get a better understanding of what truly needs to be automated. It’s totally fine to leave automating for another time and feel some pain from the toil to better assess how much of a burden it really is on the team.
Ultimately, it does depend on the nature of the task: if it’s extremely repetitive and time-consuming, usually some automation is better than no automation - if it can be implemented quickly. I’m a big believer in fast feedback loops and iteration to get better working solutions.
The key point: if you find yourself taking too long to implement an intermediary solution that is likely to be addressed by a more robust, planned solution down the road, it’s probably best to wait it out and get feedback from your team if the automation is really worth the time investment.
Final thoughts
If you made it this far, you’re probably wondering why I haven’t linked this yet to summarize my whole post:
-30-