by Katie McLaughlin
“So, Katie, what do you do?”
K: “I work in tech”
“Oh, ok, so you make (apps|websites|games)?”
For any given dinner party, if you say that you work in tech, many people expect that you are doing something ‘cool’; making computer games, websites, the next iPhone app, or the like.
Not all of us get to work on the next latest and greatest thing, however.
K: “Uh, no, I keep the cloud running.”
Thankfully, most people at a sufficiently diverse dinner party know of this ‘cloud’ concept and leave it at that, but occasionally, there’ll be that one person.
“Oh, you’re devops! What’s your stack?”
This is the question I hate to answer. It normally goes one of two ways:
K: “It depends, I’m a polyglot, so I just fix what’s broken.”
K: “It’s legacy, so there’s a bit of Perl, PHP..”
“Ew, you have to use PHP!”
Many people in tech aren’t working with the latest and greatest tools, making the next big thing. A lot of us work with legacy systems: systems that have stood the test of time, where the cost to replace them highly outweighs the cost of keeping them up. Especially in infrastructure deployments, there’s an extremely high cost and risk in changing things, so people are paid to learn the previous best thing and keep it stable.
I’ve spent many hours spelunking through Perl applications that serve as ticketing systems; PHP systems that run behind the monitoring systems; bespoke applications grown internally that serve needed purposes, where no off-the-shelf or SaaS could compare in feature parity.
It’s extremely satisfying to get to the bottom of a particularly annoying bug. In one instance, I spent a whole lot of time diving through Perl in order to find the source of a particular issue: we could link tickets in our incident tracking system, but any time we did, a link to a mythical ‘Ticket #1’ would also be added. This made for interesting effects, as the linked tickets liked to also list their parent and child ticket links. It created a very boggy system with slow page load times.
I dove through the local fork of the code base, comparing it to the open source upstream code, now several years past what was installed locally. I found the bug had been fixed in later versions, but the code base had shifted, so it wasn’t easily patchable. I also found a whole lot of internal customisations in our environment that made the software work for internal business cases, but prohibited upgrading. I eventually found the issue was related to how there’s no such thing as ‘booleans’ per se in Perl. The 1 value was being interpreted as literal integer rather than True, and it was using that as a primary key for the ticket, which was how the ‘Ticket #1’ linking situation happened. Instead of the logic asking “Is there a link? True!”, it was saying “Is there a link? 1!. Ticket #1? Ok then! link”… A bit of dynamic backporting later, and I’d fixed the issue, much to the merriment of my colleagues. They were glad to see the performance improvements, and thanked me for my ‘sacrifice’ in having to use Perl.
It’s not just internal bespoke versions of applications that can lead you to learn languages you don’t otherwise ‘need’ to know. There are so many legacy tools in the standard Linux operating system that to work in the world of DevOps it really helps to have a general understanding of 15-25 year old systems.
For example: man (manual) pages. Yes, you could search Google for the HTML-rendered versions of these things, or you could use the man command on the system. The man page for any particular app has it’s own specific way of working: man serves up the file on disk using a program called less, which allows such things as pagination and searching by keyboard controls. less itself is a rewrite of an older file pagination tool, called more (because less is more [ba-doom-ching!]). Plus there’s the arcane art of the man-page section number, because there’s a difference between, say, the system call stat and the shell script stat, sometimes referred to as stat(1) and stat(2), which are two separate entities, each with their own man page.
Or Git. A lot of companies nowadays are working with GitHub, either in private or in the open, but the underlying technology in Git is over 10 years old. And most of the functions on the command line are just shell scripts! The git submodule function is a 1000+ line shell script that calls out to other Git functions, some of which are more shell, and some of which are compiled C. Until recently, submodule contained embedded Perl code to handle some of the more interesting regular expressions required. Which means that you can’t use git submodule unless you have Perl installed on your operating system. Normally this wouldn’t be a problem unless you’re using a containerization operating system such as CoreOS, which by default doesn’t come with any scripting language environments. The shell script Perl dependency was removed around about git version 2.7.0, so depending on which version ships in your operating system, you may or may not have to have Perl to use the complete git functionality.
The only reason I know the specifics about the way that man works is because I read it in a book. But I learnt about some of the internals of git and its previous reliance on perl by hours of debugging and code spelunking. The latter was much more fun.
There are people who have to keep the lights on, keep the systems running, and that’s what I’m good at. To be taken as subpar because of the technology I use is degrading.
I admit, I’m not immune to this kind of behaviour. Before Perl I worked in the Oracle and Windows space, two words that are marked as ‘evil’ in some open source circles, and two words that seem to keep being picked up by auto-recruiter bots for years afterwards. As a coping mechanism, I’ve used self-deprecating language in the past. Now that I’m no longer working in that space, I’m can pass as having ‘done my time’, it can be far too permissible to take cheap shots at those still working with the technology that I’ve long since left behind.
Many people have no choice in their tech stack. They have no option to re-write. They have to keep the machines running and the wheels turning. They gain experience in these stacks, and then actively choose to continue working in these technologies, even across companies. To tell them that because their tech isn’t ‘cool’ or ‘hip’ is erasing their experience as irrelevant.
As much we can see our own platforms as perfect or without fault, some tools are objectively malformed or broken.
We need to break our pack mentality that different segments of tech are better than others. At the end of the day, we are all trying to solve problems and provide solutions to the best of our ability.
Most at the dinner party I started with can’t tell the difference between when they are using the web or various apps, or which underlying tools built the technologies they use, so why must we create artificial barriers that impede progress? Why must we have these walls?
As a polyglot, I interact with many different segments of the tech community, across languages and up and down the stack. I also have experience with legacy tech, so I can work on older and newer stacks, across the expanses of time (whee, I’m a time-lady!). There are so many ‘solved’ problems that keep being ‘resolved’. Nothing in tech is ever new. The legacy systems can teach the new-age systems as lot, if only they are willing to learn. The reverse is true, too. The new-age systems can show their worth, and some of their lessons can be retroactively applied to legacy systems.
If we can share our knowledge, and not degrade each other for their choice of dialect, then we can create a more advanced knowledge base to draw solutions from, and improve all our systems for the better. At the end of the day, someone needs to keep the lights on and keep the systems running. When you have a nicely-tuned and humming system, it’s awesome. It’s what I’m good at, and it’s gratifying as hell.
Illustration by Victoria Wang.
We have a print edition too! Find this issue in the shop.