Whenever I talk to recruiters and even the odd hiring manager, it is driven home for me just how much confusion there is over what separates a $95K a year “senior” system administrator from a $150K a year “senior” system administrator. They’ve both got senior in their title, and they’ve both probably got about the same amount of experience, measured in years spent employed in the field. So why does the market support pricing in such a huge salary gap?
The answer lies partially with the differences in the kind of experience the two possess, and mostly with the abilities the more expensive sysadmins has that the cheaper sysadmins lacks.
First, let’s look at experience. Let’s say the $95K guy and the $150K guy both have ten years of gap-free, industry experience. The key difference will almost certainly be that the cheaper guy has either or both:
- Worked for one or more educational institutions during his career.
- Tended to stay for longer than two years at several or many jobs.
The first strike against him is that experience gained while working in the rarified atmosphere of a college or university is pretty much useless in the real world. The infrastructure tends to be so different, attitudes about data retention and downtime so laid back, and the politics so thick and cutthroat that the individual is likely to have radically skewed ideas about what a “production environment” really is. They also tend, through no fault of their own, to automatically engage in far more politicking than is healthy for an organization. It takes years of working in the real world to repair the damage done while working at an educational institution, and some never recover from it.
The second strike is that anyone who does the same thing for more than two years suffers from the dual malaise of (a) becoming too set in his ways and losing his agility and willingness to change; and (b) seeing his technical skills stagnate, which is to say in the fast-moving world of Information Technology, fall steadily behind into obsolesce. In terms of technical knowledge, what someone knew and did three years ago is absolutely worthless today (this rule doesn’t apply to general or procedural knowledge, only specific technical knowledge.) Also, as every organization tends to have its own unique way of doing things, complete with workarounds and flaws, someone who works for too long in the vacuum of a single organization will come to forget that the rest of the world isn’t the same. As a result, when that individual moves on to another place, they will be abnormally resistant to change and will often try to recreate their past environment at the new company, leading to a breakdown in teamwork and an increase in politics and strife.
Those are the differentiating aspects in terms of experience between a low-end “senior” sysadmins and a high-end “senior” sysadmins. Let us now turn to specific knowledge, skills, and abilities, as it is within this arena that the largest differences will be seen.
The number one difference between the average $95K sysadmins and the rock star $150K sysadmins is this: the average sysadmin cannot or will not look ahead to understand the impact to the business of his day-to-day choices, while the rock star sysadmin can and does. In other words, the average sysadmin is perfectly okay with making a technology choice or implementation decision that sets in motion a time bomb that will detonate in the future and cripple or even kill the business. It is often something that seems innocuous at the time, or appears to be “easily fixed” in the future, which is then forgotten about until the day it comes back to haunt the organization.
Conversely, the rock star sysadmin always looks ahead, and is willing to go the extra mile now to avoid laying a trap in the future. This often takes the form of doing more research and making better-informed and wiser technology decisions, and in thoroughly and accurately documenting his work. The rock star sysadmin knows that you cannot be promoted if you cannot be replaced; and so does his work and writes his documentation in such a way that when he moves on, be it a promotion within the company or a move to another organization, his replacement can easily pick up where he left off and not ever have cause to curse his name.
The average sysadmin often engages in trying to emulate the rock star sysadmin by putting on his fireman hat and running around screaming. By being “Johnny on the spot” and by perpetually being the boy with his finger in the dike, he hopes to mislead the business into believing that not only is he irreplaceable and an integral part of their continued success, but also that it is only through his superior knowledge and tireless activity that the sky is prevented from falling. But don’t be fooled: it’s all an act except for the sky falling part. In this case, the sky has been so badly put above everyone’s head that should the fireman sysadmin cease in his constant (and needless) ministrations the sky really would fall, and it would be the fireman’s fault.
The rock star sysadmin has no need for these money- and time-wasting antics, as he installs the sky correctly so that it does not fall. The fireman sysadmin is his anathema; why, he wonders, do you want to spend your day running in circles throwing water over an ever-growing number of fires when you could simply build it correctly in the first place, and then focus your energies on supporting the company’s success rather than wasting their time “fixing” problems of your own creation?
Of course, fireman sysadmin fear and loath the rock stars, as they realize that while they may have management and their coworkers snowed about the real nature of the “emergencies” they are constantly “preventing”, a rock star sysadmin will see right through it to the fireman’s incompetency. So if you, as a hiring manager, are ever interviewing a sysadmin candidate who seems really sharp and on-the-ball, and your current sysadmin, who is so busy keeping the ship afloat that he barely found time to interview the candidate says that the candidate is truly horrible and incompetent, hire the candidate immediately and take a good, long look at firing your in-house fireman. Unless your business is that of setting fire to everything around you, you really don’t need a fireman.
The primary willingness that sets the fireman apart from the rock star is that of documentation. A rock star sysadmin is always willing to document his work; in fact, he wants to document it, as he knows that nobody can permanently hold in his head all the details of the IT infrastructure. He also is proud of his work, and is therefore more than willing to show it off in the form of diagrams, flow charts, and wikis. A fireman, on the other hand, is usually resistant to the idea of documentation, as he fears that it may bring to light his incompetence. He often hides this by creating copious piles and heaps of useless documentation: you know it’s useless because it makes your eyes swim within the first page or two, and contains few or no useful pictures. A rock star knows that a picture, or Visio diagram, is worth a thousand man pages, and so good documentation from a good sysadmin tends to be brief, succinct, and contain diagrams that actually shed light on the topic.
Fireman sysadmins are also fond of saying things like how their scripts and code are “self-documenting”, which is really saying nothing more than “I’m too lazy and incompetent to document my scripts.” It also often means that they desire the function and method of their scripts to remain a secret, which is always detrimental to the company in the long run. Fireman sysadmins also enjoy constantly promising documentation, or saying that it exists but is under revision, for as long as humanly possible to avoid having to actually write anything down. And last but certainly not least, beware of the time-honored trick that is a favorite among fireman sysadmins: protesting that as UNIX or Linux experts, they lack the knowledge to use industry-standard documentation tools like Microsoft Visio. They will do strange and inscrutable things like trying to create diagrams in ASCII artwork, LaTeX, or other odd, non-portable drawing methods. They’ll claim they use such either due to familiarity or because they don’t want to be “locked into” a proprietary format like Visio; but the truth of the matter is that they want to obfuscate and avoid documentation as much as possible.
The so-called “soft skills” and certain other non-traditional-sysadmin-skills are things that set the rock star $150K sysadmin apart from the rest. Here’s a partial list of things that a rock star sysadmin is both willing and capable of doing that your average $95K sysadmin won’t or can’t:
- Create a project plan, complete with Gantt chart, detailing all the steps of a major infrastructure upgrade prior to actually beginning the work.
- Write a brief business case or justification for an IT purchase that explains the reasons behind wanting to spend the money in terms that the CFO or CEO can understand.
- Chooses a “boring”, tried-and-true technology over a “cool” one because he knows the “cool” choice will lead to difficulty in support or unexpected costs down the road.
- Is willing to automate himself out of a job, as he knows that true rock star sysadmins are always in demand, and that job security is a myth that certainly cannot be realized through building a house of cards that only he can support.
- Understands the technology down to a level far deeper than anyone else. This level of knowledge may rarely be called upon in day-to-day life, but if the shit really hits the fan, and the company is losing $100K per minute, who would you rather have leading the troubleshooting: the guy who knows just enough to get by, or the guy who understands the infrastructure down to the very core of the operating system kernel?
- Understands that no server is an island unto itself, and therefore possess solid skills in the networking and security arenas. In other words, is comfortable configuring a Cisco switch or router, and in creating firewall rules.
- May dislike meetings, but doesn’t hate them. Is willing to recognize that sometimes the most efficient way of communicating and deciding upon a course of action is, in fact, to gather a bunch of people together in a room.
- Is willing, and perhaps prefers, to walk to someone’s desk and communicate face-to-face than remain hidden behind email or instant messages.
- Dresses appropriately and maintains excellent personal hygiene. Realizes that you can have the most brilliant idea in the world but won’t be heard if your listener is trying not to pass out from smelling your breath.
- Is aggressive and assertive in areas where he knows he’s right, and has the facts to back up his point of view. This latter qualification is important, as nobody likes a blowhard or a bullshit artist.
- Can effectively manage his own time and tasks, and is able and willing to report on his time and tasks to his manager. It’s one thing to claim to be good at “time management”, but quite another to be good enough at it that you actually know what you accomplished in the past week.
- Is comfortable standing and presenting in front of an audience. If you can’t sell your idea, and answer any questions about it, then you really have no business having ideas in the first place. The corollary is that part and parcel of having a great idea is presenting that idea in such a way that the layperson can understand it, and even become enthusiastic about it. In fact, the rock star sysadmin realizes that a lot of projects require a powerful sponsor within the organization that is so enthused about the idea that they are willing to evangelize on its behalf.
- Is at least comfortable using tools like Visio and PowerPoint to capture and present complex subjects in a simple manner. Realizes that if something cannot be diagrammed, then it has no business being implemented.
- Doesn’t engage in petty, useless operating system religious wars. A rock star sysadmin, even if hired to support a large farm of Linux servers, is okay with using Microsoft Outlook on Windows. A rock star sysadmin understands the difference between a core competency (say, Linux) that he is well-paid for knowing versus minimum knowledge in a tool (say, Outlook) that is required to attend to his core competency.
- Understands, and possibly even uses, terms like metric, key performance indicator, milestone, and deliverable. A rock star sysadmin is capable of providing accurate estimates on how long a project or task will take, and seeks to keep stakeholders informed of progress and completion.
And the number one characteristic that sets the rock star apart from the average sysadmin?
- Realizes and uses Information Technology as a means to the end result of empowering the organization to reach its business goals.
That last point is really what separates the rock star from the rest. The average sysamins want to play with technology for its coolness, and think nothing of choosing solutions that may cost too much or not provide enough bang for the buck or cause problems down the road. The rock star, on the other hand, makes his selections based on what will help the business succeed, both now and in the future. In other words, the rock star sysadmin constantly asks himself, “will this increase shareholder value?”
