I'm a sysadmin for 16 years, I have worked in big and small companies and I don't agree with a great part of this post.
I'm not a developer, I have never needed to disassemble or fix programs built by others, never needed to understand kernel subsystems or anything else kernel related (except may be how to replace a broken driver/module). I know nothing of C, I know just the basics of system calls and I've never heard of skb or select() and epoll().
I don't even like developing software, which is the reason I once joined the sysadmin team at the company I worked for and fell in love with this job.
I see how these skills are a nice extra, but I understand that knowing too much also creates the problem of having to work harder and stay more often after hours. And saying no isn't an option because nobody else knows how to handle the problems that may arise.
To me, a good sysadmin knows when to say no and spends the least effort in fixing errors in code. The developers should provide good programs or it doesn't get installed in production. Keep strict boundaries in what you do and don't to prevent being abused by your employer. If you are often the last one to leave because you're doing work that someone else should be doing, or asked to be stand by after hours while being the only sysadmin, step on the brakes before this will eventually burn you out or makes you hate your job (or even hobby).
Also, knowing all of what parent describes + all the 'normal' sysadmin knowledge is (eventually) impossible to keep up and will also take too much of your free time. You will regret this later in life.
You are literally the problem with system administration.
The attitude that you don't need to know how to program or understand the architecture of the systems that you run on is a highly privileged attitude that you are practitioner who can remain ignorant of their tools.
The good news is that kids these days are being raised on SREs and there's a billion people in India who will take your job and won't have an attitude about learning how to program. Ultimately, you are going to be a curious dinosaur. You are a unique product of the late 90s Internet bubble where the need for system administrators expanded so much faster than the available supply and anyone halfway technically minded got hired for really good salaries and put in charge of servers.
That is going to change and evolve.
And I spend 5-10 years learning all that knowledge back in the 90s and its 15 years later and I can state with confidence that there's no regrets. If anything my only regret is that I didn't learn how to really practice "software development" as opposed to "programming" even earlier. Going forwards, I expect that more an more the lines between systems and software is going to blur and MY advice to kids these days would be that they will only be able to be ignorant of software development practices at their own peril.
And I stated doing PC Tech work in the late 80s, became a Unix admin in the mid-90s, managed the configuration management system and base configuration of a site that grew to 30,000 linux servers in the 200Xs, and then after 15 years switched to being a Software Developer. I'm fully confident based on experience that you are offering absolutely terrible advice to someone who is just learning, and you are out of touch with the direction of your own field.
Your post seems to boil down to a logical conclusion that everyone better be a software developer, as if there is no value in abstracting to a layer where things aren't pieced together with compiled software. Feels rather myopic to me. There will always be work to do at every layer
I agree. When we hire sysadmins, we look for folks with experience using various products and weigh in extra bonus points for folks who are also SecOps. We could care less if a person knows how to code. That's what we have the developers for.
Those end up creating environments where everything just sorta-works enough to run, for as long as you put a constant stream of sysadmin time into it.
And then as see they doubting people that claim that a person can administer hundreds of servers, or that an admin can go away for a while and nothing will happen.
(Also, exactly what do you look for in SecOps? Most people that use that term are selecting for exactly the wrong things.)
I was doing "SRE" level system engineering at Amazon back in 2001 long before facebook ever existed.
Also when I grew up as a junior system engineer in the mid-90s I looked up to "system administrators" like Wietse Venema who coded in C and wrote tcp_wrappers and postfix, and Larry Wall who wrote his own programming language.
The System Administration / Software Developer is not a new invention of Facebook/Google. What is "new" is the "System Administrator" that started getting hired in the late 90s who couldn't do anything other than install programs and configure them and maybe knew how to 'ps' and 'kill' and that's about it. Even the old school Enterprise-class system admins knew more about debugging crash dumps and using their tools even if they did call up IBM when they needed a patch.
Very true, I should have been more specific when I say "created". Popularised is probably a better term.
I worked with some old and crusty Perl and C hackers that called themselves sysadmins that definitely would have qualified as "SRE"s during the 90s, it's just they were taken for granted.
All the term SRE has done is create a role that is more appreciated and better compensated with a better understood set of responsibilities and required skills.
I'm sorry for you, my fellow sysadmin who has more years on his/her shoulders than me. I'm sorry because you lack the the ultimate knowledge needed to be a good system administator. You did not ever get hold of it.
The sysadmin practice is not only installing software and OS, doing basic troubleshooting. It's more than that. It's knowing systems inside out, from kernel internals to application internals. It's knowing and handling more area than anyone else in the field.
If you do not know about these, you do not automate, yo do not code, you do not debug, you become an IT helpdesk employee.
Absolutely agree. Sysadmins' interface with code is basically to be a compliance-checker. Software, be it from in-house developers or a third party solution need to meet that compliance bar, or it doesn't go to production. If your in-house developers are unable to meet those compliance requirements, that's a them issue and not a you issue.
It's good for devs to know some ops, and ops to know some dev from a familiarity standpoint, particularly because mutual understanding fosters a good working relationship, but they're two very different roles.
Continuous Delivery is going to eliminate that from the equation. The software developer will write software and tests and if they pass the tests throughout the pipeline then it will ship to production automatically and potentially multiple times per day. Much faster than the heavy ITIL based change control process that requires you to be inserted in the middle of it will manage.
The companies that do this will have a velocity that outstrips the companies that do what you describe and will naturally have an advantage in the marketplace.
There will still be room for security compliance and someone in the company to be responsible for that, but they'll be writing tests that check software for compliance automatically, they won't be some drone in an office looking at a form and stamping their approval on a change request.
Your industry is changing. You may retire before it really hits you in the ass and kicks you out, but your advice for people getting into the industry is terrible.
The amount of examples I have in my daily job demonstrate that no, the concept that developers won't have any need for IT staff is a day that is never going to come. And while I make a point to have a relatively varied skill set, the truth is, trying to be both a developer and an admin is going to just mean you suck at both jobs.
Agreed, even finding a sysadmin-only that knows something is hard.
When I talk to dev that have no idea about system, security, networking, etc: forget about them trying to wear 2 hats...
You still need a sysadmin to set up your CI systems in a continuous delivery world. You still need people to debug performance issues -- most developers don't know enough about system I/O to do that effectively.
You're right in that there's no longer some heavy-handed change control process, but wrong that sysadmins will go away. We just call them DevOps instead, but it's the same skill set, just embedded inside a dev team.
In order to be able to usefully debug perf issues you need to have a fairly deep understanding of systems architecture though or you're no better off than the software devs that you're slamming. There are a lot of "I don't need to learn programming" system admins whose knowledge of how to debug performance issues I could teach to a decent software developer in an afternoon. There are even more SAs who cargo cult completely incorrect assumptions and tend to throw money semi-randomly at problems because the last time they had a wonky server they upgraded the RAM so this time if they spend the money or more RAM that'll fix it too.
And yes, I've had to painfully explain why Linux servers report nearly all their memory as being fully used and what the VM free is was to a Principle Architect at a major internet firm with a string of PhDs and who had built the software architecture of the company from its founding. Controlling a huge chunk of the technical direction of the company and doesn't know the first thing about the VM or how to tell if a server is really out of RAM or not. There's still going to be a skillset which is closer to the hardware, but people need to really have that skillset. Most people who call themselves SAs do not actually have that skillset, they're just semi-technical people who wield root power and throw their weight around the company.
Yeah, a system administrator who doesn't know how to program is just as useless. Why avoid a supremely useful tool because you have some personal aversion to it? You describe people who really don't have a technical mindset to problem solving and were never taught that you need data to back up your hunches. Unfortunately, a lot of those people tend to get stuck in the "system administrator" role because many sysadmin tasks can be easy and repetitive, and putting them on those tasks keeps them away from anything they could break.
Architecture is a "10,000 foot view" type job. I do a good bit of architecture work, and I don't give a flying fuck about how much RAM is in a server or how many CPUs it has. I care about the function the server performs, whether that performance is adequate relative to current demand, and whether the architecture can scale to handle 10-100x that demand. When I was a sysadmin, I used to care about RAM/CPU tuning, but it's not relevant to my current work so I have forgotten a lot of technical details. I do know that if I think a system isn't adequately tuned to its use case, I can go talk to my performance testing team and they will investigate, generate a hypothesis and test that hypothesis.
The entire problem here is the division of "us" and "them". From the customer's perspective, you are both "them". If the product doesn't work, you're both to blame no matter whose fault it was.
Work with your developers to make the application more supportable. Show them the problems you are having and ask for their help in fixing them. Too many sysadmins just throw it over the fence and say "not my problem, you fix it" and that's honestly not acceptable.
This is the entire reason "DevOps" is a thing. They are not fundamentally different roles; you're both involved in building a system that does things for customers. You bring different skills to the table, but that's often the case with every developer: you probably have a dev who is a whiz with databases, another who knows some other library really well... a sysadmin skill set is no different.
"Us and Them" makes it sounds adversarial, but it's not meant to be. It's about who is best equipped to fix the problem. More often than not, a problem with an in-house developed piece of code will be best fixed by the person who wrote that code.
I'm not a developer, I have never needed to disassemble or fix programs built by others, never needed to understand kernel subsystems or anything else kernel related (except may be how to replace a broken driver/module). I know nothing of C, I know just the basics of system calls and I've never heard of skb or select() and epoll().
I don't even like developing software, which is the reason I once joined the sysadmin team at the company I worked for and fell in love with this job.
I see how these skills are a nice extra, but I understand that knowing too much also creates the problem of having to work harder and stay more often after hours. And saying no isn't an option because nobody else knows how to handle the problems that may arise.
To me, a good sysadmin knows when to say no and spends the least effort in fixing errors in code. The developers should provide good programs or it doesn't get installed in production. Keep strict boundaries in what you do and don't to prevent being abused by your employer. If you are often the last one to leave because you're doing work that someone else should be doing, or asked to be stand by after hours while being the only sysadmin, step on the brakes before this will eventually burn you out or makes you hate your job (or even hobby).
Also, knowing all of what parent describes + all the 'normal' sysadmin knowledge is (eventually) impossible to keep up and will also take too much of your free time. You will regret this later in life.