r/sysadmin • u/Fair-Wolf-9024 • 5h ago
Fundamental approach vs pragmatism
Hello, everyone!
Sysadmin with 9 months experience here(basically nothing). My question is kinda more theoretical/philosophical rather than technical one. So I do tasks and tickets, but at the same time I read books about Linux and recently I was learning on a deep level about TTY, PTY, history of terminals, what is line discipline and etc. And when I asked my senior colleague clarifying question about it, he said that he does not know, indeed he thinks that its pointless to learn this things because its highly unlikely that I am ever going to implement this knowledge at all. He stated that it is better to learn new stuff as you get new tasks instead of building up this knowledge because you will forget it anyway. He added these things like low level components and C and all this stuff that Unix systems are running on is already a legacy and we have to shift to a next abstraction layers like developers do not need to know how assembly runs code. So my question is do I really need to know all these things and even more so on a this profound level to become highly skilled sysadmin and infrastructure specialist or I need to form a more pragmatic thinking? For example, I know what is bios, uefi, lvm, disk partition, daemons and etc, but it's more on superficial/practical level even though I can answer interview questions about it.
I think that this philosophy (understanding Linux/Unix and its components on a profound deep level even though I probably will not use this knowledge on a day to day basis ) is going to make me a strong professional by building a whole picture but at the same time since I am new to this industry and do not know it is going to work in a long run.
Thank You everyone in advance!
UPD: Thank You, sysadmin comrades, for your advices! I really appreciate it!
1
u/michaelpaoli 2h ago
colleague
thinks that its pointless to learn this things
Take what they say with at least healthy dose of skepticism - it will have bias, as will all sources, and learn it what you can, but also learn from wiser more experienced folks/sources too.
learn new stuff
Also important.
will forget it
Somebody projecting?
Good to also well know fundamentals from the ground up, what you're reading covers fair bit of that.
Some things may be forgotten, but the more you read/use, more likely to remember, and even if (many) details are forgotten the overall information and/or meta-information learned may be well remembered and highly valuable.
So, "old" stuff, much of *nix design philosophy, history, the "why", how things did and didn't change and why, al generally rather to quite useful to know.
things like low level components and C and all this stuff that Unix systems are running on is already a legacy
No. Fundamental base, with much built atop that. So yeah, a good, or at least reasonable understanding of C is still quite important. Sometimes that's dang important to be able to dig into low level details and figure out exactly what the heck is going on. E.g. strace(8) - if one has no clue at all about C and system calls, might as well be trying to look at faded broken partial Sanskrit. Reasonably well know C and system calls, and it becomes highly useful information - many many occasions I've used to find/isolate/diagnose various issues/problems/bugs, etc. Don't know that stuff, and you're working with an arm (or at least several fingers) tied behind your back.
developers do not need to know how assembly runs code
Yes ... and no. They might often get away with it. But they often cause significant problems to major disasters by failing to reasonably well understand what they ought. E.g. here's something I ran across some years back in production:
$ date -Iseconds; ls -ond . 2019-08-13T01:26:50+0000 drwxrwxr-x 2 7000 1124761600 Aug 13 01:26 . $
Yeah, that, for an active directory of data files, quite constantly adding and dropping files ... that's a quite bad situation for almost all filesystem types on *nix. Not sure how that'll behave and why it's a problem and ought not do things like that? Set up a little test/learning exercise for yourself. Create a non-prod VM. Create such a scratch directory - but not at the root of any filessytem - most any local filesystem type will do, but do be sure the filesystem has more than ample free i-nodes available (or allocates dynamically). Can create such with a sufficiently fast loop in bit of code (e.g. python, perl, etc.) - but not shell, you want fast - zero external commands - certainly at least in body of loop and its test conditions - otherwise it'll be orders of magnitude too slow. May still take quite a while to create directory that large - and don't need much filesystem space ... files can be empty, and can also mostly use hard links to avoid consuming lots of inodes - mostly just consume directory space, not a lot else. Once you've done that, shave down the resources of your VM - don't need much there ... get it down to say 1GiB of RAM, maybe leave it 515MiB of swap ... or not but go for not very performant swap - like HDD, or at least SDD. And they try some operations in the directory ... try adding a file, try removing a file, try doing ls, see how that goes for you. Now (learn and) explain why that's a major problem. Look at, e.g. squid. How does it lay out the directory structure for it's cache storage? Look likewise at most types of filesystem storage (e.g. caches) on *nix - and you'll notice a quite similar pattern - yeah, there's darn good reason for that. So ... how 'bout explain why that's the case. So, what about shrinking directories - can it be done, or how would one effectively go about that to fix such issue with such a directory. What if it's the root directory on a filesystem? What about other filesystem types? E.g. how does this behave on tmpfs? Yes, Grasshopper, when you can snatch the pebble from my hand, it will be time for you to go. Alas, developers grossly unaware of such things, will not uncommonly write highly poor applications, that will create such highly problematic situations ... in production, because yeah they didn't know, didn't check, didn't research/investigate, didn't properly test, etc. "Oops". That's the kind of thing that happens when they don't know (among many other ways of screwing things up), and of course likewise sysadmins aren't at all immune from such (hey, we generally have our own ways of majorly screwing things up ;-)).
need to know
Need and should aren't the same. Want to do it highly well, be well respected and remembered? Or could be very unpopular, despised, hated, reviled, and get a very ugly naked statue of you put up in, e.g. EU Parliament for folks to gawk at. How do you want peers, folks impacted by your work, history, etc. to view your work?
understanding Linux/Unix and its components on a profound deep level
Well and deeply understanding how things work is knowledge/skill too many lack. And not uncommonly it makes a huge difference.
E.g. large major production server "down" - entirely unresponsive. I go look at the hardware. Standard operating procedure, we'd need crash it, try to reboot, it would've hung very badly, call vendor, vendor would come out, address the hardware issue, then we'd have to recover large crashed database to consistent state ... that would've been 12+ hours of outage. But I notice not only what the indicator LEDs are telling my "by the book"/diagnostics/manual, but the particularly odd manner in which they're doing so - not at all covered by any official documentation. Yeah, all seems to indicate something seriously f*cked up on SCSI bus and at best going very very very very slow - probably a whole lot of repeated retries cycling over and over again, and then I carefully visually inspect all those connections - they don't look so good - a lot of cables rather stressed/pulled/bent, under significant tension, and most notably at their connectors ... not good. So, I very carefully start taking the cable ends, and pushing them well square and flat, firmly to their connectors, and also working to adjust bends in them to relieve tension with connectors, and as I'm doing that, host springs fully back to life - continues from exactly the place it'd seized up, everything back in full operation, database good clean and continuing in operation, etc. - all fine for at least the moment - and "good enough" for remainder of production hours for that host - saved a whole lot of outage time compared to not knowing those things and having only gone strictly by official documentation and standard procedures. And of course then the immediate to everyone was don't touch it or even look at it funny during production hours, and most immediate opportunity after that, take maintenance window to being OS down, highly well go over all those cable connections, adjust them all as they ought be, also well disconnect and reconnect, and do bit of sliding on / partly back / fully on, repeatedly on each, to also cause the contacts to get well wiped while reattaching, and then make sure they're all well secured in place, and devoid of any inappropriate tensions, bends/kinks, etc. - then check, all fine ... and never had that same issue from that host ever again.
So, yeah, well knowing and understanding how things work, from bottom up, is good knowledge/skill to have. And not uncommonly it'll be the skill that saves the day. Server room majorly overheating because both air conditioning units failed ... emergency work-around and large fan in door with door propped open ... not great but, enough to mostly keep things operational ... but somebody didn't well understand the physics, mucked with the fan aim/positioning .... and the server room temperature climbed up about another 10F - hazardously/dangerously hot. I readjusted the fan - no changes beyond that ... and within ah hour, down about 10F. Likewise, knowing the air conditioning and nature of the failure - got 'em back in working order within another hour or so - no service calls needed (and wouldn't really help - they'd just tell us what I already knew anyway)
Updating critical configuration files ... replace, or true edit-in-place - yes, there's a difference, and that difference can be critical - notably race conditions - and some other tradeoffs. And not all that claims to be edit-in-place (e.g. GNU sed's -i, likewise perl, etc.) truly is. Hint: rename(2) is an atomic operation.
Anyway, tons of details to know all over the place, and, that "higher level stuff" - there's a whole lot underneath it that makes it happen - and often knowing at least relevant bits of that, can be quite important. E.g. "Cloud" - it doesn't run on some mythical water vapor - there's actual hardware beneath it. And when Cloud provider f*cks up (sometimes they do!) it's also often highly useful/important to be able to show exactly where and how they're f*cking up - often all their standard tools and APIs won't at all show any issues at all when they f*ck up. Can oft get 'em to realize and fix it way faster if one can point them to exactly where their sh*t is broken. And yes, did get a major cloud provider to see one of their bugs that way, fix the immediate issue, and also get around to fixing the underlying bug of theirs.
Anyway, keep learning. Always more to learn, and nobody knows everything.
1
u/Fair-Wolf-9024 1h ago
Thank You for your time and investment into answering my question! It helps me to look at my work from another angle! Appreciate it!
1
u/uptimefordays 5h ago
The value of fundamental knowledge is less day to day application and more understanding how things work and why.
1
u/SwedeLostInCanada 5h ago
Eh, it is a bit of both. He’s right that you’re very unlikely going to touch a lot of these things. You won’t be writing C code.
A lot of sysadmin comes from learning on the job and solving tech issues and running infrastructure. You’ll be hands-on for things that could warrant years of study to learn to low level.
Having said that, a lot of sysadmin wizardry comes from understanding enough low level concepts to piece together a puzzle. If you read some threads here about people randomly knowing it is a DNS issue, it comes from that. A lot of solutions just comes from “knowing how Linux/vmware/cisco switches/windows servers/AD/etc work”.
TLDR: balance both learning styles. Don’t let theory get in the way of practical work.
1
u/Potential_Point_6539 5h ago
That senior is giving you the standard "good enough" mindset, but the wizards in this field are the ones who actually understand what's happening under the hood
when something breaks in a weird way and google has nothing for you, that deep knowledge is what separates you from the guys just waiting for a ticket queue to fill up
don't burn yourself out on it though, learn what interests you and let the practical stuff fill in the gaps
1
u/LLMsMustUpvoteThis 4h ago
And this low level knowledge does come up if you want to move into more cybersecurity focused roles. In learning pentesting I've had to delve into TTY black magic to get stable and properly interactive shells. Or even more "real world" sysadmin like diagnosing Windows auth issues is really helped by having a low level understanding of how Kerberos works.
1
u/brainchillzZ 5h ago edited 5h ago
Knowing something deeper than most people is good but going so far into the weeds with things that are meaningless not so much so the truth is somewhere in between …. But here is the honest truth …. Don’t learn these things because it’s your job… experiment, find niches that really interest you and deep dive them then move on to the next one and the next one and one day you’ll wake up and realize that you’re the person all your colleagues and friends come to with questions about anything because you can create an architecture for the network, for the storage, for the complex systems and software stacks in the datacenter and then actually design and implement every piece of it yourself without needing anyone’s help to do it …. Then take it further and learn how to weld, learn how to grow plants, learn how to {insert new skill here}…. That’s what being a hacker really is …. It has less to do with computers and more to do with obsessively learning how to do everything you can as well or better than most people who specialize in one thing … and the more new things you learn the better your brain will get st absorbing the next thing….. and more importantly at making connections between all of those things while you’re not even thinking about them explicitly …. And as a side benefit when you’re constantly talking about the next new thing you’re obsessed with learning you’ll drive everyone around you crazy :)
1
u/Dave_A480 5h ago
You probably will never hook a modem up to a ttySx port & configure Getty in this day and age....
I have had to mess with this stuff for embedded projects and usb-to-serial console configuration....
Learn what you want to learn....
When it comes to Linux what you absolutely NEED to learn is ssh, bash (and all the little text processing utilities like awk, sed, head, tail, sort, uniq and so on), ansible & terraform....
1
u/michaelpaoli 2h ago
will never hook a modem up to a ttySx port & configure Getty in this day and age
Oh, you'd be surprised. Okay, sure, not so much the modem bit - for most intents and purposes, dial-up modems are a thing of the past. But the rest mostly quite still applies. Hooking up serial ports on much server equipment and the like, still quite commonly applies. Likewise getty on serial ports - be they physical or virtual. E.g. VMs, very commonly setting up (virtual) serial on those and getty - but hey, virtual, at least don't have to deal with the cables and all the pinouts and adapters and such - but that's still quite relevant for the physical.
And hasn't been all that long since I'd repeatedly be saving the day, where the folks in charge of it couldn't manage to get a working terminal server connection between terminal server and the server host (and alas, I didn't have the access to go in there and straighten that sh*t out myself), but when nobody else could manage (because "of course" they'd already long gotten rid of the actual terminals - because who need that when you have terminal servers?), I'd be the one, walk in, laptop, Linux, serial - connect to serial on the host, and be able to save the day - because despite it being a *nix sysadmin team of ~20 folks, nobody else on the team actually knew how to do that - generally they didn't know sh*t about serial and left all that to other persons/teams to deal with - they just relied upon being able to make secure connection to terminal server, and then take it from that point. Most of 'em never even touched hardware in server rooms or data centers, or if they did at all, generally not beyond keyboard on some console type device or the like (e.g. KVM).
1
u/Steerable-Octopus 4h ago
Nah the industry is quite open regardless of what you want to focus on. Deep understanding or seeking abstractions are two equally valid approaches and in my opinion I think the former is more respectable even if the latter has higher earning potential.
1
u/SFJulie 3h ago
Unix philosophy is embedded in POSIX norms.
Which includes a lot of scripting usage for cross platform development (POSIX shell, grep, awk ...)
Modern Linux are straying from the POSIX path (ex: systemd) still it is a good ground for troubleshooting and writing portable software (with windows git you can even do POSIX scripting on Windows)
1
u/Ahren_with_an_h 3h ago
Red Hat is alive, well, and growing, I assure you.
You are allowed to read and learn purely because you find something interesting. You never know where a random tangent will lead you. Either way, better to stay engaged and interested than to try optimizing for productivity at the expense of the fun. You don't stick to things that aren't fun on some level.