System Administration is the combination of system support and user support.
Any rule can be modified by the application of power and policy. By contrast, rules always are subordinate to laws.
Note this rule is a rule, not a law.
System support is a subset of network support. Network support is a subset of system support.
There’s a difference between system support and user support. There may be overlap in the two positions; sometimes both are done by the same person. But the two tasks are distinct and sometimes have conflicting goals.
Great system support people often make lousy user support people and vice-versa.
The person good enough to do both system support and user support will usually be hired away by a shop where the combined tasks are too large for a single person.
Application-to-application differences confuse everyone, especially users and support staff. Ditto UNIX-to-UNIX, Windows-to-Windows, differences, etc. By contrast, complete consistency completely stifles improvement.
At any given site for any given application or feature, there’s someone who knows more about it than the support staff. Finding that person is the first step to take to diagnose any given problem.
Time to diagnose and time to fix are completely unrelated. Sometimes one approaches zero while the other approaches infinity. This is especially hard to deal with when the diagnostic person and the fix person are not the same.
One person’s improved feature is another person’s gratuitous change.
Users want applications and systems they can customize.
One user’s customization is another user’s gratuitous change.
The cost of customization is complexity. The cost of complexity is increased difficulty in administration and user support. The cost of increased difficulty in administration and user support is either lower quality of administration and user support, increased support staff, or both. Therefore increased customization means increased cost, or lower quality of support, or both.
It doesn’t matter whether customization has actually been done. The mere fact that it’s possible means you must check for it, thereby increasing the cost of problem diagnosis.
“They’re not users, they’re clients.” - Kevin Smallwood
There is a lot of popular nonsense going around about how system administrators should treat the users of the systems as customers. This is simply wrong, for two key reasons:
The user who says “Can X be done?” is usually really asking “Would someone please do X?” Make sure you answer both questions.
It’s human to blame problems on outside causes. By contrast, an outsider will always suspect the insider as the cause.
The user who says “I didn’t change anything” isn’t always lying. Sometimes they’re just ignorant or forgetful.
It’s more important for users to do their job than to answer the needs of admins. Unless of course their job is to answer that need.
The skills needed for dealing with a computer are not simply irrelevant to those needed for dealing with people; they are actually a negative. You cannot reboot, reinstall, or power cycle a human being no matter how desperately they need it.
For every statement in “Users Are Human,” change “user” to “admin” and vice-versa.
Cockpit error is the most common cause of problems.
One way of cutting costs without cutting development staff is by cutting overhead. System administration and user support are overhead.
User and system admin training are overhead. Not having them increases overhead. Go figure.
“Sure, we can do that. Here’s what it’ll cost you.”
The situation at your site doesn’t make you qualified to judge the situation at another site, and vice-versa.
Just because someone else’s support staff does it doesn’t mean your staff can do it.
The first rule of policy and power is that at your site, system administration is whatever the boss tells the admins it is.
Users will bypass admins to get the boss to tell the admins something different. That’s their right.
Historically, most system admins lived in a policy vacuum. This could be good or bad:
Now most system administrators exist in a policy fog.
The person who does your job review makes the rules. The good admins always follow those rules. See the first rule of ‘The Rules of Policy and Power’ and the ‘Rule of Rules and Law.’
The summary on Policy and Power: Be careful what you do in that vacuum. Nobody appointed you God. However, you can always be dis-appointed.
Contrary to your opinion, senior management is not stupid.
You send information and recommendations about changes up the line. What comes back seems to bear no relation to the problem, and none of the things you sent seem to be in the response.
Senior management may seem to make arbitrary or uninformed decisions, but this reflects both some ignorance on your part and some things about what happens to data as it moves up and down the line.
You may have made a great case why there's a three-year payback, but if cash flow is bad, it doesn't matter: they're going to choose the cheaper solution. Their priorities are not your priorities.
And they don't owe you an explanation.
Technical skills decline as you move up the managerial chain, and time to consider an issue declines. Thus at every step of the way, intermediates simplify and summarize. Your details will inevitably be lost, and their importance to the decision made will vanish.
Just as things get summarized on the way up, details get fleshed out on the way down. In a good organization, those details will usually put back the things that got lost on the way up. In a bad one, they may make it up as they go.
And politics influence things in both direction. C’est le guerre.
You can always incrementally add one more.
Sometimes the straw breaks the camels back. More often, the camel just goes slower and slower.
The difficulty of support does not grow linearly with the size of the site.
Eventually your site outstrips your methods, and you must bite the bullet and move to new methods.
Adding one single instance of a new kind of computer, operating system, application, peripheral, etc, has a much higher administrative cost than adding one more of what you’ve already got.