Last modified: Feb. 12, 2001

Simmons' Laws of System Administration

The Definition:

System Administration is the combination of system support and user support.

The Rule of Rules and Laws:

Any rule can be modified by the application of power and policy. By contrast, rules always are subordinate to laws.

Note this rule is a rule, not a law.

The Network Paradox:

System support is a subset of network support. Network support is a subset of system support.

The Laws Of Unanticipated Support Cost:

  • It will always cost you more to support a thing than the vendor told you.
  • It will usually cost you more to support a thing than to buy it.
  • Sometimes it costs 10x as much to support a thing as it did to buy it.
  • Refusing to support something often results in the thing being unusable.
  • Once it's installed, supporting a thing is sometimes cheaper than not supporting it.
  • Before buying, make sure you're committed to support. But see the first item in `The Laws Of Unanticipated Support Cost.'
  • The Division Between System Support and User Support:

    There's a difference between system support and user support. There may be overlap in the two positions; sometimes both are done by the same person. But the two tasks are distinct and sometimes have conflicting goals.

    The Law Of Distributed Talent:

    Great system support people often make lousy user support people and vice versa.

    The Paradox Of Dual Abilities:

    The person good enough to do both system support and user support will usually be hired away by a shop where the combined tasks are too large for a single person.

    On Complexity And Customization:

    Application-to-application differences confuse everyone, especially users and support staff. Ditto UNIX-to-UNIX differences, etc. By contrast, complete consistency completely stifles improvement.

    At any given site for any given application or feature, there's someone who knows more about it than the support staff. Finding that person is the first step to take to diagnose any given problem.

    Time to diagnose and time to fix are completely unrelated. Sometimes one approaches zero while the other approaches infinity. This is especially hard to deal with when the diagnostic person and the fix person are not the same.

    One person's improved feature is another person's gratuitous change.

    Users want applications and systems they can customize.

    One user's customization is another user's gratuitous change.

    The Laws Of The Cost Of Customization:

    The cost of customization is complexity. The cost of complexity is increased difficulty in administration and user support. The cost of increased difficulty in administration and user support is either lower quality of administration and user support, increased support staff, or both. Therefore increased customization means increased cost, or lower quality of support, or both.

    The Paradox Of Unused Customization:

    It doesn't matter whether customization has actually been done. The mere fact that it's possible means you must check for it, thereby increasing the cost of problem diagnosis.

    Smallwood's Law (Simmons' paraphrase):

    ``They're not users, they're clients.'' - Kevin Smallwood

    Users Are Not Customers:

    There is a lot of popular nonsense going around about how system administrators should treat the users of the systems as customers. This is simply wrong, for two key reasons:

    Users Are Human:

    The user who says ``Can X be done?'' is usually really asking ``Would someone please do X?'' Make sure you answer both questions.

    It's human to blame problems on outside causes. By contrast, an outsider will always suspect the insider as the cause.

    The user who says ``I didn't change anything'' isn't always lying. Sometimes they're just ignorant or forgetful.

    It's more important for users to do their job than to answer the needs of admins. Unless of course their job is to answer that need.

    The skills needed for dealing with a computer are not simply irrelevant to those needed for dealing with people; they are actually a negative. You cannot reboot, reinstall, or power cycle a human being no matter how desperately they need it.

    Admins Are Human:

    For every statement in ``Users Are Human,'' change ``user'' to ``admin'' and vice-versa.

    The `You Broke It' Principle:

    Cockpit error is the most common cause of problems.

  • Corollary: Everybody is a pilot.
  • Support Is Overhead:

    One way of cutting costs without cutting development staff is by cutting overhead. System administration and user support are overhead.

    User and system admin training are overhead. Not having them increases overhead. Go figure.

    The Joy Of Being A Contract System Administrator:

    ``Sure, we can do that. Here's what it'll cost you.''

    His Site Isn't Your Site:

    The situation at your site doesn't make you qualified to judge the situation at another site, and vice-versa.

    Just because someone else's support staff does it doesn't mean your staff can do it.

    The Rules of Policy and Power:

  • The first rule of policy and power is that at your site, system administration is whatever the boss tells the admins it is.
  • Users will bypass admins to get the boss to tell the admins something different. That's their right.
  • Most system admins live in a policy vacuum. This can be good or bad:
  • Corollary 1: Power expands to fill a vacuum. That thing which expands most easily is a gas.
  • Corollary 2: Anything that quickly expanded to fill a vacuum is easily displaced by a solid.
  • Corollary 3: A rapidly moving solid will hurt you if you're in its way.
  • The person who does your job review makes the rules. The good admins always follow those rules. See the first rule of `The Rules of Policy and Power' and the `Rule of Rules and Law.'
  • The summary on Policy and Power: Be careful what you do in that vacuum. Nobody appointed you God. However, you can always be dis-appointed.

    The Laws Of System And Network Growth:

    You can always incrementally add one more.

    Sometimes the straw breaks the camels back. More often, the camel just goes slower and slower.

    The difficulty of support does not grow linearly with the size of the site.

    Eventually your site outstrips your methods, and you must bite the bullet and move to new methods.

  • Corollary: Nobody bites the bullet until there's not enough time to do the existing work. At that point there's not enough time to make the changes.
  • Adding one single instance of a new kind of computer, operating system, application, peripheral, etc, has a much higher administrative cost than adding one more of what you've already got.

  • Corollary 1: If you buy one, you may as well buy ten.
  • Corollary 2: If you buy ten, you may as well buy eleven and keep one for spare parts.

  • Last modified: Feb. 12, 2001


    Back to Steve's home page.
    Contact, License and Copy Issues.