Note: This document is copyright © Steve Jenkin 1998. It may not be reproduced, modified or distributed in any way without the explicit permission of the author. [Which you can expect to be given.]





Lessons from the Worlds' Greatest Sys Admin - July 1998


Presented at SAGE-AU Conference, July 1998



Contents

Contents
Introduction
Background
Principles of System Admin
Some WGSA Attributes
About The WGSA
Sayings of the WGSA.
Some Sound Management Laws
So What?
How do you work with a "World's Greatest ..."
Some "Good Stuff" I learnt from friends.
Some of the WGSA's work
Summary



Introduction

[Contents] [Next]

Why


(2005) The most frequent comment I received on the 'WGSA' talk was: "So you think you are the World's Best Sys Admin?"

Answer: No, I am not the "WGSA".

This paper is about someone, who is really an amalgamn of a number of people, who regarded themselves as "The World's Best Sys Admin". They never verbalised this opinion - they just lived it.
Unfortunately, as is the case with all self-appointed 'guru's I've met, they had limited raw talent and an arrogance that prevented them admitting less-than-perfect performance, taking on-board any useful criticism or correction or learning new tools, techniques, processes, and organisations from others they didn't consider an authority.

I apologise in advance to the reader that the paper is mostly about "negative" learning, or What Not To Do...
I included a section on Good Things I've Learned to show that I wasn't totally preoccupied with the negative :-) But there were just too many good stories, and I really had to let steam off over this...

Do you think you know the identity of "The WGSA"?
You don't. For those that may even think it's them - No, it's is not you. The observations and opinions here came over a considerable period of time. It's not a single person - and it's not just Administrators. I've met "WG Programmer", Architect, Designer, Tester, Integrator, Networker, Technical Manager and CIO.

So - onto the main game - the paper as presented to SAGE-AU in Old Parliment House.
If you were there - did you catch any of the lollies I threw, or even an egg?

Feedback is something I'm interested in.
Drop me a line if you have your own stories, can add more useful models, or if you relate to this and found it useful. Suggestions for improvement gratefully accepted & acknowledged.


WHY? To codify and inform.
Once a problem is recognised and named, you can start to understand and address it.

Audience

Junior Sys Admins
- If you work for one.
Senior Sys Admins
- If you work with one.
Managers
- If you have one work for you.


Format



The BIG Questions






Background

[Back] [Contents] [Next]

I spent a year in 96/97 contracting in Sydney for what should've been a large, prosperous Australian multinational. They hadn't paid a dividend since 1990 and were taken over by a Dutch company at the end of 1996.

The I.T. group was appalling. Staff turnover in the Unix Support and Networking areas was high - close to 100% in 12-18 months! The company spent under 1% of turnover on I.T. - against the industry average of 5+%. It felt like we were doing the impossible - and we were.

They'd outsourced their mainframe, embraced 'open systems', installed a large scale WAN, gone client-server, were developing GUI and O-O applications and had an Internet presence. They'd also radically downsized in two steps:- from 200+ to 30-40 staff in 18 months.

They were fully Buzzword Compliant, but were going nowhere.

I was privileged to meet two people - both ex-telecoms technicians who had moved into computing. One, the self-proclaimed WGSA, had been responsible for setting up the Unix environment, and it's associated X.25 network, and been the Unix support manager for a couple of years, until finally taking a job in 'Technology Planning' - but just doing more of the same.

The other ex-tech I've remained good friends with. He moved into Networking after a career in Civil Aviation, then a TAFE. He had enough PC, Unix, and Internet knowledge 'to be dangerous'. He'd left behind at the TAFE an environment where just 2 of them supported and ran the whole state TAFE network, +1 for Unix, 1 for printers and passwords, and 2 on the HelpDesk. When the lot was outsourced and a crack systems company took over - they boast they can cut 10%-20% from any operation - they ended up having to spend more.

The contrast was stark and savage - one had left behind a legacy of chaos and disorder, the other was undoing the damage and providing real business productivity.

This talk is about that experience and what I've learned.


Principles of System Admin

[Back] [Contents] [Next]

These are my values and principles. Your mileage may vary.




Some WGSA Attributes

[Back] [Contents] [Next]




About The WGSA

[Back] [Contents] [Next]

Of course he was the best. He had read every single 'white paper' from the vendor, and with his photographic memory, could recite it all back. All he needed to know was in those papers, and the manuals he'd read.

He didn't need to meet and talk with his peers, he had none anyway! He had no need of professional organisations or finding out what had worked, or not, for other people.

If he didn't have time to do something himself, he would get in a contractor, create a project, or hire a consultant. Funnily, these people were always only of very modest ability. The projects mostly ran out of money in "phase 1", when only the basic work was being done and well before the real benefits were to accrue.

He'd written 25,000 lines of shell script to provide a "common" menuing and execution environment. It was a most flexible, adaptable, and configurable environment - and surprisingly similar to that run by his previous employer. Just the thing to control 12 machines... It was a real engineering triumph - for 1982! He'd built and deployed all this with no version control, configuration management, or documented release and maintenance procedures - and certainly no review.

His crowning glory, "Xferutility", 7,000 lines in a single script, heavily utilised 'comes from' control files [they just appeared places, with no trace of whence they came], and could use 'rcp', 'ftp', and e-mail to achieve the functionality of uucp. Plus, it was the transfer mechanism, the interactive menu, the scheduler, and the status reporter. All things to all people bar those left to maintain it.

Having not apparently done "Programming 1A", he'd not been introduced to the concepts of "coupling and cohesion" - put together everything that belongs together, separate unrelated concerns - and least necessary complexity.

To go from the login prompt to the first displayed menu, over a dozen files or scripts were executed - often in perverse order. The system drive defaults would overwrite the local definitions!

He also seemed unaware of basic capacity planning issues - like tracking the number of systems in the machine room and providing adequate rack space and cabling. Backups were another story entirely.

The I.T. department policy was to have separate small systems for every division, no two the same. In 12 months it went from 12 systems in the machine room, to 23. And then to 35+ in the next 6 months.

Having labelled me "a cannonball contractor who won't be around in the long term", he resigned the week he penned it, took an overseas holiday [run in the same flexible fashion], and rejoined his previous employer, through a services company, performing Network Management.




Sayings of the WGSA.

[Back] [Contents] [Next]

A few of these are paraphrases.

What I find myself saying often is :-
Why would you want it any other way?, and
(2005)Would you expect any less?.

The answers to these questions are usally: Yes, any other way, and "NO!".

Sayings and tactics of WGSA and friends:-

Remember, there are rules for him and another set for you. He will ignore e-mail, talk about you behind your back, set impossible deadlines [for you], and not keep his promises. Don't expect to be told about important stuff that impacts you, or that you happen to be expert in. You won't get invited to meetings, see reports, or be involved in the 'discussions' held before major decisions are announced. Rumour, disinformation, and 'Need to Know' are powerful tools for the WGSA.

He will casually drop bombshells, regularly spring 'surprises' on you, and practices 'Divide and Conquer' extremely well. He allocates work, but will never help or clarify what he wants. And of course, won't follow up on it. He may fly into a 'justifiable rage' if he comes back in a month and something hasn't been done to his satisfaction... It's not easy being so perfect and all-knowing all the time.

Rational argument won't work with the WGSA. What matters is that he thought it up, he's important, and the bosses [his mates], think he is an absolute Guru on everything.

And if you ever get close to criticising him or winning an argument - slander and libel work just fine for him.


Some Sound Management Laws

[Back] [Contents] [Next]

(2005) Note:I don't try to come up with any principles or Laws for The "WGSA" follows. There is probably only one: "Seize Ever Opportunity". Which isn't a bad dictum, if it respects other people, fulfills your business's needs and goals and isn't only about advancing your personal agenda.

My version of "Sound Management Laws" are presented for you to consider and understand where I am "Coming From":-

In I.T. there are special management considerations:-

Good working relationships between management and staff take time and effort to develop. They proceed through the following stages and are fragile. The whole lot, years of work, can be destroyed in an instant with a lie.
What management want are people they trust, work very hard, and consistently produce quality work. People who hold the company's best interests to heart.

Development Stages:-
    - Honesty, Integrity, Openness, Frankness, Consistency
    - TRUST
    - RESPECT
    - LOYALTY
    - COMMITMENT, CARING




So What?

[Back] [Contents] [Next]

Since the advent of the 80486 in ~91, cheap LAN's in ~94, and the Net in ~96, I.T. systems and infrastructure have become essential and critical for all business operations. Systems Administration, Networking, Help Desk, and Database Admin are the glue that holds it all together from day to day.

There is a myth that software doesn't wear out like machinery. The bits don't change, so it must be OK! By implication, you don't need to "maintain" systems and software, like you do machines.

So why aren't we all running 286's and DOS 3.3?

It's called 'bit rot'. The software doesn't change, but the environment does - which gives the same net effect. Year 2000 isn't a problem until your clock says 01/01/00.

My argument is that company profitability is related directly to, ignoring management and leadership issues, staff efficiency [$ cost / $ sales] and new product evolution. These are driven directly by I.T. capability, which requires systems be constantly upgraded and enhanced - just to stay where you are! Similarly, I.T. operations staff must be continually increasing their own efficiency just to keep up.

(2005) See the 2003 Harvard Business Review article "I.T. Doesn't Matter" by Nicholas G. Carr. His website: http://www.nicholasgcarr.com/articles/matter.html

Effective Systems Administration is the single greatest point of leverage in the I.T. infrastructure - which is itself the single greatest point of leverage in an organisation. It amplifies and extends the thinking, analysis, and decision making ability of the people in the organisation. Even sometimes the managers. It can even provide some corporate memory - a prerequisite for Knowledge.

It's obvious the software in airplanes, spaceships, nuclear reactors, medical instruments, weapons systems, banks, and ATM's has to be correct, robust, and dependable or there are disastrous, often immediate, consequences. People die or billions goes missing. [Roll on NT - reactor control!]

What's not obvious is the long, lingering decline and demise of businesses - large and small.

The cost to Australia of losing a multi-billion dollar multinational company is incalculable. Well managed and well lead, it could still be a potent force on the global stage. Instead we have lost profits, destroyed assets, and put a few thousand people out of work.

(2005) On May 28 2001, Australia's fourth largest telco, One-Tel, ceased trading on the ASX. The Packer and Murdoch families, who control the media conglomerates PBL and News Corporation, lost about A$1Billion in the debacle. A major factor in the failure was uncollected "receivables". The computer billing system was faulty.
One.Tel closely followed the failure of HIH Insurance and Impulse Airlines.

That's a disaster 10 times bigger than TWA-800 going down outside New York just after take-off in 97, and they are still fishing out pieces. Just because it is in glorious slow motion - taking a decade, not a minute to unfold - doesn't mean we shouldn't still be as concerned with businesses going down as with aircraft crashes. People lives are destroyed and assets lost just as thoroughly in both types of crashes.

The government and professional bodies should be just as concerned with these outcomes and ensuring they can never happen again.




How do you work with a "World's Greatest ..."

[Back] [Contents] [Next]

I don't have an answer.

My style has been described as "Straight Up the Middle, with lots of smoke and noise."

My only response is to recognise an intractable situation early and leave as quickly as you can. A luxury I can afford, having no dependants and a low level of debt.

I Need an answer and would like - Your Feedback.



Some "Good Stuff" I learnt from friends.

[Back] [Contents] [Next]




Some of the WGSA's work

[Back] [Contents] [Next]

Here is a [long] list of some of the wonderful technical and process problems I came across. Remember this was a largish, not huge, enterprise. There were only 75 Unix hosts, a thousand or so users [total], and a network that went to less than 100 sites. Many of the systems were front-ends to the mainframe or a production system for the business. The Unix support team was mostly 3 people, sometimes with a manager, sometimes with people doing performance analysis/reporting, or 'implementations' - such as HP Openview [I.T. Operations].

The watchword for the I.T. branch was 'CHEAP'.


Summary

[Back] [Contents] [Start]

There are some people out there that don't just think, but know, they are the best. They are dangerous. Left unchecked they will not only make life a misery for everyone around, they help bring companies, even very large ones, down.

What singles them out is their inability to take input from others.

Typical behaviours are:-

If they are well settled and well regarded, the organisation is dysfunctional. It will be soul destroying staying.

And the only defense I know against them, once entrenched, is to leave.

And thank you all for your patience. I hope you have taken something away from all this...

Questions and Comments, please.




[Top]

Page Last Updated: Wed Feb 1 19:17:47 EST 2006
02-Jul-98
URL: http://www.tip.net.au/~sjenkin/sage/wgsa.html