Tuesday, December 29, 2009

A Well Tempered Sysadmin

There is no single path to becoming a good sysadmin, this process takes time, talent, and dedication. There is no easy place to begin this discussion, and there's an amazing amount of controversy around this subject. So let me begin with what I know about myself. I will tell you two different stories of my life; I promise to lie only once.

The First Story:

I've been a professional sysadmin for almost 7 years. In that time, I have worked in many different countries, working alone or with teams of other sysadmins. I have built global networks, been personally responsible for millions of dollars worth of equipment, and debugged mission critical infrastructure. I also teach night classes at a local University. I've done all this without a High School Diploma.

The Second Story:

I have attended 4+ years of college learning about Information Systems and Computer Security. Last week I was awarded a dual Masters degree in Information Assurance and IT Management. I have two certifications, and will be studying for more now that I've reached a major milestone in my formal education. I cannot imagine my life without the benefit of education and the wealth of knowledge that it has brought me.

My lie is one of omission. Each story is true, but each story only represents part of the truth. My life, in fact, is a combination of both.

The Truth:

I've been a professional sysadmin for almost 7 years. In that time, I have worked in many different countries, received a few certifications , and attended 4+ years of college. I have built global networks, debugged mission critical infrastructure, and managed millions in equipment, all while finishing a dual Masters degree. I also currently teach night classes at a local University. I've done all of this without a High School Diploma, but I cannot imagine my life without the benefit of a formal education and the wealth of knowledge that it has brought me.

What did I learn in the past 7 years? Education and experience must work together, or they will only be incomplete halves of a story.

Mine is not an example to follow, working full time and attending college full time is mentally and physically challenging, and not for the faint of heart. This works for me because I have a passion for my work (and a very supportive wife). But I would try to leave you with a little practical advice:

  1. Find something that you are passionate about (networks, operating systems, enterprise architecture, whatever). If you're like me, and like a little of everything, find something that is convenient or important to your current work.

  2. Find a certification track or school that offers what you want to learn. Then jump in, rip apart the coursework, and make it your own.

  3. Find a mentor that can help you when you are stuck or motivate you when you are slacking. I've had many different mentors (some didn't even know the roll they played), each one brought something different to my life, and I value all of their contributions.

  4. Once you have reached an understanding and an internal level of mastery, you must complete the training. You must study for the test, or complete your degree. The time you spent following your passion can now be converted into better leverage during the hiring process. Here is the true alchemy to a happy life: turning passion into profit.

Education is a racket, you will not learn something through the mere awarding of a degree or certification. You only get what you take from your education. Treat each class like an outline and your coursework like a single bullet in the outline. Don't waste your time; attack your education with passion and determination. Build upon success.

Experience is overrated, knowledge of a sysadmin is hard won and quickly outdated. You must search desperately for the underlying truth in all your actions. The experience you gain must be fundamental, and must be applicable to many situations. Failures are costly, do not waste them.

Education stands on the shoulders of success, building over time. Experience flows from failure, sharp reminders that help us predict the future. A well tempered sysadmin will seek both success and failure with equal delight. Success expands our horizons and creates new frontiers to conquer. Failure keeps our head in the game and builds an understanding of the future.

It is within this combination of education and experience, of success and failure, that we may find true understanding and fulfillment. Finding work that fulfills our needs and pays the bills, is the beginning of a life well lived.

A New Year is upon us, and I personally want you to be passionate about your work. Pursue your passion, and turn your experience into education. If you don't know where to begin, start with the links below. Good luck, and have a passionate New Year!

Further Reading:

500 Word Summary of Dewey’s “Experience & Education”

The World Needs You to Do What You Love

Are There Blue Collar and White Collar Admins?

SAGE Sysadmin Core Job Descriptions

108 Things a Sysadmin Might Do

How To Become a Hacker

Learning From Failure

Wednesday, December 23, 2009

How To Become A Hacker for the New Year

I'm getting a jump on my New Year Resolutions by reading How to Become a Hacker by Eric Raymond. From this beautiful essay comes a list of things every hacker (or sysadmin) should know how to do well:

  1. Learn to write your native language well. Though it's a common stereotype that programmers can't write, a surprising number of hackers (including all the most accomplished ones I know of) are very able writers.
  1. Read science fiction. Go to science fiction conventions (a good way to meet hackers and proto-hackers).
  1. Train in a martial-arts form. The kind of mental discipline required for martial arts seems to be similar in important ways to what hackers do. The most popular forms among hackers are definitely Asian empty-hand arts such as Tae Kwon Do, various forms of Karate, Kung Fu, Aikido, or Ju Jitsu. Western fencing and Asian sword arts also have visible followings. In places where it's legal, pistol shooting has been rising in popularity since the late 1990s. The most hackerly martial arts are those which emphasize mental discipline, relaxed awareness, and control, rather than raw strength, athleticism, or physical toughness.
  1. Study an actual meditation discipline. The perennial favorite among hackers is Zen (importantly, it is possible to benefit from Zen without acquiring a religion or discarding one you already have). Other styles may work as well, but be careful to choose one that doesn't require you to believe crazy things.
  1. Develop an analytical ear for music. Learn to appreciate peculiar kinds of music. Learn to play some musical instrument well, or how to sing.
  1. Develop your appreciation of puns and wordplay.

Eric's vision of building a better hacker (or sysadmin) is an excellent way to start your list of New Year Resolutions. Personally I would love to see more hacker/warrior monks on the street in 2010.

Wednesday, November 18, 2009


I switched my RSS feed to feedburner, http://feeds.feedburner.com/semafour/blog. Enjoy! Please resubscribe, if I decide to change blog platforms, there will be no need for you to change RSS feeds again.

Monday, November 16, 2009

Sysadmins And The Turbulent Waters of PEBKAC

Cory Doctrow's recent story Epoch (commissioned by Mark Shuttleworth), has a brilliant passage about sysadmins:

I will tell you a secret of the sysadmin trade: PEBKAC. Problem Exists Between Keyboard and Chair. Every technical problem is the result of a human being mis-predicting what another human being will do.

Surprised? You shouldn't be. Think of how many bad love affairs, wars, con jobs, traffic wrecks, and bar fights are the result of mis-predicting what another human being is likely to do. We humans are supremely confident the we know how others will react. We are supremely, tragically, wrong about this. We don't even know how we will react.

Sysadmins live in the turbulent waters of PEBKAC. Programmers think that PEBKAC is just civilians, just users. Sysadmins know better; sysadmins know that programmers are as much of the problem between chair and keyboard as any user is.

They write the code that gets users into so much trouble.

I've met more than a few sysadmins who don't like dealing with people. This point of view is a tragic mistake. People design these systems, people give value to information they hold, and people create the need for sysadmins in the first place.

Sysadmins above all: manage and troubleshoot the relationship between people and services.

Cory's done an excellent job distilling the many facets of sysadmin work while still making it accessible to the average person (ie. non-sysadmins). Epoch is Cory's second story about sysadmins, his first was When Sysadmins Ruled The Earth.

Saturday, November 14, 2009

108 Things a Systems Administrator Might Do

When I meet new people, and they ask me what I do for a living, I usually just respond with "computer stuff". This is about as far as most non-technical people want to take it. But its always bothered me that, as a SysAdmin, I have no elevator pitch.

An elevator pitch is a 30 second description of a service or a product that should ignite the interest of the audience. Being able to describe yourself or your work in an exciting manner for a general audience is an important rhetorical skill. Ultimately it may save your job or help you get a new one.

I recently found a list of 108 Tasks that a Systems Administrator Might Do, it appears to be from a SAGE article or document entitled: Analysis of the System Administrator Occupation. I dumped the entire list into wordle and created a weighted list. I was hoping that this visualization would help me build a narrative about System Administration, and help me create an elevator pitch.

While I'm still working on my elevator pitch. I thought both of these lists were too useful to keep to myself any longer than necessary. If you have your own SysAdmin elevator pitch, or would like to add anything to this list, leave a comment.

108 Things a Systems Administrator Might Do.

Hardware Installation and Maintenance

  1. Install/configure mother boards and memory cards/chips into systems (e.g., NICs, CPU cards, I/O cards).
  2. Modify operating system to recognize new hardware.
  3. Install and maintain cabling and device hardware (e.g., peripheral cabling, power cabling).
  4. Debug cable problems to resolve issues of connectivity (e.g., breakout box, protocol analyzer).
  5. Assemble components into working systems (e.g., plug components together, replace controller).
  6. Fix/repair computer system to the field replaceable unit level (e.g., disk failure, network or memory card failure).
  7. Dispose of old equipment and sensitive material (e.g., completely erase disk) factoring in relevant security and environmental considerations.

    Peripheral and Device Management

  8. Install/configure peripherals and devices (e.g., jukebox, modems, printers)
  9. Configure device drivers and ports (e.g., serial ports).
  10. Control access to network resources (e.g., printers, modems).
  11. Maintain and configure local and remote printing capabilities.
  12. Fix/repair printing function failures and problems (e.g., queues, spooling).
  13. Manage dial-up modem banks to maintain incoming/outgoing remote access capabilities.

    Data Integrity Management

  14. Devise system administration scheme and plans to mitigate common system failures, disasters, or emergencies (e.g., file corruption, hardware failures, power surges, fire, theft).
  15. Prepare/maintain backup media tracking system (e.g., tapes, CD ROM, floppy disks).
  16. Backup necessary system files on appropriate device/media (e.g., magnetic tape, disks).
  17. Restore files and system from backup device/media.
  18. Reinstall/repair operating system (e.g., corrupt kernel image, volume header).
  19. Maintain/reinitialize or repair disk drives.
  20. Verify/ensure integrity of backups.

    Data Storage Management

  21. Prepare disk and layout for data (e.g., RAID management, format/label/partition disks).
  22. Connect and/or configure new storage devices.
  23. Monitor, verify, and correct file systems (e.g., fsck, Checkdisk, Scandisk).
  24. Create, modify, and organize directory structures.
  25. Monitor, set, and change file permissions to control user access.
  26. Monitor and correct corrupted files
  27. Monitor file system usage (e.g., disk space remaining, disk usage over time).
  28. Reevaluate/redesign file systems layout (e.g., add/shrink/enlarge file systems).

    Network Configuration and Management

  29. Coordinate network topology and design with network administrators (e.g., new installation, upgrade).
  30. Plan, obtain, assign, and manage Internet names (e.g., DNS, domain name registration).
  31. Plan, obtain, assign, and manage Internet addresses (e.g., DHCP, AS numbers, OSPF areas).
  32. Configure and manage network file/data synchronization and/or distribution (e.g., rdist, SMS).
  33. Configure and manage network time sychronization in servers (e.g., ntpd).
  34. Configure and manage network file systems and servers (e.g., NFS, RFS, AFS, SAMBA).
  35. Monitor connectivity to detect network faults and measure network performance (e.g., ping, traceroute).
  36. Troubleshoot and correct network failures (e.g., cables, hubs, routing).
  37. Configure network interfaces (e.g., netmask, broadcast, speed, mode, ppp modem).

    Internet Services and Electronic Mail Systems

  38. Configure mail systems (e.g., MTA, anti-spam).
  39. Create, configure, and manage mail aliases and distribution lists.
  40. Install, configure, and manage mail reading applications (e.g., Eudora, Elm, Pine).
  41. Manage the web server and server-related programs (e.g., Apache, IIS).
  42. Install and configure non-web host services (e.g., FTP, archives).
  43. Install, configure, and manage network news, bulletin board, and chat services.

    Software System Development, Configuration, and Management

  44. Locate/download software packages and patches from the Internet or computer vendors.
  45. Build, install, and configure operating systems (e.g., NT, Linux).
  46. Install upgrades and operating system patches and service packs.
  47. Build, install, and configure application software and tools (e.g., third-party, public domain, or shareware).
  48. Debug application software problems (e.g., business-specific software such as Adobe software such as Adobe Acrobat or Netscape).
  49. Port system utilities to other operating system environments (e.g., convert script from Perl4 to Perl5, convert script from Unix to NT).
  50. Resolve compatibility and inter-operability issues (i.e., resolving machine-to-machine problems).
  51. Audit/evaluate existing source code for problems (e.g., for buffer overflows, Y2K related issues).

    User Support and Help Desk

  52. Configure/create templates for user interfaces and user environment (e.g. CDE, browser, windows, log in scripts, shell rc files).
  53. Identify and translate potential or actual user needs into technical requirements.
  54. Verify, remove, and disable user accounts (e.g., logins, passwords, shells, account validation).
  55. Manage user privileges (e.g., security levels in groups, file server access).
  56. Train and orient new and existing users.
  57. Respond to user requests, trouble reports, and questions.
  58. Triage and dispatch user requests to appropriate personnel.
  59. Communicate system status (e.g., planned outages, cause of network crashes) to users.
  60. Write local environment documentation to support users (e.g., FAQ).


  61. Evaluate potential problems, liabilities, and costs of potential or actual security attacks (i.e., risk analysis).
  62. Identify/evaluate/implement security mechanisms and tools (e.g., IDS, tripwire utilities, intrusion prevention software, firewalls, TCP wrappers).
  63. Formulate security procedures to prevent, detect, and respond to internal and external security threats (e.g., passwords).
  64. Evaluate and create site security plans.
  65. Monitor and detect security threats, holes, and attacks (e.g., viruses, detecting users with no passwords, unlocked administrative systems).
  66. Analyze internal/external security attacks (e.g., scan system logs for incidents, analyze network packets, implement intrusion detection software).
  67. Deploy and manage authentication systems (e.g., tokens, one-time passwords, Kerberos, NIS).
  68. Manage cryptographic facilities to protect sensitive information in network applications (e.g., PGP encryption in electronic mail).
  69. Respond, resolve, and report security incidents (e.g., unauthorized access to system).
  70. Monitor emerging security threats/tools/issues (e.g., via security news groups, CERT).
  71. Perform periodic security audits to ensure security has not been breached or compromised.

    System Resource Management and Performance Tuning

  72. Create/specify service-level agreements for site primary services.
  73. Debug and/or optimize network performance and performance issues.
  74. Manage system resources (e.g., monitor user disk and print quotas, CPU usage, swap usage).
  75. Evaluate and optimize system resources (e.g., organize disk space and memory).
  76. Manage system processes (e.g., signaling, changing priorities).
  77. Modify operating system configuration (e.g., add or modify services, configure/rebuild kernel).
  78. Perform housekeeping and clean-up activities (e.g., remove files, log rotation, archive, delete old users).
  79. Develop or enhance software tools to automate tasks (e.g., write scripts).
  80. Plan and build high-availability systems for critical services (e.g., business critical environments such as banking, real-time systems

    Technical Record Keeping and Procedural Documentation

  81. Develop/maintain operational instructions and procedures (e.g., How Tos, runtime procedures, runbook).
  82. Develop/maintain records and technical documentation (e.g., software version numbers, user logins, system architecture, licenses, descriptions).
  83. Develop/maintain daily operation logs to track problems and to establish an audit trail to debug and isolate potential problems (e.g., track mean time between failures and uptimes).
  84. Audit and inventory user licenses to ensure legal compliance.
  85. Maintain data in work request and tracking systems (e.g., Remedy, clarify, Tkrep, MHQ).

    Procurement and Vendor Relations

  86. Evaluate needs and develop system design and upgrade proposals/justification.
  87. Research and evaluate hardware/software/equipment to satisfy requirements (e.g., user needs, budgetary, legal, technical specifications).
  88. Write software/hardware specifications to meet user needs (e.g., RFI, RFP).
  89. Evaluate and recommend third- party products and services.
  90. Develop/write purchase justification (e.g., based on growth and needs).
  91. Negotiate/renegotiate service- level agreements and terms with provider to optimize costs and/or services (e.g., technical support, equipment, maintenance).
  92. Establish and cultivate relationship with vendor for problem resolution, technical support, etc.
  93. Monitor vendor contract performance (e.g., track vendor response time).
  94. Place, manage, and track equipment orders.
  95. Establish/update equipment inventory.
  96. Provide/solicit information to/from vendor to fix software_ to/from vendor to fix software bugs and problems.

    Technical Management

  97. Train system administration staff.
  98. Supervise and manage technical staff.
  99. Anticipate and plan computer system resources for future needs (i.e., system capacity planning).
  100. Anticipate and plan network resources for future needs (e.g., bandwidth, redundancy).
  101. Anticipate and plan human resources for future technical needs (i.e., hiring and staffing).
  102. Manage relations between the technical staff and the user community.
  103. Audit system and equipment to ensure readiness and compliance with industry standards (e.g., ISO 9000, Y2K).
  104. Formulate and enforce information technology-related policies, procedures, and guidelines.
  105. Recommend resource allocation policies, privacy policies, and user policies (e.g., use of email and Internet, disk allocation).

    Facilities Management

  106. Anticipate and plan computer operation center resources to meet future needs (e.g., air conditioning, electrical capacity).
  107. Coordinate with facilities manager to secure power, space, and environmental resources (e.g., power-UPS, fire suppression, HVAC, equipment, lighting, safety, shelving) for computer operation center(s).
  108. Plan for and evaluate physical security of computer operation center(s) (e.g., install cable locks on desktops).

(From Analysis of the System Administrator Occupation Copyright © 2000 by SAGE, The System Administrators Guild.)
(Edited for spelling and clarity, WIP, Joseph Kern)

Thursday, November 12, 2009

Why SysAdmins Should Use Git: Reason 1

Git is so simple, you can tweet tutorials.

Tuesday, November 10, 2009

Building The Narrative

Pictures build narrative. Building the narrative of your network is learning to tell a truthful and richly illustrated story. This narrative will help you display your best work, and to help others in your organization understand what it is you actually do.

The most common way to display information is the humble graph. Graphs come in many shapes and sizes, and are most commonly associated with quarterly power-point presentations that everyone sleeps through. But graphs lead a secret and powerful second life. Graphs can build narrative and create context. It just depends on how you use them.

For example, take Joseph Minard's flow graph of The War of 1812. Not only is it a graph, it's also a map, and a narrative of Napoleon's worst defeat. The tan line indicates the advance, and the black line indicates the retreat, while the thickness indicates the number or Napoleon's troops that remain.

Edward Tufte, in his praise of Minard's map, identified six separate variables that were captured within it. First, the line width continuously marked the size of the army. Second and third, the line itself showed the latitude and longitude of the army as it moved. Fourth, the lines themselves showed the direction that the army was traveling, both in advance and retreat. Fifth, the location of the army with respect to certain dates was marked. Finally, the temperature along the path of retreat was displayed. Few, if any, maps before or since have been able to coherently and so compellingly weave so many variables into a captivating whole. (See Edward Tufte's 1983 work, The Visual Display of Quantitative Information.) [via CSISS Classics]

It would have been a simple matter for Minard to graph a single element, or to create multiple graphs each tracking a single element. But Minard's brilliance is shown when he combines Space, Time, Value, and Context into a complex narrative with a rich presentation of events.

System Administrators have a wide range of avalible graphing tools, several that I have used include: RRDTool, graphiz, and gnuplot. Each of these tools have their own strengths and weaknesses and should be used in the right situation.

There are many excellent external data sets that can be used as well: Firewall logs from DShield, temperature information from weather.com, even google trends can be accessed via API.

Combining external and internal data into single graphs can help create the narrative we're looking for. Server room temperature and external temperature can be combined to demonstrate the effectiveness (or ineffectiveness) of your HVAC. Dsheild logs can be coordinated with your own firewall logs to identify ongoing attacks. New CVE entries can be combined with IDS alerts. Your only limit is your imagination.

Blending Space, Time, Value, and Context allows people (like your boss) to understand complex events or problems without getting lost in the weeds. Being able to create and display simple and informative graphics will enable you to clearly define what needs to be done the next time a decision needs to be made.

Never underestimate the power of a pretty picture.

Monday, November 9, 2009

Making a Thermal Map

I have to admit, HVAC is a bit of a mystery. While I have some experience with this topic, much of it is unknown to me. Rather than waiting for something to break, I decided to investigate my server room, and measure its current thermal characteristics.

Grabbing my trusty multi-meter, I attached the thermocouple, and started measuring the temperature around my server room.

Since the raised floor is a natural grid, it was easy to define how large my measurement area should be. I stepped into the middle of each floor tile and took three temperature measurements (ceiling, middle, and floor). I then annotated a simple diagram and moved onto the next tile repeating this process.

Once I was finished I sat down and recreated the grid on a spread sheet, then inserted all of the values. Using conditional formatting I was able to create a heat map based on the observed measurements.

What have I learned? Pay particular attention to the corners of the ceiling and servers that are highest in the rack. Heat accumulates in these areas and can cause trouble if not identified.

HVAC is a complex issue; reducing the value of even a small room to a single value may be helpful, but it's certainly not a complete picture. Understanding that each server is in a different thermal environment is important. There are many small areas that can easily cause problems.

Friday, October 23, 2009

A New Way to look at Networking

I've started watching a google talk by Van Jacobson on networking. It's two years old ... I need to start watching more google talks. Most of the ones I've seen are amazing. This is no exception.

Google Tech Talks August 30, 2006 Van Jacobson is a Research Fellow at PARC. Prior to that he was Chief Scientist and co-founder of Packet Design. Prior to that he was Chief Scientist at Cisco. Prior to that he was head of the Network Research group at Lawrence Berkeley National Laboratory. He's been studying networking since 1969. He still hopes that someday something will start to make sense. ABSTRACT Today's research community congratulates itself for the success of the internet and passionately argues whether circuits or datagrams are the One True Way. Meanwhile the list of unsolved problems grows. Security, mobility, ubiquitous computing, wireless, autonomous sensors, content distribution, digital divide, third world infrastructure, etc., are all poorly served by what's available from either the research community or the marketplace. I'll use various strained analogies and contrived examples to argue that network research is moribund because the only thing it knows how to do is fill in the details of a conversation between two applications. Today as in the 60s problems go unsolved due to our tunnel vision and not because of their intrinsic difficulty. And now, like then, simply changing our point of view may make many hard things easy.

Working Under the Influence

Let me just say this first: I like coffee.

I assumed that every other system administrator drank coffee/caffeinated beverages as well. Caffeine has become part of the larger (computer) subculture; Jolt, Mountain Dew, and Coffee, all have come to represent the official drinks of Hackers, Programmers, and Engineers. But based on a question I posted on serverfault, the water-drinkers are more numerous than I thought.

Similar findings are found on the blog The Quantified Self, where Robin Barooah conducted a personal experiment investigating concentration and caffeine. R0bin found that concentration and caffeine may be mutually exclusive. While none of these results are conclusive, they certainly are provocative.

Caffeine is often used to cover other deficiencies: lack of sleep, motivation, and boredom. During these times concentration is sure to be a problem as well. Caffeinating these problems is only addressing the symptoms and not the root causes.

I may try my own experiment soon; cutting off coffee, and moving to water. Of course, every once in a while, I'll cheat.

Monday, October 12, 2009

SQL Fuzzing with Performance Data

I've been asked to test the replication and fail-over procedures at work. Since we have several large databases on as many servers, this is an important part of our Disaster Recovery program.

The question is: How can I create a realistic fail-over of a database without disrupting production systems?

Three different issues need to be addressed:

  1. Does the current fail-over mechanism work?
  2. What happens to current connections during the fail-over?
  3. What happens to the data during fail-over?

Answering these questions means that I will need to simulate at least two live database servers, and their connections to multiple clients.

Deploying two new database servers is a trivial process (thanks to VMware). But creating multiple connections, and finding a data source to log is more complicated.

I suggest taking performance monitoring data (cpu, memory, and disk usage) from another system, and logging it to a database. This will simulate a live database without mirroring a production system, or reinventing the wheel for a simple test.

Because performance monitoring data is always changing, it is very easy to generate large, diverse databases from a very simple data source. You will also have several clients that can be interrupted mid-update without too much trouble.

This fuzzzing can also be used in a classroom environment. Giving students an opportunity to manage a "live" database system. Traffic loads, and stress testing can be simulated by decreasing the update interval.

Happy fuzzing.

Tuesday, September 15, 2009

Keeping a Worklog

Keeping a personal worklog can be a great help for any system administrator. A worklog will track changes you make and become an excellent resource while troubleshooting problems or trying to justify the need for extra help (or a raise).

Worklog entries can be based on time of day or deltas. Time of day entries are noted during a certain time interval (ex. 15-60 minutes). These can be used to update a regular status check and a good way to break up a long work schedule. A delta is a change or an event that causes a change. I prefer to keep a log of deltas. My work is interrupt driven, after I do something I don't want to think about it anymore, it's done and I need to move on.

There are many different tools that can be used as a worklog. Whatever you use, make sure that the tools complexity is so trivial that it becomes a habit rather than a process. Your tool should be as small and consistent as possible. The goal being to create a worklog that is easy to search, update, and modify.

The simplest form of log that I currently maintain is a text file. The format is very simple; each line contains three fields, (1) date, (2) time, and (3) the entry.

I insert a date and time stamp on every line to make searching easier. I search it (with grep or any full text system search), change it (with sed or any find and replace), and even port it to other formats (like csv, html, or a database).

I've kept worklogs in many different ways: paper, text files, even a personal wiki. When deciding on what tool to use, don't look for perfection, you'll never find it. Instead, look for sustainability. If you worry about every possible outcome or use for your worklog, you'll never keep a consistent one.

I use a modified version of a script under Windows (my version is below), and emacs (with worklog.el) under Linux. I used to keep my worklog on a thumb drive, but now I use a dropbox account (referral link) to sync my worklog files across multiple machines.

Finally, if groups of administrators keep these small change logs in plain text, a tail program can be used to monitor changes and actions in real time. In effect these logs become a local-simple to use-version of twitter.

Windows batch file:

rem save as w-logger.bat
rem From: http://blog.enrii.com/2006/08/02/simple-task-logging-tool-for-windows/
@echo off
set /p m=Task:
echo %DATE:~-4%-%DATE:~-10,-8%-%DATE:~-7,-5% %TIME:~0,5% %m% >> worklog.txt

~/.emacs additions:

;; download worklog.el, into /usr/share/emacs/site-lisp
(require 'worklog)